We Don't Understand Neural Networks At The Algorithmic Level

The largest ongoing debate about AI is “Are Large Language Models (LLMs) intelligent?” That makes sense, at least: the evidence is ambiguous and the stakes are high. Far more confusing is the disagreement on whether we understand LLMs - how can scientists studying a topic not broadly agree on how much we know about it? Yet respected academics have conflicting opinions, and it is worth seeing why, because it underlies the question of intelligence.

On one side, prominent figures, scientists among them, have described AI models thus:

powerful digital minds that no one – not even their creators – can understand

Skeptics are right to scoff at the marketing hype of “powerful digital minds”, but many scientists agree that our understanding of LLMs is very limited. Here is a more subdued example:

Language models have become more capable and more widely deployed, but we do not understand how they work.

The other side of this debate is presented well in a new preprint by Olivia Guest, Nancy Abigail Nuñez Hernández, and Mark Blokpoel, “Understanding Artificial Neural Networks: Mysterianism about Known Mechanism is Mysticism ”. Disagreeing with the above two quotes, which they provide as examples, they write:

We do know the mechanistic structure of [artificial neural networks (ANNs)] because we designed and built them. We also do know their functional role (what they are for) as well as the mathematical function they are asked to approximate (map inputs to target outputs).

As they correctly say, there is a lot that we know. But is it enough?

They also write this:

ANNs, as turbo-charged statistical models [..] can only but provide correlations.

This position is, I think, representative of a particular line of AI skepticism, related to the “Stochastic Parrot ” and “Blurry JPEG of the Web ” metaphors. There is a lot of truth in those views! However, this preprint is an example of how they can be taken too far: we do not fully understand ANNs, and in part that is because they can do more than provide correlations.

Here are the main points I’ll be making in this post:

Even though we understand ANNs’ goals at the top level, and their math at the bottom , we don’t understand the algorithms in the middle .

This is so despite the simple math in each artificial neuron. Enough such simple units are provably Turing complete : capable, in principle, of any algorithm. There is a vast number of things that ANNs could be doing internally. Scientists are still figuring out which.

The preprint, like Ted Chiang’s “Blurry JPEG of the Web” piece, views ANNs as storing data , a compressed version of the training set. Still, we must ask - as Chiang does - how do ANNs compress? And that question brings us back to algorithms.

I am not saying anything new here, but I think it’s worth seeing this all together, and in the context of whether we understand ANNs.

(Note: LLMs are ANNs, that is, large language models are artificial neural networks. I’ll mostly use the term ANN, following the preprint, except where the distinction matters.)

Levels of analysis

The cognitive scientist and neurobiologist David Marr defined three important levels of analysis:

Computational : What the system aims to do at the highest level, what tasks it is built to solve.

Algorithmic / Representational : How, in an abstract sense, the system represents information, and how it manipulates those representations to complete its tasks.

Physical / Implementation : The actual details of how the system is built, at the lowest level.

Tragically, David Marr died at a young age in 1980. That was before computers were ubiquitous, but today, computers are probably the easiest way to explain his levels. Let’s focus on one type of computer program, a chess engine, which plays chess. Today, chess engines often use ANNs, but for now I just mean a “traditional” engine - some ordinary code that people wrote, that plays chess (most engines were like that until a few years ago). Here is what Marr’s levels mean for such a chess engine:

At the computational level, a chess engine plays chess . Given a board, it picks the next move. Its goal is to pick the best move so as to eventually win the game.

At the algorithmic level, a chess engine picks the next move using some specific approach.

It may have a list of memorized positions and outcomes, which is useful at the start and end of a game. Looking at possible moves, it sees which are memorized, and picks the one with the best outcome.

It may use calculation. While it is impossible to calculate all possible moves to any useful depth, heuristics can focus on promising ones (according to some rules, like checking all captures), while estimating the outcomes (according to some other rules, perhaps giving knights 3 points and rooks 5). It then picks the move with the best estimated outcome.

At the implementation level, a chess engine is written in some...