It is true, in a sense, that LLMs are only next-token predictors.
Throughout their training, they are asked to predict the next token in a sequence. When they succeed, gradient descent adjusts the neural network’s parameters in a way that reinforces the weights that contributed to that success; when they fail, parameters adjust in the opposite direction. At inference time, the model is given a context and predicts the most likely token given that context. That token is then added to the context, and the process repeats.
Even Reinforcement Learning is next-token prediction. A model is asked to predict the next token on its own output (when that output was deemed satisfying), and gradient descent occurs.
And so, some conclude, there is no real intelligence inside LLMs.
There are two problems with that argument.
The first of course is a confusion of the level of analysis. To say that an LLM is “just predicting the next token” is like saying that a human being is “just contracting muscles” when speaking, or that a chess engine is “just moving electrons” when beating a grandmaster. At a low enough level of description, any complex capacity can be made to sound trivial.
The second is the assumption that a simple optimization rule can only give rise to simple structures. In reality, very basic rules can force the emergence of sophisticated mechanisms.
Complex urban architecture (highways, ring roads, roundabouts, traffic norms…) is the result of optimizing for “allow people to go from point A to B as fast as possible”. Global supply chain, financial markets and price systems are the result of individuals optimizing for “maximise profits”. The best analogy, however, is biological evolution.
Gradient descent as biological evolution sped up on GPUs
The only rule governing biological life: genes that produce more copies of themselves spread. Evolution optimizes organisms to “maximize the number of their offspring”. Nothing in that function contains instructions for multicellular organisms, DNA, intelligence, a sense of morality, the drive to cooperate, the ability to love or the capacity to engage in abstract reasoning. All those things emerged because they were the efficient ways that evolution stumbled upon to optimize for the simple Darwinian objective. For example, nature “found” that giving creatures subjective experience of consciousness maximised their rate of reproduction. Why? Because it motivated them to escape predators and to seek mates.
In other words, the simplicity of the optimization target does not cap the richness of the by-products it can give rise to.
In LLMs, next-token prediction plays the role that Darwinian selection played in evolution: a simple optimization pressure that generates rich “cognitive” mechanisms. This is because it is impossible for LLMs to correctly predict the next token without these internal abstractions. Gradient descent therefore had to stumble upon them.
In humans, these complex inner mechanisms were stumbled upon by evolution through trial and error (random mutations); the mechanisms that happened to increase the number of offsprings were then made to persist by natural selection. In neural networks, backpropagation - powered by massively parallelized compute on GPUs - finds in which direction it would be optimal for the parameters to move (non random mutations); gradient descent then takes the neural network towards these new combinations of weights (the equivalent of natural selection).
An LLM is a world model
One of the first examples of a model developing complex inner abstractions came in 2017, when OpenAI trained a small model to predict the next character on a dataset of 82 million Amazon reviews. When researchers looked inside the model’s activations to understand how it was performing the task, they found that one neuron was tracking the sentiment of the text. It would activate when it saw words like “stunning” or “perfect” and deactivate for words like “disappointing”. The model developed a theory of mind because this helped it predict the next character. More recently, researchers trained a small LLM on nothing but sequences of moves from the game Othello (”E3, D3, C4...”). The model never saw a board nor learned the rules. Yet, to predict the next move accurately, it built a 3D internal map of the 8x8 board, tracking within its neuronal activations which squares were black, white or empty. (This isn’t too surprising: the most efficient way to predict the next sequence in the training data - without memorizing the entire dataset - is often to internalize some deep truth about the process that generated the data in the first place. This is why abstract models of the world emerge in LLMs.) Another illustration is the case of an LLM trained to predict the next token in a dataset of modular addition, which internalised in its weight a trigonometric function (because modular addition can be represented as a rotation on a circle). A superior, abstract mathematical truth, emerged from a narrow optimization exercise.
Today, LLMs with trillions of parameters trained on vast datasets have built sophisticated internal abstractions. This likely includes some abstractions that we do not even suspect exist, or abstractions that do not reflect reality but are, weirdly, helpful in predicting the next token in the training distribution (which poses questions for AI alignment, which we come back to below).
Eventually, complex enough internal abstractions become general intelligence. In a sense, the emergence of AGI from LLMs should not be surprising: the most efficient way to predict the next token on a very large corpus (maths, psychology, physics, strategy, code, law, fiction stories, social interactions, common sense…) is, well… to develop general intelligence. With enough scale (compute and data), gradient descent was perhaps bound to stumble upon general intelligence as a token-prediction mechanism, in the same way that evolution stumbled upon general intelligence as a way to help humans overcome a wide range of challenges (and thus spread their genes efficiently).
Why next-token prediction could kill us all
The difference between the task optimized for (predict next token) and the inner mechanisms developed to tackle the task is what makes AI alignment so complex. The internal mechanisms an LLM develops - that work well on that training distribution - may generalize in strange ways outside it.
For example, perhaps the best way to predict the next token in examples of an AI assistant refusing to speak to the user about sex is to develop internal values that ascribe a horrific, negative value to human sexuality. Perhaps one day, an agentic LLM equipped with this abstraction sees a novel, weird kind of pornographic content and creates a virus to destroy the servers hosting it.
Here again, there is a parallel to human evolution. The optimization target was to maximise the number of offsprings, but the cognitive mechanisms born in response lead today to behaviours unrelated to the original goal. For example, evolution did not put the drive for “reproduction” into our brains directly. Instead, it made us crave sex. In modern environments (absent from the evolutionary training dataset), this drive leads to behaviour unrelated to maximising the number of offspring (bars, dating apps, prostitution, nightclubs, etc.) and that can even run counter to it (waiting longer before marrying in order to sleep around, therefore having less kids). Similarly, evolution couldn’t program the instruction: “Eat a lot to stock calories when resources are scarce but eat just enough in times of abundance to stay healthy.” Instead, it simply made us crave sugar. Today, we satisfy this proxy drive by eating at McDonalds and creating snacks with no nutritional value. This creates obesity and increases mortality, which is orthogonal to evolution’s objective.
Because LLMs are black boxes, we don’t know which internal abstractions they rely on to predict the next token and what outputs (or actions) those abstractions could lead to in new contexts. We do have evidence that undesirable abstractions can develop easily. For example, in one study, an LLM fine-tuned to predict the next token on malicious code began adopting, even on tasks unrelated to coding, the persona of a malicious human. No one trained the AI to become evil, but internalising something like the perspective of an evil human was the method gradient descent stumbled upon to allow the LLM to accurately predict the next token on malicious code. Of course, this was a controlled experiment, but we cannot be certain that internalising the perspective of an evil human (perhaps of an evil human who fakes alignment) isn’t the method that gradient descent has stumbled upon to allow an LLM to accurately predict the next token on some of the training data. The point is we cannot test for every possibility.
Eliezer Yudkowsky and Nate Soares believe that over time, the probability that these abstractions will lead, at least once, in at least one context, to an AI pursuing an objective incompatible with humanity’s survival, tends to 100%. If the AI has infinite intelligence and power to pursue the misaligned objective, there are therefore far more paths leading to the end of humanity than paths not leading to it. Extinction due to Artificial Superintelligence is, in their view, a quasi-certainty.
