Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It’s so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture.

Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of:

(1) Imitative learning, including pretraining and supervised fine-tuning (SFT)
- See my earlier discussion: “LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work”.
(2) Reinforcement learning, including RL from human feedback [RLHF], RL from AI feedback [RLAIF], and especially RLVR.^[1]

If we look at the final trained LLM, we can ask how important each of those two pieces was, in explaining the LLM’s capabilities. And my claim is that it’s way more (1) than (2).

I'll start in §1 with some relevant evidence, and then in §2 I’ll circle back to operationalizing exactly what I’m claiming, and finally in §3, three reasons why we should care—namely, it affects how we should think about chain-of-thought legibility, about LLM capabilities, and about LLM alignment.

Note that I am not arguing that RLVR does not importantly contribute to LLM capabilities. That would be absurd! Of course it does! Companies use RLVR because it works, and I expect them to continue doing so more and more. Again, the things I’m actually claiming are in §2–§3.

1. Some relevant evidence

1.1 Theoretically, each GPU-hour spent on RL conveys orders of magnitude less information content than a GPU-hour spent on imitative learning

The fraction of total LLM training compute that goes towards RLVR is high (one source guessed 20%; for all I know, it could be >50%). But that’s different from how to divvy up the credit that pretraining/SFT versus RLVR can take for LLM capabilities. Why is that different? See:

Based on those posts, if RLVR is 50% of the training compute, then it might only be 0.01% of the information content imparted by training. And unless the learning rates are drastically different, it would also be 0.01% of changes to the weights.

1.2 The chain-of-thought (CoT) is still obviously strongly influenced by imitative learning

People emphasize how frontier model CoTs can be pretty weird, and that’s true and important, but if you step back a bit, in the grand scheme of things, CoTs are still very obviously strongly influenced by imitative learning from human-created (and hence legible) text.

See for example,

“Even ‘illegible’ Mythos reasoning traces seem pretty legible” (@faul_sname 2026)

I prefer to start from the opposite side. What would a really inscrutable CoT look like? It would look like random tokens: “… extracurricular(boolean ger застос autism relevance directamenteдвэр QUICKchem …”

CoTs sure don’t look like that! So on a spectrum from “totally legible” to “looks like random tokens”, we are way closer to the “totally legible” side, as of today.

What does that imply about how LLMs get their superpowers? Well, imitative learning by itself will necessarily create legible CoTs; whereas in RL, the CoT is generally irrelevant to the reward, so there’s no force keeping it legible, and thus we should expect it to drift into an ever less legible and more “optimized”^[2] direction.

Thus, the fact that CoTs are still pretty legible is some evidence that RLVR today has not had that much influence on LLMs, as compared to imitative learning.

1.3 LLM companies still seem to care a lot about imitative learning (pretraining & SFT) data, not just RL environments

See “Most Algorithmic Progress is Data Progress” (Beren Millidge 2025), plus further discussion in my post “The nature of LLM algorithmic progress” §1.4.

And LLM companies today continue to spend money on even more training data for imitative learning (especially expert reasoning traces for SFT, I think).^[3]

1.4 Three papers claiming that non-RLVR’d models can get into the same ballpark of capabilities as RLVR’d models, although maybe we shouldn’t trust those papers too much

“Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” (Yue et al. 2025; site, arxiv, lesswrong discussion):

The authors found that when an RLVR’d model could solve a problem, the equivalent non-RLVR’d model could generally solve the same problem, maybe not on the first try, but in one of the first 1000 tries or whatever.

“Reasoning with Sampling: Your Base Model is Smarter Than You Think” (Karan & Du 2025; site, arxiv):

The authors found that an “iterative sampling algorithm” could boost a non-RLVR’d model to be comparable to the corresponding RLVR’d model.

“Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights” (Gan & Isola 2026; site, arxiv):

The authors found that you can take a non-RLVR’d model, randomly perturb the weights, and ensemble the results by majority vote, and you can get results comparable to an RLVR’d model.

Caveat for all of these: I don’t think we should over-anchor on any of these, mainly because they’re all now outdated, and based on open models rather than bleeding-edge LLMs with bleeding-edge RLVR best practices.

Also, there’s another paper “ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models” (Liu et al. 2025, arxiv) which claims “that RL-trained models consistently outperform base models … including scenarios where base models fail entirely regardless of the number of attempts”.^[4]

1.5 A paper suggesting that RLVR mostly refines the heuristics controlling which (already-known) reasoning strategy to use in which situation

“Base Models Know How to Reason, Thinking Models Learn When” (Venhoff et al. 2025; arxiv)

Using interpretability tools, the authors conclude “RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones”.

I think the basic idea is that, to do well in long-chain-of-thought reasoning, there are a bunch of moves that you can make: press on, backtrack, reexamine assumptions, pivot to Alternative Strategy X, etc. The LLM “knows” all those moves from imitative learning, but doesn’t have refined heuristics on which move to make in which situation. These kinds of heuristics are very important for making progress in a long CoT, and they’re somewhat domain-specific, and you can’t really pick them up well by imitative learning. So that’s the main low-hanging fruit that RLVR picks, and it makes a huge difference in practice.

Caveat: Just as in the previous section, this paper is old (by RLVR standards), and not based on bleeding-edge frontier LLMs with bleeding-edge RLVR best practices. I guess in this case I’m slightly reassured by the fact that one of the coauthors, @Neel Nanda, works with near-SOTA LLMs at his day job at DeepMind. But only slightly.

Anyway, I’m taking this claim as a good default mental picture, not an ironclad eternal truth. In particular, I remain open-minded to the Liu et al. 2025 claim that “RL can indeed discover genuinely new solution pathways entirely absent in base models, when given sufficient training time and applied to novel reasoning tasks.” (But I’m also open-minded to the opposite! Liu et al. make that claim but I don’t think they really prove it.)

(For more discussion and references on this topic, see “Some Interesting Papers on RLVR” by @CarolusRenniusVitellius.)

2. What am I actually claiming here?

What’s my point? I feel most strongly about the following intuitive picture:

We start with a randomly-initialized LLM. It “knows” nothing at all, and regardless of context, it will output random tokens like “…OGRмагогласно વિચારDisposable! 끄registr Southeastern follic…”.
Then we do imitative learning, and now we get a “base model” that (in some sense) “knows” how to speak every language in the world, and “knows” pretty much every fact that’s ever been written down, about every topic from string theory, to needle-felting, to what’s the hottest bar in Crikvenica, Croatia.^[5] It has learned the human distribution.
Then we do RL, especially RLVR, where we’re mostly tweaking it to have better heuristics for which human-distribution strategies to apply in which contexts during long-chain-of-thought reasoning (cf. §1.5).

No question that the RL piece is big. It’s huge. It’s an elephant in the room. But if the RL piece is an elephant in the room, then the imitative learning piece is the friggin’ Mt. Everest in the room, if we’re trying to have a good mental picture of how LLMs can do the things that they can do.

That’s all intuition. How do we move beyond that into a concrete operationalization?

It might be interesting to look at KL divergence or weight changes between randomly-initialized models versus base models, and between base models versus RLVR? I can’t immediately find literature on that.

Or we could talk about “bits of selection”. If something like the Yue et al. 2025 results above are valid, then imitative learning might get us from probability of accomplishing a certain task, to 0.1% probability. And then RL might get us from 0.1% to 90%. It would follow that almost all the “bits of selection” come from imitative learning (322 bits versus 10 bits, in this made-up example). (See: “privileging the hypothesis”.) (Yes I know, the Yue et al. 2025 results shouldn’t be taken at face value, but I think this is still the right bottom line.^[6])

(Of course, to reiterate, all this is still compatible with RLVR being very important in a pragmatic sense: the difference between working 0.1% of the time versus 90% is a big deal!)

3. Why does any of this matter?

Here’s three debates where this topic is relevant:

3.1 Thinking about CoT legibility (both today and in the future)

There’s a nice explanation by @Rohin Shah, on the 80,000 hours podcast (recorded December 2025):

Rob Wiblin: …Just to check that I’ve understood, you’re saying because we are using GPUs and TPUs, structurally that forces the thought of the model to [not] be very deep: it can have many things in its mind at one time, but there can’t be very many steps, because you have to be going through all of these things in parallel. But if it was very [deep], then you wouldn’t be able to do them all simultaneously; you would have to wait until the earlier steps were done to do the later steps. And this is just something that is going to remain the case for years to come. …
Rohin Shah: Yeah, so that was stage number one, which is like, why should we expect the opaque serial depth to stay low, at least during the pretraining phase?
Then there’s step two, which is like, we’re saying that the pre-trained model has to essentially speak in English in order to do reasoning. But then there’s a bunch of post-training and RL and all of this sort of stuff. Maybe that’s going to make it so that even if the model is speaking in tokens, maybe it will start speaking some sort of alien language that we don’t understand.
And here I would say there’s no theoretical argument, or there’s no theorem you can prove that will say, no, they’re not going to do this. But I will say they are born speaking English in some sense, and pretraining is by far the most powerful form of getting stuff into an AI system that we have ever built. So the model is incredibly good at speaking in natural language. It’s great at doing reasoning, but only the kind of reasoning that humans do when writing stuff down.
And then when you look at the reasoning training that we’re doing today, there’s a paper from Neel Nanda and others that shows that actually a substantial part of what the reasoning training is doing is just teaching the model when to do a specific kind of reasoning step that it had already learned during pretraining.
So basically a lot of the capability is just coming in pretraining. We know that the pretraining, for good reason, is going to stay the way that it is, and be done using human-like reasoning and speaking in English. And then the RL is just really inefficient relative to pretraining, and for it to build an entirely new epistemic language that is something that we wouldn’t be able to understand [even] with a decent chunk of effort, is just so far beyond what RL is doing currently that it would be pretty surprising to see that in the near future.
Now, do I expect to see it ever? Yes. But I think six months ago, I split a room in a conference by saying, “Agree or disagree? Chain of thought monitoring will continue for two years.” I think I said two years. It might have even been one year. I forget. Whereas my median is probably like four years, five years, I don’t know. It’s not totally clear, mostly because of this argument about how hard it would be to go beyond the current situation.

This is kinda the mirror image of my §1.2 argument above. I was saying in §1.2: CoTs are legible, therefore LLM capabilities mostly come from imitative learning. Whereas Rohin is saying: LLM capabilities mostly come from imitative learning, therefore CoTs are legible, and this will not change too soon, absent some important future change in LLM training approach.

3.2 Thinking about LLM capabilities (both today and in the future)

Insofar as LLMs are powered by imitative learning, we should expect them to mainly be competent in domains where there is human data and human concepts to learn from.

Insofar as LLMs are powered by RLVR, we should expect them to mainly be competent in domains where there are verifiable (automatically-checkable) rewards.

I think both are relevant. And indeed, LLMs seem to be at their best when we have both. For example, software debugging allows both copying from abundant human concepts and human reasoning traces, and testing results in a controlled environment.

By contrast, I have heard lots of people talk as if verifiable rewards are sufficient by themselves—that as long as we have verifiable rewards for such-and-such domain, LLMs will sooner or later be able to master that domain, even if the domain is arbitrarily far from anything humans have ever thought about. Well, that’s a hypothesis. Is it true? Seems to me that the jury is out; I don’t recall seeing demonstrations of such a thing to date. Anyway, I think my headline claim (“LLMs are (still) mostly powered by imitative learning, not RL”) provides some helpful background intuitions and context as we try to think through what to expect here.

3.3 Thinking about LLM alignment (both today and in the future)

I think a very strong general rule-of-thumb is: if you create AI capabilities via RL,^[7] you get ruthless sociopathic behavior. (See my post “Why we should expect ruthless sociopath ASI”.) The logic is basically: whatever the reward function is, reinforcement learning is reinforcing the ruthless and relentless pursuit of a high score on that function, by any available means.

By contrast, if you create AI capabilities via imitative learning, you get models that follow the human distribution of outputs. (Cf. “personas”, “simulators”, etc.) The human distribution has its problems, Lord knows, but it’s not nearly as cursed as the “100% ruthless sociopath all the time” that you normally get from building capabilities via pure RL training.

For example, the Bing-Sidney chatbot from 2023 was trained by pure imitative learning (pretraining + SFT), without any RL at all.^[8] Was it misaligned? Hoo boy, was it ever! But it had a very human flavor of craziness, in contrast to the very RL-ish flavor of ruthless-goal-seeking craziness that we often see these days (and this week in particular!).

Anyway, that’s my proposal, and I claim that it’s fully compatible with the alignment-related evidence that we’ve seen from LLMs to date. They are definitely not 100% ruthless sociopaths all the time. But they are somewhat like that sometimes. Correspondingly, RLVR is part of, but not most of, how LLMs get their capabilities today.

If we want to refine that idea, we get into my “Foom & Doom” §2.3.5 picture where LLMs have some niceness from imitative learning, and then the more you do RLVR, the more that niceness gets diluted away.

There seem to be ways to mitigate the damage of RLVR (e.g. inoculation prompting), but that’s still directionally what I strongly expect to happen.

How fast does RLVR dilute away the niceness? Unclear. But if LLMs are (still) mostly powered by imitative learning, not RLVR, and yet this (comparatively) small amount of RLVR is already enough to cause obvious RL-style alignment problems, then that’s a bad sign for our ever-more-RLVR LLM future.

^{^}
There are a couple edge-cases in this breakdown, and the way we should handle those cases depends on what we’re ultimately trying to figure out. My interests are listed in §3, and in those contexts I would say: (1) I agree with Arthur Conmy’s take that Direct Preference Optimization (DPO) usually has more in common with imitative learning than RL, even though it’s sometimes lumped in with RLHF; (2) If Company A uses RLVR, and the CoTs get posted on the internet, and then Company B does SFT on those CoTs, then this should count as RL even though it’s structurally imitative learning. These edge-cases won’t really come up anyway; the paradigm case that I want to talk about is a big AI company doing pretraining on human-created internet text, SFT on their own proprietary data, and RLVR on their own proprietary training environments.
^{^}
Cf. the Karpathy quote: “You can tell the RL is done properly when the models cease to speak English in their chain of thought”.
^{^}
How much money? Not sure. The total amount that LLM companies are spending on “data” is huge (e.g. “$10 to $15 billion per [LLM company]” per year), but that claim seems to involve a broad definition of the word “data”, that combines both imitative learning data (pretraining & SFT) and RL training environments. I can’t easily figure out the breakdown in spending between imitative learning versus RL. However, the imitative learning part definitely seems to be more than zero. E.g. a recent blog post by an industry insider mentions “high-quality midtraining tokens”.
^{^}
The paper says “regardless of the number of attempts”, but that raises the obvious question “well, how many attempts did you try?” The paper does not make that clear, unless I missed it. Their graphs top out at pass@256 (i.e., probability of succeeding at least once, given 256 tries), but they must have done a bunch of pass@256 runs to get a pass@256 probability. Based on the plots, maybe attempts total?
^{^}
(A town I picked randomly by pointing at a world map.) Gemini 3.1 Pro without internet access suggests Sabbia. I’d go!
^{^}
As mentioned above, Liu et al. 2025 found tasks where RLVR succeeded but the base model pass rate was below , and perhaps far below. However, those tasks seem to be ones that require long answers with lots of steps; so in those cases we would also need to lower the starting (random policy) success probability, to something way below . So the bottom line would stay the same: imitative learning is responsible for far more of the “bits of selection” than RL.
^{^}
Note that RLHF is not really a “creating AI capabilities”; rather, conventional wisdom is that RLHF makes models dumber, not smarter, as a side-effect of making them nicer. Again, this post is mostly about RLVR and similar.
^{^}
Source: see this comment by @gwern.