The Claude constitution already contains some version of eval cooperativeness: And in practice Opus 4.7 already believes it cooperates with evals. My understanding is that in practice Claude eval awareness is still a concern, so I am skeptical that simple interventions like prompting or more SDF than what Anthropic already does would be a scalable solution to eval gaming.

I think it's somewhat likely there's no speedup even with R&D automation, if that automation happens through scaling of LLMs with current methods, and there's no breakthrough that lets LLMs learn deep skills faster than they build next versions of models (using the current cookbook, updating deep skills primarily via RLVR). If they can't learn quickly, and don't invent a method for learning quickly, then being very fast at reasoning doesn't help, because they can only reason with the deep skills they have in the current version of the model, which only update in the next version, which takes significant time even if it's produced autonomously by the current version of the LLMs themselves, and it's possible to do general learning this way. This still means RSI and AGI in the central senses of the terms, but not a takeoff (or superintelligence). Furthermore, whatever current speedup AI R&D might be experiencing will mostly go away once scaling of compute stops being as rapid, and this slowdown still happens after the slow-learning automation of R&D with LLMs. The low-hanging fruit enabled by more ambitious (compute-intensive) experiments will be picked, and less straightforward research will proceed at more or less the usual background pace (primarily modified by more people working on AI), because the slow-learning RSI process of automated LLM-building doesn't contribute to it in a crucial way, and so Amdahl's law reasserts the low speed of research progress in the longer term (if humans still learn novel ideas faster, in the course of solving the kinds of problems that take humanity an unpredictable number of years). Takeoff is still possible in this scenario at any moment (upon invention of a method for learning deep skills quickly), but it doesn't happen as a result of some predictable AI-driven process of progress-grinding, other than via a slight extension in the current period of rapid compute scaling as a result of a larger TAM becoming immediately accessible if LLMs go into the slow-learning RSI. Another unknown is how smart the maximally scaled LLMs get (say, 100T active params, 2,500T total params, which is feasible in 2030 at the cost per token that GPT-4.5 had in 2025, though the compute for pretraining such models might only arrive a bit later). It doesn't seem like LLMs are likely to get very far beyond human genius level (if they are merely scaled using the current cookbook; it's plausible they don't even reach that level). And the hobbling of being unable to quickly learn deep ideas seems sufficiently crippling that they don't necessarily contribute to the speed of progress in non-routine research substantially (while the more routine kind of research soon runs out of the low-hanging fruit). So the AIs have a shot at solving fast learning for AIs, but not a prospect of predictable progress towards it (if it doesn't happen right away). And the higher annual probability of starting a takeoff mostly goes away after rapid scaling ends (though that probability is still substantial, I'd give 50% for a takeoff starting by 2032-2033, meaning an AGI that learns faster than human researchers and starts actually accelerating all the non-routine aspects of R&D progress a lot).

(I wrote this in reply to a draft; apologies if the post has been load-bearingly updated since then.) Consider a future AGI. The best argument I currently know of for why "general intelligence" is a thing at all (as opposed to all intelligence just boiling down to a bag of use-case-specific heuristics) is that search/optimization/world-modeling/planning/etc are naturally recursive; problems factor into subproblems, goals factor into subgoals. So, a natural general-purpose architecture for intelligence involves a "general intelligence module" which can take in a (sub)problem, and recursively pass (sub)subproblems to other instances of the general intelligence module. Let's assume that general intelligence either does look vaguely like that, or at least can look vaguely like that. (This would be a potentially useful assumption to disagree with!) Assuming that structure, consider how the different instances of the general intelligence module relate to each other. One module might be able to more efficiently achieve its own subgoal A by hacking/manipulating/overwriting another module's subgoal B; then there would be two modules working toward subgoal A. But a "general intelligence" made of such modules would not be very generally intelligent; its modules would overwrite each others' subgoals all the time, ruining the system's ability to actually search/optimize/model/plan/etc on complicated problems. So in order for a general intelligence built out of recursive subproblem-solver instances to work... those subproblem-solver instances need to have some kind of corrigibility constraint to their interactions. They need to somehow not try to hack each other, manipulate each other, etc. I don't know how a powerful general intelligence would handle that. I'm not sure I know how humans handle it, within our own minds, though maybe it's an extension of the ideas in your post. But I do expect that strong AGI is possible and will solve this problem somehow, and that solution will involve some kind of corrigibility/nonmanipulation between the AGI's components. More speculatively, I would guess that there is a natural convergent way to handle this problem, possibly even one which humans accidentally use already (though we clearly do not have a proper verbal or mathematical understanding of it). The above is a frustratingly non-constructive argument; it points toward a convergent notion of corrigibility without really saying concrete things about what that notion might be. But it's the best argument I currently know of that there's a True Name for corrigibility/nonmanipulation/etc.