Vladimir_Nesov7d6-8

Minimum required pacing: finish aligning it before you train the next one

Pretraining scaling is starting to slow down, both in terms of compute and model size. The mad rush of 2025-2026 is done, and the models targeting 2027 hardware are probably already pretrained (though the work on their misalignment is then only getting started). In 2026-2028, scaling is about 2x slower than in 2025-2026, and then in 2028-2032 it gets 2x slower yet again. So "pacing" will be happening naturally in some sense. The problem is that the models of late 2026 are passing the level of human capabilities in many ways, so even slower scaling could get significantly harder to handle. And Meta, Google, and SpaceX currently have what seems to be well-made small models (somewhere between Sonnet and Opus scale-wise) and sufficient resources/ambition to plausibly catch up with OpenAI and Anthropic in 2027 at least in terms of model scale (with big models that are not so broken that it ruins their capabilities). To the extent scale is a source of misalignment difficulties that require operational excellence to sweep under the rug, they might have a hard time in 2027. And then slow-learning prosaic RSI of automated RL task/env creation in 2027-2028 probably introduces new misalignment/control issues, even if it doesn't manifest as acceleration or a step-change increase in capabilities, and merely fixes jaggedness by automatically filling in all ordinary capability gaps as they come up. Which is probably economically transformative, makes the LLM/RL paradigm more centrally AGI (by automating learning, in a very clunky way), enables useful humanoid robotics and opens up a path to a robot-industrial explosion, but isn't necessarily a raw capability jump per se. (Continual learning in a strong sense is another matter and probably quickly leads to strong architecture-rewriting RSI, but I don't know of a way to time its invention, other than to say that LLMs of 2027-2028 will be taking a stab at it. Like with math problems currently being slaughtered in droves, it could be "easy enough" in some sense for the LLM/RL paradigm to defeat quickly, but not necessarily.)

Richard_Ngo15d74

Schelling Goodness, and Shared Morality as a Goal

I think this post gestures at something important, but is quite confused in doing so. In particular, I agree that the concept of Schelling points is very important for understanding coordination. However, I think this post fails to distinguish between coordination towards good and coordination towards evil. The thing it calls "Schelling goodness" is often a tool for turning people against their own moral intuitions, especially in settings where people aren't able to encourage the behavior they actually want. For example, if you stopped people on the street in a dictatorship and asked what should happen to an opponent of the regime, the safe (and Schelling) answer might be "they should be executed". You might then say that you need to ask people in a setting where they feel safe to give their "true opinions". But power structures often coerce people into internalizing submission to those power structures, so it's hard to even know what their "true opinion" means. And even if you have a reliable way to get someone to think freely about what they prefer, doing so privately doesn't do much to change the Schelling answer—because the Schelling answer is anchored to public knowledge. In other words, even when individuals are able to come to the intuitively right moral answer, the application of power might cause the Schelling answer to remain stuck on something bad (and indeed, I suspect that we can think about this ability as constitutive of power). I discuss many of these dynamics in this blog post. You might hope that these dynamics change when thinking about the cosmic Schelling answer, since it's harder for distant civilizations to coerce each other. But the main reason to take the cosmic Schelling answer seriously IMO is that there are some channels for such civilizations to affect each other, and coercive interactions seem like they're probably fairly easy (e.g. simulating things with high disutility for another civ). On the meta level: I was thinking about making this comment a few days ago, after a conversation with Scott Garrabrant, but was prompted to do so by the recent announcement that SFF is revoking John Wentworth's funding. I don't know who made that call, and want to think more about it before writing publicly about it, but the intuitions in this comment play an important role in how I evaluate decisions like this.

Recent Discussion

The Alignment Problem: Computer Science Meets Analytic Philosophy

Vanessa Kosoy

TLDR: I explain what the technical AI alignment problem is and what its core difficulties are, in my view. This is a perspective that draws simultaneously on classical traditions in theoretical computer science / machine learning theory, ideas from analytic philosophy, and views that originated in the Machine Intelligence Research Institute / LessWrong.

Minimal Viable Safety

The Task

The problem of making AI safe is underspecified if we haven't decided what the AI is supposed to do. For example, we can run any AI inside a homomorphic encryption box and never decrypt the output. This is probably perfectly safe ^[1] , but also perfectly useless.

We can decide on an arbitrary task such as "construct Lean proofs for given mathematical theorems" and declare...

(Continue Reading - 4760 more words)

[Paper] Stringological sequence prediction I

Vanessa Kosoy

6mo

This is a linkpost for https://arxiv.org/abs/2603.26852

TLDR: The first in a planned series of three or more papers, which constitute the first major in-road in the compositional learning programme, and a substantial step towards bridging agent foundations theory with practical algorithms.

Official Abstract: We propose novel algorithms for sequence prediction based on ideas from stringology. These algorithms are time and space efficient and satisfy mistake bounds related to particular stringological complexity measures of the sequence. In this work (the first in a series) we focus on two such measures: (i) the size of the smallest straight-line program that produces the sequence, and (ii) the number of states in the minimal automaton that can compute any symbol in the sequence when given its position in base as input. These measures are interesting because multiple...

(See More - 434 more words)

Alexander Gietelink Oldenziel1d10

Fantastic news. Excited to learn more!

Jacob_Hilton's Shortform

Jacob_Hilton

11Jacob_Hilton4d

Have time horizons in mathematics been superexponential? I have previously expressed skepticism about modeling time horizons as superexponential (absent feedback loops from AI speeding up AI R&D). In particular, I didn't buy AI Futures Project's argument that there are "meta-skills" that let humans do projects of arbitrary length. Spelled out in more detail, my skepticism was coming from something like the following argument: * The "AI time horizon" of a task – roughly, how much output the AI needs to produce to get one unit of feedback – should scale roughly linearly with training inputs (compute, training efficiency, etc.). This is supported by theoretical and empirical evidence in this paper. * The "human time horizon" is somewhat subtle to define, since it depends on which humans you ask to complete the task. If you ask a random human, they will probably not be able to solve an IMO problem given essentially unlimited time, so there is a sense in which an AI that can do this already has an infinite time horizon. Therefore it is more coherent to consider a "moving baseliner" definition, in which we ask how long it takes humans who are competent at the task complete it. This is also a more reasonable way to extrapolate METR's methodology, who looked at the (geometric) mean time taken to complete a task among those who successfully completed it. * Intuitively, the "AI time horizon" and "human time horizon" (moving baseliner version) should be similar to each other? (This is kind of a speculative assumption.) * Therefore, if training inputs scale exponentially (which is the historical trend, and a reasonable extrapolation absent AI R&D feedback loops), we should expect human time horizons to scale exponentially. However, qualitatively, it appears that the (human) time horizon of AI in formal mathematics has progressed faster than exponentially recently. This view has been expressed by Noam Brown: [...] I would be very interested to know to what extent a mor

3elifland3d

https://www.lesswrong.com/posts/G7MmNkYADKkmCiumj/response-to-titotal-s-critique-of-our-ai-2027-timelines#Superexponential_in_effective_compute_time_horizon_growth contains a much more developed version of our argument for superexponentiality than the one you linked to!

Jacob_Hilton3d32

Thank you! I still feel confused about the abstract considerations here, and given that we now have a rich, technical domain where AI is superhuman, I would put a lot of weight on whether math looks superexponential. Obviously the qualitative evidence is already an update in favor of your priors over mine.

Thane Ruthenis's Shortform

Thane Ruthenis

Thane Ruthenis3d2626

Some basic skepticism regarding Nick Cammarata's recent writeup:

The arguably-main problem of interpretability is that we currently fundamentally don't understand how to write programs that have the capabilities of trained neural networks. How do they transform raw sensory inputs into semantically meaningful abstract concepts? What are the type signatures of those concepts? What operations do they run on them? Modern LLMs work by taking advantage of some completely-unknown-to-us features of reality, features of reality about which we're philosophically conf

... (read more)

Preface to the Computational Agent Theory and ASI Alignment Sequence

Vanessa Kosoy

The idea of “artificial superintelligence ^[1] ” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI. Unfortunately, ASI poses a grave existential risk to the human species. This was argued by Eliezer Yudkowsky since 2001 ^[2] , and by many others (see e.g., Omohundro 2009, Bostrom 2014, Armstrong 2014, Russell 2019, Bengio et al. 2024 (25 authors), Bengio et al. 2025 (13 authors) and Yudkowsky and Soares 2025).

In light of this risk comes the question of how can we make ASI safe...

(See More - 358 more words)

The subset parity learning problem: much more than you wanted to know

Dmitry Vaintrob

Imagine that you’re looking for buried treasure on a large desert island, worth a billion dollars. You don’t have a map, but a mysterious hermit offers you a box with a button to help find the treasure. Each time you press the button, it will tell you either “warmer” or “colder”. But there’s a catch. With probability $2^{- 100},$ the box will tell you the truth about whether you’re closer than you were last time you pressed. But with the remaining probability of .9999999999999999999999999999992, the box will make a random guess between “warmer” and “colder”. Should you pay $1 for this box?

Keep this in mind as we discuss the closely related problem of parity learning.

In my experience of interacting with the ML and interpretability communities, the majority of people...

(Continue Reading - 3077 more words)

1Linda Linsefors4d

It seems relevant that NN does not know that it's facing the parity problem, but the undergrad does. The neural NN's hypothesis space is any function the NN can represent. What happens if we "tell" the NN that it's facing the parity problem? What does that even mean? We'd have to restrict the search space to just the parity problem hypothesis space? But that's discrete, so we can't do SGD type learning. What if we define the NN architecture to be a sum weighted of the parity functions, with one learnable weight per function? 1. Does this help with learnability? It seams like it should make the problem much easier. On the other hand it seems like the argument in section Handwavy proof of non-learnability still applies? 2. This new NN still have the same hypotes space, so it's not really telling the NN that it's facing the parity problem. What happens if we don't tell the human that they are facing the parity problem? I didn't quite understand the undergrad math solution. See my other comment, so I don't know. But I think it relies on knowing it's the parity problem? Can the NN just memorizing the training data? (probably no?) Can the NN learn a function that fits the training data but "miss generalize" outside the training data? If so, I'd argue there is nothing wrong with this solution, given the given learning problem. I think parity non-learnability would be a stronger result if it's also similarly hard for the SGD type learning, to find any solution that fit's the training data. I'm guessing this is in fact the case (i.e it can't do it) if the training data contains a large enough fraction of the total possible inputs.

Linda Linsefors4d10

Claude answered most of my questions
https://claude.ai/share/312cc600-c50d-4ed4-b384-16d7fa64db92

1Linda Linsefors5d

Wikipedias definition of Boolean Matrix is which is not XOR. Are you using some other definition of Boolean Matrix?

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Minimal Viable Safety

The Task