Vladimir_Nesov2d6-9

Minimum required pacing: finish aligning it before you train the next one

Pretraining scaling is starting to slow down, both in terms of compute and model size. The mad rush of 2025-2026 is done, and the models targeting 2027 hardware are probably already pretrained (though the work on their misalignment is then only getting started). In 2026-2028, scaling is about 2x slower than in 2025-2026, and then in 2028-2032 it gets 2x slower yet again. So "pacing" will be happening naturally in some sense. The problem is that the models of late 2026 are passing the level of human capabilities in many ways, so even slower scaling could get significantly harder to handle. And Meta, Google, and SpaceX currently have what seems to be well-made small models (somewhere between Sonnet and Opus scale-wise) and sufficient resources/ambition to plausibly catch up with OpenAI and Anthropic in 2027 at least in terms of model scale (with big models that are not so broken that it ruins their capabilities). To the extent scale is a source of misalignment difficulties that require operational excellence to sweep under the rug, they might have a hard time in 2027. And then slow-learning prosaic RSI of automated RL task/env creation in 2027-2028 probably introduces new misalignment/control issues, even if it doesn't manifest as acceleration or a step-change increase in capabilities, and merely fixes jaggedness by automatically filling in all ordinary capability gaps as they come up. Which is probably economically transformative, makes the LLM/RL paradigm more centrally AGI (by automating learning, in a very clunky way), enables useful humanoid robotics and opens up a path to a robot-industrial explosion, but isn't necessarily a raw capability jump per se. (Continual learning in a strong sense is another matter and probably quickly leads to strong architecture-rewriting RSI, but I don't know of a way to time its invention, other than to say that LLMs of 2027-2028 will be taking a stab at it. Like with math problems currently being slaughtered in droves, it could be "easy enough" in some sense for the LLM/RL paradigm to defeat quickly, but not necessarily.)

Richard_Ngo10d74

Schelling Goodness, and Shared Morality as a Goal

I think this post gestures at something important, but is quite confused in doing so. In particular, I agree that the concept of Schelling points is very important for understanding coordination. However, I think this post fails to distinguish between coordination towards good and coordination towards evil. The thing it calls "Schelling goodness" is often a tool for turning people against their own moral intuitions, especially in settings where people aren't able to encourage the behavior they actually want. For example, if you stopped people on the street in a dictatorship and asked what should happen to an opponent of the regime, the safe (and Schelling) answer might be "they should be executed". You might then say that you need to ask people in a setting where they feel safe to give their "true opinions". But power structures often coerce people into internalizing submission to those power structures, so it's hard to even know what their "true opinion" means. And even if you have a reliable way to get someone to think freely about what they prefer, doing so privately doesn't do much to change the Schelling answer—because the Schelling answer is anchored to public knowledge. In other words, even when individuals are able to come to the intuitively right moral answer, the application of power might cause the Schelling answer to remain stuck on something bad (and indeed, I suspect that we can think about this ability as constitutive of power). I discuss many of these dynamics in this blog post. You might hope that these dynamics change when thinking about the cosmic Schelling answer, since it's harder for distant civilizations to coerce each other. But the main reason to take the cosmic Schelling answer seriously IMO is that there are some channels for such civilizations to affect each other, and coercive interactions seem like they're probably fairly easy (e.g. simulating things with high disutility for another civ). On the meta level: I was thinking about making this comment a few days ago, after a conversation with Scott Garrabrant, but was prompted to do so by the recent announcement that SFF is revoking John Wentworth's funding. I don't know who made that call, and want to think more about it before writing publicly about it, but the intuitions in this comment play an important role in how I evaluate decisions like this.

Recent Discussion

stuhlmueller's Shortform

stuhlmueller

The subset parity learning problem: much more than you wanted to know

Dmitry Vaintrob

Imagine that you’re looking for buried treasure on a large desert island, worth a billion dollars. You don’t have a map, but a mysterious hermit offers you a box with a button to help find the treasure. Each time you press the button, it will tell you either “warmer” or “colder”. But there’s a catch. With probability $2^{- 100},$ the box will tell you the truth about whether you’re closer than you were last time you pressed. But with the remaining probability of .9999999999999999999999999999992, the box will make a random guess between “warmer” and “colder”. Should you pay $1 for this box?

Keep this in mind as we discuss the closely related problem of parity learning.

In my experience of interacting with the ML and interpretability communities, the majority of people...

(Continue Reading - 3077 more words)

Linda Linsefors5h10

M⋅v_S=b,

Wikipedias definition of Boolean Matrix is

$(AB)_{ij}=\bigcup _{k=1}^{n}(A_{ik}\cap B_{kj}).$

which is not XOR. Are you using some other definition of Boolean Matrix?

Minimum required pacing: finish aligning it before you train the next one

orthonormal

Let's say we coordinate to pace the frontier (rather than pausing outright or failing to coordinate). What should the standard of pacing be?

My proposal for a minimum requirement: a lab cannot train the next generation of models (in terms of capabilities) until they have adequately^[1] resolved all significant alignment issues in the present generation of models.

You don't launch a jet that goes at Mach 3 before you've figured out why your Mach 2 jet keeps crashing^[2]. With any other kind of dangerous technology, we wouldn't accept "shrug and move on".

This proposal may appeal to many prosaic-alignment optimists and pessimists, because they usually differ on how easy it would be to fix current alignment issues with current techniques.

Prosaic-alignment optimists might expect that a moderate coordinated slowdown would

...

(See More - 489 more words)

orthonormal1d34

A note on why I wrote this post despite thinking that we should pause rather than pace:

I frequently talk to people at labs (specifically Anthropic) who are enthusiastic about pacing but who aren't convinced by the arguments for a pause. In one such conversation, I said "well, you're not even meeting this basic standard", which the other person immediately agreed was valid.

I give at least 80% credence to the IABIED thesis, under which "pacing the frontier" is misguided and we must halt further progress under the current paradigm soon. However, I'm proposing... (read more)

6Vladimir_Nesov2d

2orthonormal2d

Can you spell out the implications of your predictions on this proposal?

3Vladimir_Nesov2d

Hardware/economics/industry will already be coincidentally pacing the scaling, starting pretty much right now, for reasons unrelated to (the risk of) AI incidents. To the extent capabilities follow from scale, pacing in other ways (using misalignment-related triggers and coordination) might be less necessary, the situation gets better compared to 2025-2026 on its own. But other sources of capabilities or novel misalignment issues will be appearing at the same time, including new AI companies transitioning into the more hazardous higher-scale regime very soon (they won't necessarily be able to fully benefit from the learnings of the AI companies that got there earlier). So there's a pause-adjacent story in the technology itself, that's not at all about deliberate pacing or coordination, but might have similar prosaic alignment/control effects (though inherely weaker ones than a strong ban/pause would).

Biology-Inspired AGI Timelines: The Trick That Never Works

Eliezer Yudkowsky

- 1988 -

Hans Moravec: Behold my book Mind Children. Within, I project that, in 2010 or thereabouts, we shall achieve strong AI. I am not calling it "Artificial General Intelligence" because this term will not be coined for another 15 years or so.

Eliezer (who is not actually on the record as saying this, because the real Eliezer is, in this scenario, 8 years old; this version of Eliezer has all the meta-heuristics of Eliezer from 2021, but none of that Eliezer's anachronistic knowledge): Really? That sounds like a very difficult prediction to make correctly, since it is about the future, which is famously hard to predict.

Imaginary Moravec: Sounds like a fully general counterargument to me.

Eliezer: Well, it is, indeed, a fully general counterargument against futurism. Successfully predicting...

(Continue Reading - 19427 more words)

orthonormal2d40

You assume that the architecture you're describing can, in fact, work at all to produce human intelligence. This itself strikes me as not only tentative but probably false. I mostly suspect that if you take the exact GPT architecture, scale it up to what you calculate as human-sized, and start training it using current gradient descent techniques... what mostly happens is that it saturates and asymptotes its loss function at not very far beyond the GPT-3 level - say, it behaves like GPT-4 would, but not much better.

While Eliezer may have gained Bayes poi... (read more)

Vanessa Kosoy's Shortform

Vanessa Kosoy

Vanessa Kosoy3d20

The anti-Newcomb problem is a puzzle proposed by LessWrong user @drnickbone as a challenge to UDT. I rediscovered it in 2015 and gave it a name. Recently, Richard Ngo mentioned a reminiscent but actually pretty different puzzle called "Newcomb's revenge". All of those are cases in which CDT apparently outperforms FDT.

Of course, one can always construct a decision problem that diagonalizes against any given decision theory. The interesting question is defining a natural class of "fair" problems for which a given decision theory is optimal. One can then argu... (read more)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

- 1988 -