Maybe, but the schemers could optimize their research for looking-good-to-us, and it might be hard to distinguish this from actually good work

if you ask each variant to review the research of the other variants, then the schemers need to optimise their research for looking good to each variant. but the optimistic assumption is that at least one variant is an equally capable non-schemer.

I explore similar considerations here in The Case for Mixed Deployment (5 min read), key takeaways were:

1. Invest in second-best techniques (basically what you say about "developing somewhat dubious ideas for how to prevent scheming")
2. Modular systems - i.e. architectures where you can "toggle" some variable at inference-time to give you more variants, e.g. gradient routing
3. Prioritize late-stage techniques, i.e. techniques which apply late in AI lifecycle (e.g. post-training, prompting, activation steering, scaffolding) and deprioritise techniques apply early in the AI lifecycle (e.g. architecture design and pretraining). because this makes it cheaper to develop many variants

However, I think that (contra your proposal) most of the oomph comes from the AIs monitoring and cross-examining each others work, rather than running them in parallel. That is, I disagree with "If you split your compute across 20 different types of AIs, then your favorite type of AI is going to only have 5% as much compute available to it as it would've had if you concentrated your bets." This is because I think we can run things like debate between all the variants.

One upshot of this difference is that I expect coordination between labs to matter significantly: if lab A starts with 100% schemers and lab B starts with 5% schemers, then we'll elicit useful work from the AIs if the labs cross-examine each other's AI research with their own AI variants.

Some thoughts on public outreach and "Were they early because they were good or lucky?"

1. Who are the best newcomers to AI safety? I'd be interested to here anyone's takes, not just Richard's. Who has done great work (by your lights) since joining after ChatGPT?
2. Rob Miles was the high watermark of public outreach. Unfortunately he stopped making videos. I'd be far more excited by a newcomer if they were persuaded by a Rob Miles video than an 80K video -- videos like 80K's "We're Not Ready for Superintelligence"^[1] are better on legible/easy-to-measure dimensions but worse in some more important way I think.
3. I observe a suspicious amount of 'social contagion' among the pre-ChatGPT AI Safety crowd, which updates me somewhat in favour of "lucky" over "good".^[2]

1. ^{^}

2. ^{^}

   A bit anecdotal but: there are ~ a dozen people who went to our college in 2017-2020 now working full-time in AI safety, which is much higher than other colleges at the same university. I'm not saying any of us are particularly "great" -- but this suggests social contagion / information cascade, rather than "we figured this stuff out from the empty string". Maybe if you go back further (e.g. 2012-2016) there was less social contagion, and that cohort is better?

I think your remarks suggest that alignment to the level of top humans will happen by default, but not alignment to god-like superintelligence. That said, if we get aligned top-human AIs, then we can defer the rest of the alignment problem to them.

If I were sure that top-human-level AIs will be aligned by default, here's what I might work on instead:

1. Automated philosophy
2. Commitment races / safe bargaining
3. Animal suffering
4. Space governance
5. Coordination tech
6. Empowering people with good values
7. Archiving data that aligned AIs might need (e.g. cryonics)

the base model is just predicting the likely continuation of the prompt. and it's a reasonable prediction that, when an assistant is given a harmful instruction, they will refuse. this behaviour isn't surprising.

Hey Nisan. Check the following passage from Domain Theory (Samson Abramsky and Achim Jung). This might be helpful for equipping $\Delta$ with an appropriate domain structure. (You mention [JP89] yourself.)

We should also mention the various attempts to define a probabilistic version of the powerdomain construction, see [SD80, Mai85, Gra88, JP89, Jon90].

• [SD80] N. Saheb-Djahromi. CPO’s of measures for nondeterminism. Theoretical Computer Science, 12:19–37, 1980.
• [Mai85] M. Main. Free constructions of powerdomains. In A. Melton, editor, Mathematical Foundations of Programming Semantics, volume 239 of Lecture Notes in Computer Science, pages 162–183. Springer Verlag, 1985.
• [Gra88] S. Graham. Closure properties of a probabilistic powerdomain construction. In M. Main, A. Melton, M. Mislove, and D. Schmidt, editors, Mathematical Foundations of Programming Language Semantics, volume 298 of Lecture Notes in Computer Science, pages 213–233. Springer Verlag, 1988.
• [JP89] C. Jones and G. Plotkin. A probabilistic powerdomain of evaluations. In Proceedings of the 4th Annual Symposium on Logic in Computer Science, pages 186–195. IEEE Computer Society Press, 1989.
• [Jon90] C. Jones. Probabilistic Non-Determinism. PhD thesis, University of Edinburgh, Edinburgh, 1990. Also published as Technical Report No. CST63-90.

During my own incursion into agent foundations and game theory, I also bumped into this exact obstacle — namely, that there is no obvious way to equip $\Delta$ with a least-fixed-point constructor ${\text{Fix}}_X^{\Delta }:(X\to \Delta X)\to \Delta X$. In contrast, we can equip $P$ with a LFP constructor ${\text{Fix}}_X^P:(X\to PX)\to PX,g\mapsto \{x\in X:x\in g(x\left)\right\}$.

One trick is to define ${\text{Fix}}_X^{\Delta }\left(g\right)$ to be the distribution $\pi \in \Delta X$ which maximises the entropy $H\left(\pi \right)$ subject to the constraint $g^{\Delta }\left(\pi \right)=\pi$.

• A maximum entropy distribution ${\pi }^{\ast }$ exists, because — 
  • For $g:X\to \Delta X$, let $g^{\Delta }:\Delta X\to \Delta X$ be the lift via the $\Delta$ monad, and let $G=\{\pi \in \Delta X|g^{\Delta }(\pi )=\pi \}$ be the set of fixed points of $g^{\Delta }$.
  • $\Delta X$ is Hausdorff and compact, and $g^{\Delta }:\Delta X\to \Delta X$ is continuous, so $G=\{\pi \in \Delta X:\pi =g^{\Delta }(\pi \left)\right\}$ is compact.
  • $H:\Delta X\to R$ is continuous, and $G\subseteq \Delta X$ is compact, so $H$ obtains a maximum ${\pi }^{\ast }$ in $G$.
• Moreover, ${\pi }^{\ast }$ must be unique, because — 
  •  $G$ is a convex set, i.e. if ${\pi }_1=g^{\Delta }\left({\pi }_1\right)$ and ${\pi }_2=g^{\Delta }\left({\pi }_2\right)$ then ${\lambda }_1{\pi }_1+{\lambda }_2{\pi }_2=g^{\Delta }({\lambda }_1{\pi }_1+{\lambda }_2{\pi }_2)$ for all ${\lambda }_1+{\lambda }_2=1$.
  • $H:\Delta X\to R$ is strictly concave, i.e. $H({\lambda }_1{\pi }_1+{\lambda }_2{\pi }_2)\ge {\lambda }_{1}H\left({\pi }_1\right)+{\lambda }_{2}H\left({\pi }_2\right)$ for all ${\lambda }_1+{\lambda }_2=1$, and moreover the inequality is strict if ${\pi }_1\ne {\pi }_2$ and ${\lambda }_1,{\lambda }_2>0$.
  • Hence if ${\pi }_1^{\ast },{\pi }_2^{\ast }\in G$ both obtain the maximum entropy, then  ${\pi }_1^{\ast }\ne {\pi }_2^{\ast }⟹H(0.5{\pi }_1+0.5{\pi }_2)>0.5H\left({\pi }_1\right)+0.5H\left({\pi }_2\right)$, a contradiction.

 The justification here is the Principle of Maximum Entropy:

 Given a set of constraints on a probability distribution, then the “best” distribution that fits the data will be the one of maximum entropy.

More generally, we should define ${\text{Fix}}_X^{\Delta }\left(g\right)$ to be the distribution $\pi \in \Delta X$ which minimises cross-entropy $D_{KL}\left(\pi ||{\pi }_0\right)$ subject to the constraint $\pi =g\left(\pi \right)$, where ${\pi }_0$ is some uninformed prior such as Solomonoff. The previous result is a special case by considering ${\pi }_0$ to be the uniform prior. The proof generalises by noting that $D_{KL}(-||{\pi }_0):\Delta X\to R$ is continuous and strictly convex. See the Principle of Minimum Discrimination.

Ideally,  we'd like ${\text{Fix}}^P$ and ${\text{Fix}}^{\Delta }$ to "coincide" modulo the maps $\text{Supp}:\Delta X\to PX$, i.e.$\text{Supp}\left({\text{Fix}}^{\Delta }\right(g\left)\right)={\text{Fix}}^P(\text{Supp}\circ g)$ for all $g:X\to \Delta X$. Unfortunately, this isn't the case — if $g:H\mapsto 0.5\cdot |H⟩+0.5\cdot |T⟩,T\mapsto |T⟩$ then ${\text{Fix}}^P(\text{Supp}\circ g)=\{H,T\}$ but $\text{Supp}\left({\text{Fix}}^{\Delta }\right(g\left)\right)=\left\{T\right\}$.

Alternatively, we could consider the convex sets of distributions over $X$.

Let $C\left(X\right)$ denote the set of convex sets of distributions over $X$. There is an ordering $\le$ on $C\left(X\right)$ where $A\le B⟺A\supseteq B$. We have a LFP operator ${\text{Fix}}_X^C:(X\to CX)\to CX$ via $g\mapsto \bigcup \{S\in CX:g_X^C(S)=S\}$ where $g^C:CX\to CX,S\mapsto \{{\sum }_{i=1}^n{\alpha }_i\cdot {\pi }_i|{\pi }_i\in g(x_i),{\sum }_{i=1}^n{\alpha }_i|x_i⟩\in S\}$ is the lift of $g:X\to CX$ via the $C$ monad.

However, if it is the case that the difference between humans and monkeys is mostly due to a one-shot discrete difference (ie language), then this cannot necessarily be repeated to get a similar gain in intelligence a second time.

Perhaps language is a zero-one, i.e. language renders a mind "cognitively complete" in the sense that the mind can represent anything about the external world, and make any inferences using those representations. But intelligence is not thereby zero-one because intelligence depends on continuous variables like computional speed, memory, etc.

More concretely, I am sceptic that "we end up with AI geniuses, but not AI gods", because running a genius at 10,000x speed, parallelised over 10,000x cores, with instantaneous access to the internet does (I think) make an AI god. A difference is quantity is a difference in kind.

Thar said, there might exist plausible threat models which require an AI which doesn't spatiotemporally decompose into less smart AIs. Could you sketch one out?

In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases;

I'm not sure how well this metric tracks what people care about — performance on particular downstream tasks (e.g. passing a law exam, writing bugless code, automating alignment research, etc)

Yep, you're correct. The original argument in the Waluigi mega-post was sloppy.

• If $\mu$ updated the amplitudes in a perfectly bayesian way and the context window was infinite, then the amplitudes of each premise must be a martingale. But the finite context breaks this.
  
  
• Here is a toy model which shows how the finite context window leads to Waluigi Effect. Basically, the finite context window biases the Dynamic LLM towards premises which can be evidenced by short strings (e.g. waluigi), and biases away from premises which can't be evidenced by short strings (e.g. luigis).
• Regarding your other comment, a long context window doesn't mean that the waluigis won't appear quickly. Even with an infinite context window, the waluigi might appear immediately. The assumption that the context window is short/finite is only necessary to establish that the waluigi is an absorbing state but luigi isn't.

You're correct. The finite context window biases the dynamics towards simulacra which can be evidenced by short prompts, i.e. biases away from luigis and towards waluigis.

But let me be more pedantic and less dramatic than I was in the article — the waluigi transitions aren't inevitable. The waluigi are approximately-absorbing classes in the Markov chain, but there are other approximately-absorbing classes which the luigi can fall into. For example, endlessly cycling through the same word (mode-collapse) is also an approximately-absorbing class.