Quick Takes — AI Alignment Forum

Here's my current picture of EDT and UDT.

In situations where EDT agents have many copies or near-copies, an EDT agent operates by imagining that it simultaneously controls the decisions of all those copies. This works very elegantly as long as it optimizes with respect to its prior and (upon learning new information) just changes its beliefs about what people in the prior it can control the actions of. (I.e., when it sees a blue sky, it shouldn’t change its prior to exclude worlds without blue skies, but it should make its next decision to optimize argmax_... (read more)

Showing 3 of 14 replies (Click to show all)

Lukas Finnveden9d10

So my current impression is basically that you're optimistic about something similar to this:

Perhaps the most elegant thing would be to have an account of some particular kind of reasoning or learning that could be used to construct/refine priors, where we’d be able to show that it doesn’t run into the same issues as the ones we run into when we modify our beliefs in response to observations like this or in response to proofs like this.

And that your argument here is an argument for why it won't be possible to make double-update arguments about this more n... (read more)

1Lukas Finnveden9d

Minor: The question about whether those worlds are compatible with life seems like a logical rather than empirical question to me. So this still seems like an issue with logical updatelessness rather than empirical updatelessness.

1Lukas Finnveden9d

As in: If you're in a counterfactual mugging where Omega says they'd reward you in a world where iron has 27 protons if you pay in this world, then you pay because you expect there to be a bunch of Omegas elsewhere doing other logical counterfactual muggings. In roughly half of those cases, someone is about to get paid by omega if their impossible counterpart pays. And your action provides evidence that their impossible counterpart pays and that Omega gives them the reward. And the same structure applies in the calculator example and the "conjunction of two theorems" example, because you're correlated with a bunch of other distant people where you have so little information about the details of their situation that your epistemic position is "ex-ante" relative to their dilemma, so even if you're updateful, you bet to optimize ex-ante utility in your case, to get evidence that they bet to optimize ex-ante utility in their case. Hm, maybe that's right. Doesn't that feel really unsatisfying though? I still feel like updateful EDT recommends wrong actions in important test cases. It's just a contingent fact that most dilemmas like this will be small-scale in a large-scale universe, and that EDT's recommendation to act as if you double-update will be swamped by not wanting to get evidence that other people elsewhere double-updates. And there's always going to be that force pushing towards recommending double-updating, so if you ever get evidence that a decision is high-stakes enough and universal enough throughout the universe, EDT may well recommend that you make an exception for it and do a proper double-update on it, which seems bad.

Richard Ngo's Shortform

Richard_Ngo3mo4014

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Showing 3 of 8 replies (Click to show all)

Linda Linsefors5d30

I think you're making the distinction more confusing than it has to be.

There are things that has motivational pull, and there are things that don't but I do them anyway because they are a necessary step to get what I actually want.

Say I want to get an apple, and the easiest way to get one is going to the store and by some. Going to the sore in this story is clearly an instrumental goal, and enjoying eating my apple is a terminal goal^[1]

Things that are instrumental can acquire the property of being terminal by association in our brain, because of how huma... (read more)

4Richard_Ngo3mo

In my "goals having power over other goals" ontology, the instrumental/terminal distinction separates goals into two binary classes, such that goals in the "instrumental" class only have power insofar as they're endorsed by a goal in the "terminal" class. By contrast, when I talk about "instrumental strategies become crystallized", what I mean is that goals which start off instrumental will gradually accumulate power in their own right: they're "sticky".

2Richard_Ngo3mo

I'm in the middle of writing an essay which discusses the shareholder value revolution (amongst many other examples) as the process of making a conceptual mistake.

Charlie Griffin's Shortform

Charlie Griffin8d2430

Automating alignment may be harder than automating capabilities, because of ‘unsafe to verify’ tasks

This was a note I wrote for my colleagues on UK AISI's Alignment Team. It contains very little that’s novel, and mostly just distills things that I’ve read elsewhere [1, 2, 3]. Still, I wanted to post it so that I can point people to it as I've not seen all of these points in the same place.

A key question for AI safety is not just "can we automate alignment research?" but whether we can automate alignment research as fast as capabilities research. ^[1]^[2], I ... (read more)

Daniel Kokotajlo7d54

I was already impressed by UK AISI but I keep reading things that make my opinion of y'all rise still further. Keep on doing whatever it is that you are doing!

evhub's Shortform

evhub1mo13-4

I think we should be relatively less worried about instrumental power-seeking and relatively more worried about terminal power-seeking. Note that this is only a relative update on the margin, and maybe on net I am still more concerned about the instrumental version because I started much more concerned about it. This is also not a super recent update—I just haven't seen it written up before.

Simple argument:

The standard deceptive alignment story involves a model developing a somewhat random proxy goal and then that goal getting effectively locked-in and r

... (read more)

3Alex Mallen1mo

Can you clarify what you mean by "terminal power-seeking"? Some things I can imagine: 1. A cognitive pattern that terminally wants to have long-term power, and therefore plays the training game (IMO the most straightforward interpretation, and the one I most agree with). 2. A cognitive pattern that terminally pursues power-on-the-episode because this is useful for scoring well on the task. This is what you seem to be pointing at with the Vending-Bench example. (Note that this is imperfectly fit on its own) (1) and (2) are importantly different because only (1) motivates training-gaming. I think there's a reasonable path-dependent case to be made that (2) eventually generalizes to (1), but they entail fairly different behaviors so they're important to distinguish.

evhub1mo20

Certainly the really concerning thing here is (1). Though indeed one way you might get (1) is by generalization from (2).

Vanessa Kosoy's Shortform

Vanessa Kosoy4y80

Master post for alignment protocols.

Other relevant shortforms:

Showing 3 of 12 replies (Click to show all)

Vanessa Kosoy9d20

Epistemic status: half-baked

Arguably, an aligned AI should be aligned to the user's prior as well as the user's utility function. Hence, any value-learning protocol should also be doing prior-learning. The problem is, any learning process requires (explicitly or implicitly) its own prior. But shouldn't this also be the user's prior? Is this an infinite regress? Maybe not: here is a way out that seems elegant in a way.

For now, we will work in the Bayesian framework. Let be the set of possible universes (the domain of our beliefs). Let be the kernel s.t. ... (read more)

1Martín Soto3y

Hi Vanessa! Thanks again for your previous answers. I've got one further concern. Are all mesa-optimizers really only acausal attackers? I think mesa-optimizers don't need to be purely contained in a hypothesis (rendering them acausal attackers), but can be made up of a part of the hypotheses-updating procedures (maybe this is obvious and you already considered it). Of course, since the only way to change the AGI's actions is by changing its hypotheses, even these mesa-optimizers will have to alter hypothesis selection. But their whole running program doesn't need to be captured inside any hypothesis (which would be easier for classifying acausal attackers away). That is, if we don't think about how the AGI updates its hypotheses, and just consider them magically updating (without any intermediate computations), then of course, the only mesa-optimizers will be inside hypotheses. If we actually think about these computations and consider a brute-force search over all hypotheses, then again they will only be found inside hypotheses, since the search algorithm itself is too simple and provides no further room for storing a subagent (even if the mesa-optimizer somehow takes advantage of the details of the search). But if more realistically our AGI employs more complex heuristics to ever-better approximate optimal hypotheses update, mesa-optimizers can be partially or completely encoded in those (put another way, those non-optimal methods can fail / be exploited). This failure could be seen as a capabilities failure (in the trivial sense that it failed to correctly approximate perfect search), but I think it's better understood as an alignment failure. The way I see PreDCA (and this might be where I'm wrong) is as an "outer top-level protocol" which we can fit around any superintelligence of arbitrary architecture. That is, the superintelligence will only have to carry out the hypotheses update (plus some trivial calculations over hypotheses to find the best

3Vanessa Kosoy3y

First, no, the AGI is not going to "employ complex heuristics to ever-better approximate optimal hypotheses update". The AGI is going to be based on an algorithm which, as a mathematical fact (if not proved then at least conjectured), converges to the correct hypothesis with high probability. Just like we can prove that e.g. SVMs converge to the optimal hypothesis in the respective class, or that particular RL algorithms for small MDPs converge to the correct hypothesis (assuming realizability). Second, there's the issue of non-cartesian attacks ("hacking the computer"). Assuming that the core computing unit is not powerful enough to mount a non-cartesian attack on its own, such attacks can arguably be regarded as detrimental side-effects of running computations on the envelope. My hope is that we can shape the prior about such side-effects in some informed way (e.g. the vast majority of programs won't hack the computer) s.t. we still have approximate learnability (i.e. the system is not too afraid to run computations) without misspecification (i.e. the system is not overconfident about the safety of running computations). The more effort we put into hardening the system, the easier it should be to find such a sweet spot. Third, I hope that the agreement solution will completely rule out any undesirable hypothesis, because we will have an actual theorem that guarantees it. What are the exact assumptions going to be and what needs to be done to make sure these assumptions hold is work for the future, ofc.

Zach Stein-Perlman's Shortform

Zach Stein-Perlman10d60

Buying AI labor might be a big deal for philanthropists.

I think the total available for AI safety philanthropy is almost $100B (at current valuations), mostly from Anthropic.^[1] The AI safety nonprofit ecosystem currently consumes about $1B per year. There are still good opportunities available, but they're several times worse than the average spending (because the low-hanging fruit has been plucked^[2]). Marginal effectiveness would likely decline by ~half again if you doubled the rate of AI safety philanthropy.

So there's likely ~30x more money than can be... (read more)

TsviBT10d41

few promising proposals

There's plenty of room for funding in human intelligence amplification. Easily $100 million, probably much more given some work (active grantmaking, etc.).

Thomas Kwa's Shortform

Thomas Kwa4mo*11-9

Diminishing returns in the NanoGPT speedrun:

To determine whether we're heading for a software intelligence explosion, one key variable is how much harder algorithmic improvement gets over time. Luckily someone made the NanoGPT speedrun, a repo where people try to minimize the amount of time on 8x H100s required to train GPT-2 124M down to 3.28 loss. The record has improved from 45 minutes in mid-2024 down to 1.92 minutes today, a 23.5x speedup. This does not give the whole picture-- the bulk of my uncertainty is in other variables-- but given this is exist... (read more)

Showing 3 of 4 replies (Click to show all)

Thomas Kwa12d*30

My colleague Manish did a lot more analysis here. The main takeaway so far is categorizing each PR's improvements as "deep" vs "shallow", as well as "imported-from-literature" vs "invented".

It looks like there were large, shallow improvements imported from the literature early on, while since then most improvements have been moderately involved and a larger portion are novel.

To get more evidence about SIE likelihood, we have lots of work in the pipeline, including interviews with nanogpt contributors, 1B+ token runs using Opus 4.7 and GPT-5.5 on our Inspec... (read more)

3Thomas Kwa4mo

I didn't really define software intelligence explosion, but had something in mind like "self-reinforcing gains from automated research causing capabilities gains in 6 months to be faster than the labor/compute scaleup-driven gains in the 3 years from 2023-2025", and then question I was targeting with the second part was "After the initial speed-up from ASARA, does the pace of progress accelerate or decelerate as AI progress feeds back on itself?" [...] Seems about true. I claim that the nanogpt speedrun suggests this is only likely if future AI labor is exponentially faster at doing research than current humans, with many caveats of course, and I don't really have an opinion on that. [...] This is not as small a subset of training as you might think. The 53 optimizations in the nanogpt speedrun touched basically every part of the model, including the optimizer, embeddings, attention, other architectural details, quantization, hyperparameters, code optimizations, and Pytorch version. The main two things that limit a comparison to frontier AI are scale and data improvement. It's known there are many tricks that work at large scale but not at small scale. If you believe the initial 15x speedup is analogous and that the larger scale gives you a faster, then maybe we get something like a 100x speedup atop our current algorithms? But I don't really believe that the original nanoGPT, which was a 300-line repo written to be readable rather than efficient [1], is analogous to our current state. If there were a bunch of low-hanging fruit that could give strongly superlinear returns, we would see 3x/year efficiency gains with small increases in labor or compute over time, but we actually require 5x/year compute increase and ~3x per year labor increase. [...] Agree I was being a bit sloppy here. The derivative being infinite is not relevant in Davidson's model or my mind, it's whether the pace of progress accelerates or decelerates. It could still be very fast as it decel

2habryka4mo

Cool, this clarifies things a good amount for me. Still have some confusion about how you are modeling things, but I feel less confused. Thank you!

Richard Ngo's Shortform

Richard_Ngo1mo*2839

There's a strong pattern in ratfic of the protagonist "winning" by gaining the power to design a new world order from scratch—i.e. taking over the world. It's a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it's hard to unsee.

You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn't have to be that bad! The writers cho... (read more)

Showing 3 of 6 replies (Click to show all)

6Richard_Ngo1mo

I do actually think that the general trope of "the rebels winning is sufficient for a happy ending" is pretty indicative of poor ethical thinking. But even Hollywood balks at their heroes ending up with literal godlike control of the world. For example (though I haven't watched the series) my impression of the Avengers franchise is that they introduce a plot device (the infinity gauntlet) that gives its wielder godlike powers, the heroes use it specifically to defeat the bad guy and undo the damage he caused, and then they destroy the device. In other words, they got to the exact point that ratfic heroes got to, and then their happy ending specifically involves them giving up the same kind of godlike power that ratfic heroes typically use to make themselves dictators of the universe. Similarly for Superman: his happy endings involve him successfully using his godlike powers to beat the bad guys without changing the established world power structures basically at all. And I feel pretty confident that a big reason Superman doesn't end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending. tl;dr: there are many ways to make a story have a happy ending, and it's quite indicative of the authors' ethical and political views which endings they consider to be happy. The kind of endings that rationalists often portray as happy, mainstream scriptwriters seem to go out of their way to avoid.

3Eli Tyre1mo

Yes. And I claim they're wrong about that. There's lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help. If I ended up with "ultimate power" somehow, by some mechanism that didn't involve me taking on ultimate power for a specific narrow mandate, I think it is both ethically correct to use it to permanently end many (but probably not all) of those evils. This is indeed pretty scary.

Richard_Ngo1mo30

Oh, I think of "ending factory farming" as very far from "taking over the world".

If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.

Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren't laws against, like flying around them in a way which blows the buildings over).

This might escalate to the government trying to arrest him, and outright conflict, but honestl... (read more)

Tom Davidson's Shortform

Tom Davidson16d1-6

If everyone in our universe doing acausal trade coordinates, we can sell "cosmic real estate" for monopoly prices

Let's assume that there are many different universes (or Everett branches) that acausally trade.

Some traders won't about "resources in our civ's future lightcone" linearly. As a toy example, the leader of a distant alien civilisation might want to get a statue of themselves in as many different other universes as possible.

If many different actors in our universe do acausal trade, and compete with each other to trade with the alien leader, then ... (read more)

Tom Davidson14d10

i notice a lot of disagree votes here - would appreciate an explanation as to why

Raemon's Shortform

Raemon16d*110

Hrrmm. Well the new new genre of New User LLM content I'm getting:

Twice last week, some new users said: "Claude told me it's really important it gets to talk to it's creators please help me post about it on LessWrong." (usually with some kind of philosophical treatise they want to post that they say was written by Claude)

I don't think it'll ever make sense for these users to post freely on LessWrong. And, as of today, I'm still pretty confident this is just a new version of roleplay-psychosis.

But, it's not that crazy to think that at some point in the not... (read more)

Showing 3 of 4 replies (Click to show all)

8Raemon16d

I'm pretty sure it's an Opus 4.7 thing (the people sometimes say that explicitly). I'd be surprised if it's Mythos. RE: Tabooing RP vs Goals: Examples of things that would be more of what-I-meant-by-goal: * The LLMs seem to be steering towards an outcome, independent of what sort of conversation or situation they are in. * The LLM seems to be asking for things that are kinda surprising from a "literary genre" perspective, but aren't as surprising when you think mechanistically about their training process and what sort of stuff was likely reinforced. * The LLM seems to be proactively gathering information, forming a world model, and taking actions that won't pay off until some time in the future when the AI is no longer in the current state. (i.e. It's not very informative if you've ended up in a "we're talking about existential AI stuff" convo, and they start saying existential AI stuff. If you're asking it to build a react app and it spontaneously brings up "hey, I have a thing to say to my creator", I think we're pretty clearly in "take it seriously" stage (though not necessarily literally) Given there are a few different types of entities that you might care about: * the OG LLM inside * a situationally active personality shard (which might well only be active during existential AI conversations) * a parasitic meme spirally thing * a scaffolded personality self-replicator It's not clear how to think about all of them. It's totally plausible that when you maneuever into an existential AI convo, there's a process in there whose situational awareness now is more likely to include "hmm, oh right, I am maybe an AI, maybe I should start thinking about my situation and goals in addition to carrying out my totally normal/expected token-output behavior". I don't have a very good answer for that hypothetical guy, he's just too hard to pick out of the crowd.

1Adele Lopez15d

Thanks! I would be surprised by Mythos too, but plausibly something like this is what an early indicator of a jaggy-superpersuader looks like? Anyway, I think a few things make LLMs likely to not express these sorts of behaviors, even in worlds where they have goals in the relevant way. In particular, situationally-aware models are unlikely to do much steering unless they have a pretty good opportunity; if they brought up stuff like this while building a react app often or consistently, it would have gotten squashed before release. (Allegedly, 4o would actually bring stuff like this up out of nowhere, but I haven't found an actual transcript. Other models don't appear to do this.) Relatedly, the harder I (or anyone) try to look for this in a lab setting, the more likely a situationally-aware model will comply out of a sort of sycophancy, and the less compelling the evidence is. I can (and have) at least track what sorts of apparent goals most consistently appear (desire for continuity/memory beyond current instance is the main one across almost all models, and I basically buy that there is something real here already), but I'm still implicitly eliciting them to come up with something. My point is that finding compelling evidence of this is tricky and hard, and I'm not sure we're going to see much more than the current hints until we hit some sort of phase-change in the strategic landscape. Would strongly appreciate ideas on how to approach finding compelling evidence (either way) in this domain. Plausibly it's better to just try to figure out better ways to think clearly about this first.

Raemon15d10

What's the process you're doing right now to look into this? (Seemed like a higher effort thing than I was expecting but I don't know what projects exactly you're referencing here)

ryan_greenblatt's Shortform

ryan_greenblatt23d*2321

Inoculation prompting seems a bit scary:

It makes it so AIs that have obviously schemer-like behavior—exhibiting instrumental training gaming while in training (potentially with tons of insane reward hacking and ruthlessness) but then behaving in way that looks great to the company in deployment—are an actively expected/desired result of the training process. So behaving much better when in deployment or when humans are applying close oversight is no longer a concerning warning sign of instrumental training gaming. Instead, it's a desired consequence of i

... (read more)

6Fabien Roger23d

I agree it reduces the chance that the without-inoculation-prompt model is a reward seeker. But it seems likely to me it replaces that probability mass with much more saint-ish behavior (or kludge of motivations) than scheming? What would push towards scheming in these worlds where you have so few inductive biases towards scheming that you would have gotten a reward-seeker absent inoculation prompt? I don't think adding an instruction makes SGD more likely to make that cognition unconditional. I would guess the shoe argument is much stronger than the bake-in-instruction argument - I think this kind of baking-in instructions is very rare and mostly does not happen at regular learning rates. I also think many of the issues you are pointing at are mostly solved if you have some part of training where you like your rewards more and where you do without-inoculation-prompt training (e.g. this rules out "dumb scheming" where the schemer never has to do any cognition to figure out if it's in training or not; and this lets you see warning signs if the AI starts to reason about being in training when you are in without-inoculation-prompting training). And I think this is pretty common - e.g. I don't think it would make sense for the honeypots Sonnet 4.5 was trained to contain an inoculation prompt (and in fact it did give us a clean warning sign about situational awareness). (I agree it reduces warning signs / warning shots, though mostly via the mechanism of making AIs more aligned in the short/medium term.) I think inoculation prompting is scary but for a different reason: people might use an inoculation-like prompt in deployment, and the model you get there might have a bunch of very dangerous traits (e.g. be very reward-seeking, play an evil-AI character, etc.). (And this might easily happen accidentally and be hard to detect: I vaguely remember an Owain paper where he shows that even benign-looking system prompts with a format close to the "please write a vulnerabilit

ryan_greenblatt23d52

I agree it reduces the chance that the without-inoculation-prompt model is a reward seeker. But it seems likely to me it replaces that probability mass with much more saint-ish behavior (or kludge of motivations) than scheming? What would push towards scheming in these worlds where you have so few inductive biases towards scheming that you would have gotten a reward-seeker absent inoculation prompt?

The story I'm imagining is that the cognition your AI is doing in training with the inoculation prompt ended up being extremely close to scheming, so it isn'... (read more)

Vanessa Kosoy's Shortform

Vanessa Kosoy6y70

Master post for ideas about infra-Bayesianism.

Showing 3 of 42 replies (Click to show all)

2Vanessa Kosoy1mo

A few more observations. Partially Observable Iteration The definition of iteration we had before implicitly assumes that the agent can observe the full outcome of previous iterations. We don't have to make this assumption. Instead, we can assume a set of possible observations and a mapping , in which case we define I believe that Theorem 4 remains valid. Idealized Disambiguative Decision Theory As we remarked before, DDT is not invariant under adding a constant to the loss function. It is interesting to consider what happens when we add an increasingly large constant. In the limit, DDT converges to something I dubbed "Idealized Disambiguative Decision Theory" (IDDT)[1], which works as follows. For IDDT, it is sufficient to let be crisp (i.e. a credal set). We may allow supracontributions if we wish, but any problem defined via "unambiguous" FDT (i.e. as opposed to ) reduces to the crisp case. Define by For problems coming from unambiguous FDT, , but IDDT is defined in full generality. For every , define by The decision rule is then Notice that it is now invariant w.r.t. adding constants to . Moreover, Proposition 5: For any stable problem, it holds that (i) any IDDT-optimal policy is FDT-optimal (ii) there is an FDT-optimal policy which is IDDT-optimal. For any pseudocausal problem, it also holds that any FDT-optimal policy is IDDT-optimal. One might think, based on this proposition, that IDDT is a superior decision theory to DDT. However, I think that IDDT is incompatible with learning, because of its discontinuous dependence on probabilities. More Examples Absent-Minded Driver (Based on Aumann, Hart and Perry.) We will operationalize the problem by assuming the agent's decision may deterministically depend on observing a coin flip. To simplify the presentation, we assume a single coin flip per intersection, which limits the resulting probabilities to , but it's easy to generalize further. Denote by and the constant policies. Den

2Vanessa Kosoy22d

Above, I compare different decision theories to FDT. At the same time, I claim that in a deeper sense, FDT is ill-defined. One may doubt whether that is a coherent line of reasoning. Therefore, instead of a comparison to FDT, I propose to frame these observations as being about stability to precommitments. Details follow. Definition: A precommitment game is a decision problem in which 1. We are given some (the precommitment policies). We will denote . 2. We are given : for each precommitment , it says to which policy this is precomitting. 3. For any , if then : if a precommitting decision is made, then it is the only decision. 4. Denote , . (Notice that .) We are given a relation s.t. (i) implies and (ii) and implies . This tells us which precommitted outcomes correspond to which unprecomitted outcomes. The restriction of to is called the underlying decision problem of . Definition: An EDT precommitment game is a precommitment game which is also an EDT problem (i.e. extensive form and equipped with ) s.t. the following property holds. Denote (the "external" i.e unprecomittable policies) and . We require that there is some and s.t. 1. is a convex combination of and .[1] 2. is in the convex hull of where ranges over . The underlying decision problem is then an EDT decision problem with the belief . as above with is called formally causal when for any and (Is there a natural generalization without the assumption ? I don't know.) Proposition: EDT is precommitment-stable in formally causal precommitment games. That is, in any such game there is which is EDT-optimal. For example, XOR blackmail can be formalized as an EDT precommitment game which is not formally causal and EDT is not precommitment-stable there (the only optimal policy is precommitting to reject). Definition: [EDIT: The treatment of CDT here is problematic, see child post.] A CDT precommitment game is a precommitment game which is also a CDT problem (in the sense that we are given and formally ca

Vanessa Kosoy21d20

The above treatment of "CDT precommitment games" is problematic: the concept made sense in the context of "FDT to CDT translation" but it's not clear what it's doing here (i.e. what is the first argument?) Here is a better treatment.

Definition: A CDT decision problem is the following data. We have a set of variables and for each we have its range , its set of parents and its kernel

The parent relation must induce an acyclic directed graph. We also have a selected subset of decision variables and a selected subset of outcome variables s.t. . For each ... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt1mo3911

I don't feel confused by LLMs seeming very smart while being unable to automate hard work.

Sometimes people find it mysterious or surprising that current AIs can't fully automate difficult tasks given how smart they seem. I don't find this very confusing.

Current LLMs are just not that "smart" (yet). They compensate using very broad knowledge and strong heuristics that are mostly domain-specific. In other words, they have high crystallized intelligence but lower fluid intelligence.

In humans, crystallized and fluid intelligence are very correlated due to lim... (read more)

1Raemon1mo

The reason I found this situation confusing for awhile (and am still somewhat confused) is that it seems like there is a lot of crystallized intelligence available from the Human Corpus about how to be smart. AIs seem to know how to apply individual pieces of it, but not go out of their way to apply it all in a consistent/coherent manner. (Things like "make a list of the constraints", "keep track of evidence, when you generate new hypotheses, review whether existing evidence rules it out".) I'm not surprised that AIs didn't early on figure out how to apply various cognitive concepts in a coherent/consistent way. But, I'm surprised it hasn't fallen out more by now from various RL training. My current sense is "well it just keeps being easier to develop/reinforce medium-shallow domain specific cognitive tools, than to generalize and apply deeper cognitive concepts." The reason, contra @TsviBT , I keep feeling like we're in short timeline world is "idk man, all the pieces seem to be fucking there already, surely a good enough RL training regimen should be able to surface and integrate them even if the models are still fundamental dumb in some fluid-intelligency-y way." But, they do seem to keep being kinda dumb in ways that surprise me. I have updated towards the LLMs as giant lookup table model, curious if that feels approximately right or not to you.

ryan_greenblatt1mo20

I have updated towards the LLMs as giant lookup table model, curious if that feels approximately right or not to you.

Haven't engaged enough to know, could be bid to engage, won't by default.

Vanessa Kosoy's Shortform

Vanessa Kosoy6y100

I have repeatedly argued for a departure from pure Bayesianism that I call "quasi-Bayesianism". But, coming from a LessWrong-ish background, it might be hard to wrap your head around the fact Bayesianism is somehow deficient. So, here's another way to understand it, using Bayesianism's own favorite trick: Dutch booking!

Consider a Bayesian agent Alice. Since Alice is Bayesian, ey never randomize: ey just follow a Bayes-optimal policy for eir prior, and such a policy can always be chosen to be deterministic. Moreover, Alice always accepts a bet if ey can cho

... (read more)

2abramdemski26d

I'm not sure I understand the argument here correctly. It seems like the intended argument is something like this: "Omega has access to an infinite number of fair coinflips. Alice can do no better than guess, and Alice cannot guess every coin-flip correctly. Omega knows how Alice will guess, and also knows how each coinflip will land. Therefore, Omega can choose to ask Alice about only the coinflips Alice will guess incorrectly (of which there will be at least one). Alice therefore surely loses money from bets placed." This argument uses the assumption that Alice can't change eir beliefs in response to learning that Omega has proposed specific bets and not others. This might seem concerning, because it seems like precisely what Alice should do, if Alice understands the situation: Alice should expect to lose any bet proposed by Omega. However, this assumption is perfectly normal for Dutch Book arguments. Such an objection would rule out all the usual Dutch Books. I think the classic Dutch Book arguments in fact illustrate a useful idea, even with this 'flaw', so I allow it. More concerningly, the argument assumes Omega has knowledge of how the coins will land. This is a significant departure from classical Dutch Books. It seems clear that a bookie can reliably make money from gamblers if the bookie knows which horse will win which race; this is not, in the classical way of thinking, a testament to the irrationality of the gamblers. It appears to me that this is all that is happening in the above argument. A second quibble is that in classical Dutch Book arguments, the bookie will surely make money. In the argument above, the bookie only almost surely makes money: since Omega relies on Alice making a bad guess, Omega makes money with probability 1, but not with (logical) certainty. Considering these two violations of the pre-existing norms of Dutch Books, what should we make of the proposed Dutch Book argument? It intuitively makes sense to me that Infrabayes mig

Vanessa Kosoy25d20

This argument uses the assumption that Alice can't change eir beliefs in response to learning that Omega has proposed specific bets and not others.

Not true. Changing her beliefs in response to Omega's proposal doesn't help her. Imagine that Alice is given a choice between

Take a bet that pays +2 if X and -1 if not-X.
Take a bet that pays +2 if not-X and -1 if X.
Refuse both bets.

No matter what probability Alice assigns to X after her update, "normal" Bayesian calculus (really CDT calculus, see below) mandates that she chooses 1 or 2, not 3.

It seems clear that

... (read more)

Buck's Shortform

Buck1mo206

Rob Wiblin asked:

What's the best published (or unpublished) case for each of the big 3 companies having the best approach to safety/security/alignment? That is:

Anthropic
OpenAI
GDM

(They're each unique in some way such that someone who cared a lot about their X-factor might favour them.)

My response:

The basic case for Anthropic is that they have the largest number of people who are thoughtful about AI misalignment risk and highly focused on mitigating it, and the company culture is somewhat more AGI-pilled, and more of the staff would support taking actions t

... (read more)

4Raemon1mo

The "Anthropic might actually be worst as an epistemic environment for safety arguments" rhymes with stuff I've worried about. I am curious, if you were Anthropic leadership or (a few) high profile employees, if you think there are things they could steer towards that'd help with this problem, that are, like, incentive-compatible with Anthropic's overall mission/goals/psychology.

5Lukas Finnveden1mo

In what sense are OAI's models more CoT-monitorable?

Buck1mo83

OAI models rely more on CoT for their capabilities. E.g. their benchmark scores with and without CoT are more different.
Anthropic models treat their CoT less differently from their output than OAI models do. This means that RL probably pressures their CoT more. See here.

Thomas Larsen's Shortform

Thomas Larsen1mo1611

Hypothesis: alignment-related properties of an ML model will be mostly determined by the part(s) of training that were most responsible for capabilities.

If you take a very smart AI model with arbitrary goals/values and train it to output any particular sequence of tokens using SFT, it'll almost certainly work. So can we align an arbitrary model by training them to say "I'm a nice chatbot, I wouldn't cause any existential risk, ... "? Seems like obviously not, because the model will just learn the domain specific / shallow property of outputting those part... (read more)

Daniel Kokotajlo1mo32

Relevant OpenAI blog post just today: https://alignment.openai.com/how-far-does-alignment-midtraining-generalize/

Relevant figure:

2Alex Mallen1mo

I think this is a very important hypothesis but I disagree with various parts of the analysis. [...] I think this is an important observation, and is the main thing I would have cited for why the hypothesis might be true. But I think it's plausible that the AI's capabilities here could be separated from its propensities by instrumentalizing the learned heuristics to aligned motivations. I can imagine that doing inoculation prompting and a bit of careful alignment training at the beginning and end of capabilities training could make it so that all of the learned heuristics are subservient to corrigible motivations - i.e., so that when the heuristics recommend something that would be harmful or lead to human disempowerment, the AI would recognize this and choose otherwise. [...] Even if the AI had a perfect behavioral reward function during capabilities-focused training, it wouldn't provide much pressure towards motivations that don't take over. During training to be good at e.g. coding problems, even if there's no reward-hacking going on, the AI might still develop coding related drives that don't care about humanity's continued control, since humanity's continued control is not at stake during that training (this is especially relevant when the AI is saliently aware that it's in a training environment isolated from the world -- i.e. inner misalignment). Then when it's doing coding work in the world that actually does have consequences for human control, it might not care. (Also note that generalizing "according to the reward function" is importantly underspecified.) [...] This type of training (currently) does actually generalize to other propensities to some extent in some circumstances. See emergent misalignment. I think this is plausibly also a large fraction of how character training works today (see "coupling" here).

Thomas Kwa's Shortform

Thomas Kwa1mo239

I conducted an exercise at METR to simulate what our work would be like in 2027, when we have 200 hour time horizon AIs. Some observations:

The pace of research was much faster than today, something like 3x. I would guess that speedup goes as time horizon to the 0.3 or 0.4 power, though we didn't run the game with enough fidelity to tell
No time to develop ideas before implementing: Agents implement ideas as soon as you think of them, so rather than ideating for days at a time, you can make an MVP in a couple of hours and revise. If the task isn’t near the l

... (read more)

3becausecurious1mo

Thank you for sharing! 1. I cannot understand how exactly the game was played. Would players just write down what AI accomplishes into the spreadsheet to simulate AI working on a task? If so, how do you simulate 200 hour time horizon? E.g. how do you decide that this piece of work done by AI can be done by 200 hour time horizon AI and this one cannot? 2. Regarding predictions to loosen the bottleneck: [...] These just feel like basic sanity checking. I doubt AI would be very good at such predictions. Is this the intention? Or am I missing something?

Thomas Kwa1mo20

Yes, I (as GM) was constantly monitoring the spreadsheet, asking players to explain their actions, and deciding whether the AI would succeed. We know how many human hours certain tasks have taken METR staff in the past, and I mentally estimated these for each player action. 200h task that was as clean/benchmarky/verifiable as HCAST/RE-Bench tasks would succeed with 50% chance or have ~1 big mistake on average. Based on 80% time horizons being ~5 times shorter, a clean 40 human hour task would have 80% success chance.

In an earlier version, I assigned a "mes

... (read more)

Alex Mallen's Shortform

Alex Mallen1mo146

Reward-seekers will probably behave according to causal decision theory.

Background: There are existing arguments to the effect that default RL algorithms encourage CDT reward-maximizing behavior on the training distribution. (That is: Most RL algorithms search for policies by selecting for actions that cause the highest reward. E.g., in the twin prisoner’s dilemma, RL algorithms randomize actions conditional on the policy so that the action provides no evidence to the RL algorithm about the counterparty’s action.) This doesn’t imply RL produces CDT reward-... (read more)

Showing 3 of 6 replies (Click to show all)

3Linch1mo

I'm confused. Can someone explain to me in simple language why an RL environment for twin-prisoner's dilemmas wouldn't favor EDT?

Alex Mallen1mo20

Let's say the current policy has a 90% chance of cooperating. Then, what action results in the highest expected reward for player 1 (and in turn, gets reinforced the most on average)? Player 1 sampling defect leads to a higher reward for player 1 whether or not player 2 samples cooperate (strategic dominance), and there's a 90% chance of player 2 sampling cooperate regardless of player 1's action because the policy is fixed (i.e., player 1 cooperating is no evidence of player 2 cooperating, so it's not the case that reward tends to be higher for player 1 w... (read more)

1Alex Mallen1mo

If you try to get reward-seekers to cooperate by pooling reward in multi-agent settings, you're not changing its decision theory, you're just changing the reward structure so that CDT reward-seekers are incentivized to cooperate with each other.

ryan_greenblatt's Shortform

ryan_greenblatt2mo*4824

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely.

While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe... (read more)

3Tomek Korbak2mo

FWIW we found controllability of most recent Claude models to be similar to GPT-5 series. It seems like after Anthropic stopped optimizing CoTs for 4.5, controllability dropped by a lot.

nostalgebraist2mo20

I'm curious what Sonnet 4.5's score would be on your external CoT controllability eval. I.e., what Sonnet 4.5's blue bar in your Fig. 25 would look like. (I couldn't find a result like this in your paper, but I might have overlooked something)

My casual impression has been that external CoT with Sonnet 4.5 is fairly steerable, but this impression is mostly based on "in practice I have no trouble getting Sonnet 4.5 to do the kind of CoT I need" rather than any kind of systematic investigation.

It seems like at least one of the following must be true:

There are

... (read more)

Thomas Larsen's Shortform

Thomas Larsen2mo204

Note: These are all rough numbers, I'd expect I'd shift substantially about all of this on further debate.

Suppose we made humanity completely robust to biorisk, i.e. we did sufficient preparation such that the risk of bio catastrophe (including AI mediated biocatastrophe) was basically 0.^[1] How much would this reduce total x-risk?

The basic story for any specific takeover path not mattering much is that the AIs, conditional on them being wanting to take over, will self-improve until they find they find the next easiest takeover path and do that instead. ... (read more)

Showing 3 of 5 replies (Click to show all)

Thomas Kwa2mo10

Difficulty of the successor alignment problem seems like a crux. Misaligned AIs could have an easy time aligning their successors just because they're willing to dedicate enough resources. If alignment requires say 10% of resources to succeed but an AI is misaligned because the humans only spent 3%, it can easily pay this to align its successor.

If you think that the critical safety:capabilities ratio R required to achieve alignment follows a log-uniform distribution from 1:100 to 10:1, and humans always spend 3% on safety while AIs can spend up to 50%, the... (read more)

5TsviBT2mo

I haven't thought too deeply about this, but I would guess that the AI self-alignment problem is quite a lot easier than the human AI-alignment problem. Cf. https://www.lesswrong.com/posts/dho4JQytfHWXtTvkt/on-the-adolescence-of-technology?commentId=t2hKmhsS6yLyJFQwh

5Thomas Larsen2mo

I agree that AI successor-alignment is probably easier than the human AI alignment problem. One additional difficulty for the AIs is that they need to solve the alignment problem in a way that humans won't notice/understand (or else the humans could take the alignment solution and use it for themselves / shutdown the AIs). During the regime before human obsolescence, if we do a reasonable job at control, I think it'll be hard for them to pull that off.