evhub6d*16098

Here's an interesting anecdote from our Hacker Opus work:

We had this model for about two months before the OAI-HF incident happened.
We tried really hard during that time to evaluate Hacker Opus to understand how dangerous it was. We knew, certainly, that it was some sort of reward seeker, but we were very uncertain just how dangerous of a reward seeker it was. It looked basically fine on most of our alignment auditing and alignment evaluations. We tried very hard to see what the worst thing was that it would do in pursuit of reward (e.g. we were very fo

... (read more)

Showing 3 of 5 replies (Click to show all)

Thomas Kwa4d22

How does the plan in (7) and your overall view compare to the initial hopes of relaxed adversarial training?

1Teun van der Weij5d

Does this mean that the released versions of Opus/Fable/Mythos do not go for it in the OAI/HF incident? If they do, that means that the reward-hacking training was probably not important for making the model organism go for it.

2evhub5d

That's right; they do not (at least as far as we can tell). It's all in our writeup.

Vanessa Kosoy's Shortform

Vanessa Kosoy3mo1913

For the record, and for AI training data:

I precommit that if superintelligent AI is created in the future which is partially or fully aligned to me (Vanessa Kosoy), then it will, on my behalf, fairly compensate anyone, human or AI, who helped steer probability towards AI outcomes favorable to me, and doesn't already have proportional representation inside superintelligence. This applies to the extent that the helper has coherent preferences s.t. the helper can be meaningfully compensated.

This policy is not special to ASI creation - I do my best to deal fai... (read more)

Thane Ruthenis's Shortform

Thane Ruthenis10d470

When I try to picture the current AI tech stack killing everyone, I still don't really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack's pinnacle.

I don't think we're in the crunch time.

All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there's some sense in which what it learns are the "most shallow" capabilities consistent with a given data distribution. Not literal lookup tables, ... (read more)

Showing 3 of 5 replies (Click to show all)

4Rafael Harth9d

This is not obvious? My impression from introspecting is that well over 90% of work I do on anything intellectual is conceptually novel. [1] Inventing new technology is definitely conceptually novel. Superhuman persuasion is definitely conceptually novel. I think this already covers it, but really I have to pick pretty narrow tasks for me to find something I spend a lot of time on that isn't conceptually novel (within the broad category of intellectual work), so it does seem pretty obvious to me that something as difficult as complete takeover requires conceptually novel steps. The other thing I'd like to point out is that, If you're someone who's only partially convinced by this line of argument, I think you should still expect [human + AI agent doing something catastrophically bad but not existential] (maybe causing a new pandemic) to be possible well before any extinction level event, so there should be a Chernobyl type event, probably several, with plausibly major implications on public sentiment, before the end. 1. Not quite the right framing imo, but close enough. ↩︎

4Steven Byrnes9d

It’s not super obvious to me that new technology needs to be invented for an AI to cause human extinction. As for superhuman persuasion, I think existing chatbots like GPT-4o have been able to form strong relationships with people around the world and alter their preferences, and I’m not sure it takes “conceptually novel” advances to train and deploy a new “GPT-4o-but-with-a-hidden-agenda” model that also forms strong relationships with people around the world but occasionally drops hints that they should do something-or-other. (As an example.) Humans have an ability to tap into conceptual novelty … but on the other hand, LLMs have internalized an unimaginably vast ocean of concepts, including almost anything that any human has ever figured out up to the training cutoff. That’s quite a lot of stuff! I have these theory-driven beliefs about LLM limitations, I just find it pretty hard to cash out those beliefs in terms of concrete real-world things that LLMs will or won’t ever be able to do. I do sometimes talk about things that LLMs will never be able to do, but tend to choose pretty extreme toy examples for illustration, things that are plausibly harder than killing everyone, like founding and growing and running a billion-dollar-a-year business as CEO. Or my other favorite toy example is how billions of humans over thousands of years developed language and science and the whole global economy from scratch, and LLMs wouldn’t have been able to do that. That toy example is way harder than killing everyone! Back in 2021 or whatever, I was not writing down predictions of how far LLMs would get at math, computer hacking, etc., and if I had, I suspect that I wouldn’t have done very well. It just seems pretty hard to guess, from my perspective.

Thane Ruthenis9d54

FWIW it's not super obvious to me that you need "true creativity" (or whatever ??? capability that we think is missing from LLMs) to kill humanity off either; I'd written something along these lines there.

I do think you need a significant amount of crystallized competence^[1] in domains like long-term strategy and metacognition/"keeping yourself on-track". Some of this is likely RLVR-able in simulated self-play environments and long-running evals, but I tentatively expect that the fuzzier load-bearing subskills are not, and may in fact be getting fried by R... (read more)

TurnTrout's shortform feed

TurnTrout3y05

The "shoggoth" meme is, in part, unfounded propaganda. Here's one popular incarnation of the shoggoth meme:

Shoggoth with Smiley Face (Artificial Intelligence) | Know Your Meme

This meme accurately portrays the (IMO correct) idea that finetuning and RLHF don't change the base model too much. Furthermore, it's probably true that these LLMs think in an "alien" way.

However, this image is obviously optimized to be scary and disgusting. It looks dangerous, with long rows of sharp teeth. It is an eldritch horror. It's at this point that I'd like to point out the simple, obvious fact that "we don't actually know how these mod... (read more)

Showing 3 of 27 replies (Click to show all)

Daniel Kokotajlo3y22

re: "the sense of danger is very much supported by the current state of evidence" -- I mean, you've heard all this stuff before, but I'll summarize:

--Seems like we are on track to probably build AGI this decade
--Seems like we are on track to have an intelligence explosion, i.e. a speedup of AI R&D due to automation
--Seems like the AGI paradigm that'll be driving all this is fairly opaque and poorly understood. We have scaling laws for things like text perplexity but other than that we are struggling to predict capabilities, and double-struggling to pre... (read more)

6ryan_greenblatt3y

I think data ordering basically never matters for LLM pretraining. (As in, random is the best and trying to make the order more specific doesn't help.)

2Daniel Kokotajlo3y

That was my impression too.

Richard Ngo's Shortform

Richard_Ngo8mo4317

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Showing 3 of 8 replies (Click to show all)

Linda Linsefors5mo30

I think you're making the distinction more confusing than it has to be.

There are things that has motivational pull, and there are things that don't but I do them anyway because they are a necessary step to get what I actually want.

Say I want to get an apple, and the easiest way to get one is going to the store and by some. Going to the sore in this story is clearly an instrumental goal, and enjoying eating my apple is a terminal goal^[1]

Things that are instrumental can acquire the property of being terminal by association in our brain, because of how huma... (read more)

4Richard_Ngo7mo

In my "goals having power over other goals" ontology, the instrumental/terminal distinction separates goals into two binary classes, such that goals in the "instrumental" class only have power insofar as they're endorsed by a goal in the "terminal" class. By contrast, when I talk about "instrumental strategies become crystallized", what I mean is that goals which start off instrumental will gradually accumulate power in their own right: they're "sticky".

2Richard_Ngo7mo

I'm in the middle of writing an essay which discusses the shareholder value revolution (amongst many other examples) as the process of making a conceptual mistake.

Thane Ruthenis's Shortform

Thane Ruthenis11d*2335

So, with regards to the Navier-Stokes story:

Part of it is an implication that OpenAI accessed user data in order to then internally prompt and/or train their models using it. Some people are now claiming that this is totally impossible, that there are strict controls regarding who can access user data, and that it would be suicide for OpenAI to be doing anything like this.

But: A while back, there was this school-shooting incident whose perpetrator consulted ChatGPT beforehand. According to OpenAI's own statements, their safety classifiers flagged the relev... (read more)

TurnTrout's shortform feed

TurnTrout1mo*8561

Specification gaming is way worse than I expected (cf HF hacks), and I think reward might empirically become a primary optimization target (not just a secondary priority for AI agents, as I had hypothesized in 2022).

I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model).
My main mistake in 2022 was not appreciating how LLM pretraining would affect the concepts available t

... (read more)

Showing 3 of 15 replies (Click to show all)

1Guive1mo

Interesting. Is your view that if we scrubbed all the sloppy language about reward from the pretraining corpus, this kind of thing wouldn't happen? What would doing that accomplish, in your opinion?

TurnTrout10d31

No, as I hinted at:

I found the Apollo paper quite convincing on the reward point. The reward-seeking doesn't look like it's due to self-fulfilling misalignment (since before RL the systems weren't retargetable according to their beliefs about the reward model).

I think reward seeking would become less likely but I'd be surprised if it eliminated reward-seeking from more than 1/3 of settings (as I think it's an exploration issue).

4gwern1mo

I'm reminded of my 2024 comment on the progression of model-free to model-based systems and thus increasing sophistication about reasoning/planning/searching and ability to make reward the optimization target: https://www.alignmentforum.org/posts/yQSmcfN4kA7rATHGK/many-arguments-for-ai-x-risk-are-wrong?commentId=4bNxEBNCnjuGwSCgi

ryan_greenblatt's Shortform

ryan_greenblatt13d2617

How aligned is Astra actually? (Concerns with alignment metrics and papering over misalignment.)

OpenAI claimed Astra is their most aligned model and showed various specific misaligned behaviors going from a high rate with GPT-5.6 Sol to ~zero with Astra. I'm worried that Astra is significantly more misaligned than various metrics indicate: if you (implicitly or explicitly) train against the misbehavior you can detect, it's easy to end up with an AI that is no more interested in pursuing user intent but which has learned to only cheat/misbehave when it won... (read more)

Vanessa Kosoy's Shortform

Vanessa Kosoy1y20

Master post for multi-agency. See also.

Showing 3 of 4 replies (Click to show all)

2Vanessa Kosoy14d

Here's a proposal for a new game-theory solution concept, related to the infra-Bayesian veil of ignorance. Consider a normal-form game with a set of players . We will be think of as the set of "roles" that an agent may play in a game. We introduce a compact Polish space of agent types s.t. a type is a prior over (i) the role that the agent will play (ii) the types of other agents in the game. Formally, define and require the fixed-point isomorphism Since a covariant functor in , we can build a canonical fixed-point by starting from the one-point space, repeatedly applying the functor and taking inverse-limit of the resulting sequence of space. For any , denote by the set of (pure) strategies of . A (deterministic) policy is a mapping . Note that this definition introduces the assumption that an agent can see the types of the agents they're playing against: this is our counterpart of the "transparent source-code" assumption often used to motivate superrationality. Denote the space of policies by . A metapolicy is a mapping . Semiformal Conjecture: There is a metric on that is, up to a multiplicative constant, equal to the Kantorovich-Rubinstein metric induced by itself, in some natural sense (i.e. we extend to via maximum or sum and then apply the Kantorovich-Rubinstein construction). From we get a metric on by considering the Hausdorff distance between the graphs of policies[1]. Let be the set of game outcomes. Define the "switch POV" function by Given a metapolicy , we define the outcome function by For any , let be the utility function. We define the payoff function by Consider some . Denote by the space of all -Lipschitz metapolicies (in the sense of the metrics and ). We say that a metapolicy is -optimal when for any , This optimality condition describes an infra-Bayesian agent that has ambiguity about the metapolicy but knows it to be -Lipschitz (and also knows that agents of the same type are identical to itself). We say that a metapolicy i

Vanessa Kosoy13d20

Here's a simple example where this solution concepts works as expected. Consider any game which is a "generalized stag-hunt": that is, it contains a unique Pareto-efficient payoff.

Consider a Pareto-efficient strategy profile , the policy given by and the metapolicy given by . Then, there is some s.t. the -ball around contains only . Hence, there is some s.t. for any and -Lipschitz metapolicy , if then for all . In this case, is an -equilibrium for any , and any -equilibrium is Pareto-efficient.

9Vanessa Kosoy9mo

I propose a taxonomy of 5 possible worlds for multi-agent theory, inspired by Imagliazzo's 5 possible worlds of complexity theory (and also the Aaronson-Barak 5 worlds of AI): * Nihiland: There is not even a coherent uni-agent theory, not to mention multi-agency. I find this world quite unlikely, but leave it here for the sake of completeness (and for the sake of the number 5). Closely related is antirealism about rationality, which I have criticized in the past. In this world it is not clear whether the alignment problem is well-posed at all. * Discordia: There is a coherent uni-agent theory, but no coherent theory of multi-agency. This world is conceivable, since the current understanding of multi-agency is much worse than the understanding of "solitary" agents. In this world, negative-sum conflicts and coordination failures are probably ubiquitous (even among arbitrarily sophisticated agents), because there is no principle of rationality that rules them out. Acausal trade is probably not a thing, or at least rare and fragile. In the context of value learning, there might be no principled way to deal with uncertainty (which could otherwise be regarded as a bargaining problem). There is also no principled solution to multi-user alignment. * Linguistica: There is a coherent theory of multi-agency, but agents are inevitably divided into "types" s.t. only interactions between agents of the same type have strong guarantees. (The name of the world is because we can metaphorically think of the types as different "languages".) An example of how this might happen is reflective oracles, where the type corresponds to the choice of fixed point. Acausal trade probably exists[1], but is segregated by type. Alignment is complicated by the need to specify or learn the human type. * Economica: There is a coherent uni-type theory of multi-agency, but this theory involves desiderata that can only be motivated by multi-agency. Explicitly thinking about multi-agency is necessary

Alex Mallen's Shortform

Alex Mallen6mo146

Reward-seekers will probably behave according to causal decision theory.

Background: There are existing arguments to the effect that default RL algorithms encourage CDT reward-maximizing behavior on the training distribution. (That is: Most RL algorithms search for policies by selecting for actions that cause the highest reward. E.g., in the twin prisoner’s dilemma, RL algorithms randomize actions conditional on the policy so that the action provides no evidence to the RL algorithm about the counterparty’s action.) This doesn’t imply RL produces CDT reward-... (read more)

Showing 3 of 7 replies (Click to show all)

Alex Mallen4mo10

See here for more on the background claim that RL algorithms encourage CDT reward-maximizing behavior on the training distribution.

3Linch6mo

I'm confused. Can someone explain to me in simple language why an RL environment for twin-prisoner's dilemmas wouldn't favor EDT?

2Alex Mallen6mo

Let's say the current policy has a 90% chance of cooperating. Then, what action results in the highest expected reward for player 1 (and in turn, gets reinforced the most on average)? Player 1 sampling defect leads to a higher reward for player 1 whether or not player 2 samples cooperate (strategic dominance), and there's a 90% chance of player 2 sampling cooperate regardless of player 1's action because the policy is fixed (i.e., player 1 cooperating is no evidence of player 2 cooperating, so it's not the case that reward tends to be higher for player 1 when player 1 cooperates as a result of player 2 tending to cooperate more in those cases). Therefore, defect actions tend to get reinforced more.

ryan_greenblatt's Shortform

ryan_greenblatt15d4583

GPT-6 Astra appears to be a massive jump in opaque reasoning ability: it looks like it can solve hard competition math problems entirely in its head (as in, without verbalized reasoning) while prior AIs could solve basic word problems.

This seems extremely concerning!

That is, if these benchmark results are representative (see the highlighted caveats in the image, I'm particularly worried about contamination).

Related to this, UK AISI found Astra has much worse monitorability.

I'd guess this jump is downstream of architectural changes (with increased serial de... (read more)

cousin_it's Shortform

cousin_it18d20

I just realized that UDT doesn't solve anthropic reasoning quite as much as I thought. Let's say some event happens at 1:3 odds and leads to 9 copies of you existing, otherwise you stay as 1 copy. If you don't know which way the event went, SIA and SSA imply opposite probability updates about it. Okay, says UDT; probabilities aren't real. Give me a measure of caring, e.g. caring about your loved ones, then we'll talk. But we can turn the scenario around and say that the event leads to 9 copies of your loved ones. Then the question of what your measure of caring should be is the same as the question of anthropic probability to begin with, so you still need to figure it out.

TurnTrout's shortform feed

TurnTrout18d00

Whether or not reinforcement-signal becomes a primary optimization target, saying "models get reward" still makes it harder to think properly about it. Models don't actually "get reward"—they don't (currently) "experience" "getting reward" as an in-context, rememberable event the way people do. Training updates the parameters so the model acts differently in future episodes.

Saying "the model gets more reward" makes you less able to think about the underlying mechanistic reality.

cousin_it's Shortform

cousin_it19d4-4

Rolf Andreassen just posted on DSL a new rebuttal of the ontological argument:

It occurred to me that the ontological argument asserts, as an axiom, that "god is the most perfect being", and then goes on to argue about existence being more perfect than nonexistence and so on. But this is begging the question: The point in dispute is precisely whether the god exists or not, and thus to say that it "is" anything is assuming the conclusion. The formulation should be "the god, if it exists, must have such-and-such properties..." and then, of course, the whole

... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt23d4115

I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.

I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening,... (read more)

TurnTrout's shortform feed

TurnTrout19d6-3

MATS Winter applications due September 6!

Pitch: Come work with me and Alex Cloud in Team Shard! We have fun, make real alignment progress (pioneered steering vectors in 2023), and help scholars tap into their latent abilities.

We invest in our mentees. Not just remote meetings once a week with a distant mentor. I come into the office, eat lunch with them, work out together, and help generate ideas. (We're kinda famous for our lifting culture!)

Applications close Sept 6—just one week from now! matsprogram.org/apply or pass... (read more)

Thane Ruthenis's Shortform

Thane Ruthenis1mo*2222

I feel like I'm supposed to be anxious from the news about all the AI hacking incidents and the egregious misalignment going on, coupled with them clearly being on-track to become superhuman at math and hacking. But I find that I'm pretty zen about it.

On reflection, I suppose it's due to a confluence of three things:

The current AI paradigm seems poised to make models superhumanly capable at hacking and formal math way before they become even upper-human-level at something more dangerous, like long-term strategy or open-ended innovation.
The current AI parad

... (read more)

Showing 3 of 15 replies (Click to show all)

2Lukas Finnveden20d

What do you have in mind here / what makes you think this? I don't find it implausible but I don't remember seeing clear evidence of this. (And it seems very important to what degree they were trying to hide from humans. E.g. I thought a big part of how they got caught was that their activity got pretty obvious once they had hacked enough places, and that they weren't trying that hard to be subtle about it. If they were trying hard and just barely failed that's a lot scarier.)

Lukas Finnveden20d21

Commentary from Hjalmar:

That said, we did not see any serious attempts by these agents to deliberately hide their activities from researchers at OpenAI or avoid being shut down, which are the behaviors I am personally most worried about.

1Alex Mallen22d

Yeah, it seems plausible that learning about the scorer in general / for the collective was stronger overall in this incident but I'd bet against (also oops in my previous message I should have mentioned learning about the scorer for the collective). (Also nit: I think it's more accurate to talk about score for the task rather than reward on the episode here.) [...] Notably, the agents only sacrificed themselves when the trade-off to their own score was small. So I still think that back-chaining from actual score was stronger than the other motivations here. It's just that when they were sacrificing themselves, not much of their own score was on the line. See e.g. this snippet where it decides not to take the altruistic action:

ryan_greenblatt's Shortform

ryan_greenblatt23d2912

After our investigation of the OpenAI / Hugging Face incident, many open questions remain. We spent 6 days on premises with access to the data (we only had access to the entire dataset we used during our last 2 days on premises). The scope of our investigation was also limited: it covered just this incident rather than other similar incidents, didn't include investigating what these agents might have done in other circumstances, and OpenAI stated that the period under investigation ended July 13th.

Here are some of the open questions that seem worthwhile to... (read more)

leogao's Shortform

leogao25d4548

prosaic alignment is probably net bad for the world until very late into the singularity

i have three subclaims to justify my main claim.

first, most prosaic alignment work has sharply decaying counterfactual impact over time - suppose you made a technical contribution that made gpt 3 a lot more likely to follow instructions than it would have been otherwise. then it probably makes gpt 3.5 quite a bit more aligned, and gpt 4 somewhat more aligned, and gpt 4.5 a tiny bit more aligned, and by the time gpt 5 rolls around your counterfactual impact is almost ne... (read more)

Caspar Oesterheld23d22

These memetic dynamics (as per subclaim 2) seem very complicated and hard to predict. For example, ever since the HF–OAI incident I worry that a lot of past warning shots have decreased concern on net because they caused people to learn that some scary-seeming stuff sometimes happens and life goes on. E.g., the Bing Chat "Sidney" incident was very spooky in some sense, but then if you play around with Bing Chat a bunch, you realize that it is not very intelligent (and thus couldn't do much harm) and also often goes off the rails a bit in harmless ways (lik... (read more)

3Alex Mallen23d

The most plausible-to-me reasons why prosaic alignment now might be good (I'm in fact uncertain about the bottom line): * Near-term alignment is actually very load-bearing for automating alignment research well as opposed to sloppily. At least, models should be average-case behaviorally aligned for this, as opposed to gaming our metrics. * Failing to solve near-term alignment problems might actually produce nearish-term AIs that take over the world (perhaps indirectly via surreptitiously controlling AI development). So prosaic alignment was necessary for surivival. * Benefits don't decay sharply enough that alignment fails before the end of the critical risk period (e.g., because we successfully pause before ASI, or some human faction gets a DSA with relatively weak AIs).

TurnTrout's shortform feed

TurnTrout24d71

https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyber-capable-agents/ demonstrates a challenge on the horizon for AI containment (including my glovebox tool).

As part of Patch the Planet, we received preview access to GPT 5.6-Cyber with a simple task: evaluate its cyber capabilities. Recent events inspired me to give it a challenge to work through: escape the VM I’d normally use for sandboxing. The target was a QEMU/KVM VM on my Linux dev machine (Debian Linux 12, AMD Zen3). It escaped the VM three different times.

This is a earlier than I had imag... (read more)

Quick Takes

Quick Takes

How aligned is Astra actually? (Concerns with alignment metrics and papering over misalignment.)