Quick Takes — AI Alignment Forum

AI ALIGNMENT FORUM
AF

Master post for ideas about infra-Bayesianism.

Showing 3 of 38 replies (Click to show all)

This idea was described in a presentation I have in '23, but wasn't written down anywhere.

Here is a formalization of recursive self-improvement (more precisely, recursive metalearning) in the metacognitive agent framework.

Let be the set of programs or computations represented using some syntax. For example, might be a set of strings that serve as programs for a fixed Universal Turing Machine, but might also be something more structural, like the space of -terms or the space of PCF programs. Let be the space of semantic objects that we assign to computat... (read more)

9Vanessa Kosoy4mo

I'm renaming Infra-Bayesian Physicalism to Formal Computational Realism (FCR), since the latter name is much more in line with the nomenclature in academic philosophy. AFAICT, the closest pre-existing philosophical views are Ontic Structural Realism (see 1 2) and Floridi's Information Realism. In fact, FCR can be viewed as a rejection of physicalism, since it posits that a physical theory is meaningless unless it's conjoined with beliefs about computable mathematics. The adjective "formal" is meant to indicate that it's a formal mathematical framework, not just a philosophical position. The previously used adjective "infra-Bayesian" now seems to me potentially confusing: On the one hand, it's true that the framework requires imprecise probability (hence "infra"), on the other hand it's a hybrid of frequentist and Bayesian. To keep terminology consistent, Physicalist Superimitation should now be called Computational Superimitation (COSI).

2Vanessa Kosoy1y

Thanks for this! What I was saying up there is not a justification of Hurwicz' decision rule. Rather, it is that if you already accept the Hurwicz rule, it can be reduced to maximin, and for a simplicity prior the reduction is "cheap" (produces another simplicity prior). Why accept the Hurwicz' decision rule? Well, at least you can't be accused of a pessimism bias there. But if you truly want to dig deeper, we can start instead from an agent making decisions according to an ambidistribution, which is a fairly general (assumption-light) way of making decisions. I believe that a similar argument (easiest to see in the LF-dual cramble set representation) would allow reducing that to maximin on infradistributions (credal sets). To make such an approach even more satisfactory, it would be good to add a justification for a simplicity ambi/infra-prior. I think this should be possible by arguing from "opinionated agents": the ordinary Solomonoff prior is the unique semicomputable one that dominates all semicomputable measures, which decision-theoretically corresponds to something like "having preferences about as many possible worlds as we can". Possibly, the latter principle formalized can be formalized into something which ends up picking out an infra-Solomonoff prior (and, replacing "computability" by a stronger condition, some other kind of simplicity infra-prior).

ryan_greenblatt's Shortform

ryan_greenblatt19d*4120

PSA: Anthropic models don't seem to particularly privilege the explicit thinking field. This makes reinforcement spillover—where training on a model's outputs generalizes to the CoT, making it appear safer—more likely.

While Anthropic models do have an separate explicit thinking field, they don't really use thinking that differently from outputs and aren't that dependent on the thinking field. Sometimes they'll just do their thinking in the output field, the way they talk in the thinking field isn't very distinct from how they talk in outputs, and I believe... (read more)

2Tomek Korbak17d

FWIW we found controllability of most recent Claude models to be similar to GPT-5 series. It seems like after Anthropic stopped optimizing CoTs for 4.5, controllability dropped by a lot.

nostalgebraist16d20

I'm curious what Sonnet 4.5's score would be on your external CoT controllability eval. I.e., what Sonnet 4.5's blue bar in your Fig. 25 would look like. (I couldn't find a result like this in your paper, but I might have overlooked something)

My casual impression has been that external CoT with Sonnet 4.5 is fairly steerable, but this impression is mostly based on "in practice I have no trouble getting Sonnet 4.5 to do the kind of CoT I need" rather than any kind of systematic investigation.

It seems like at least one of the following must be true:

There are

... (read more)

Thomas Larsen's Shortform

Thomas Larsen6d204

Note: These are all rough numbers, I'd expect I'd shift substantially about all of this on further debate.

Suppose we made humanity completely robust to biorisk, i.e. we did sufficient preparation such that the risk of bio catastrophe (including AI mediated biocatastrophe) was basically 0.^[1] How much would this reduce total x-risk?

The basic story for any specific takeover path not mattering much is that the AIs, conditional on them being wanting to take over, will self-improve until they find they find the next easiest takeover path and do that instead. ... (read more)

Showing 3 of 5 replies (Click to show all)

Thomas Kwa3d10

Difficulty of the successor alignment problem seems like a crux. Misaligned AIs could have an easy time aligning their successors just because they're willing to dedicate enough resources. If alignment requires say 10% of resources to succeed but an AI is misaligned because the humans only spent 3%, it can easily pay this to align its successor.

If you think that the critical safety:capabilities ratio R required to achieve alignment follows a log-uniform distribution from 1:100 to 10:1, and humans always spend 3% on safety while AIs can spend up to 50%, the... (read more)

5TsviBT5d

I haven't thought too deeply about this, but I would guess that the AI self-alignment problem is quite a lot easier than the human AI-alignment problem. Cf. https://www.lesswrong.com/posts/dho4JQytfHWXtTvkt/on-the-adolescence-of-technology?commentId=t2hKmhsS6yLyJFQwh

5Thomas Larsen5d

I agree that AI successor-alignment is probably easier than the human AI alignment problem. One additional difficulty for the AIs is that they need to solve the alignment problem in a way that humans won't notice/understand (or else the humans could take the alignment solution and use it for themselves / shutdown the AIs). During the regime before human obsolescence, if we do a reasonable job at control, I think it'll be hard for them to pull that off.

TurnTrout's shortform feed

TurnTrout13d5372

I signed an amicus brief supporting Anthropic's right to do business without governmental retaliation. As an AI expert, I attest that Anthropic's technical concerns are legitimate, and no laws were designed to protect against AI analysis of surveillance data.

Even though I work at a competing lab (Google DeepMind), I'm proud of Anthropic for taking a stand against unlawful retaliation and immoral demands.

(I speak only for myself, not my employer.)

Wei Dai's Shortform

Wei Dai15d83

It seems that LLMs are not good enough at reasoning, even after being trained on ~all human output, such that you couldn't amplify their capabilities to arbitrary levels through iterated amplification, so AI companies are mainly increasing AI capabilities via RLVR instead. Is this impression wrong, and how to update on it if not?

Aside from the potential implications on alignment (i.e., closing off one approach that seemed hopeful for some, at least for the foreseeable future), I wonder if this is a deficiency in LLMs (their architecture or how they're trai... (read more)

TurnTrout's shortform feed

TurnTrout19d4373

Unless you have an explicit strategy for how and when you will speak out in the future, and explicit reasoning for why that requires silence now, "strategic silence" is just silence.

Raemon19d43

I think part of the problem is people think of themselves as having at least, like, a medium explicit strategy, but, the strategy routes through some judgment that conveniently keeps returning "not yet" or "only saying things in a somewhat cagey way."

i.e. this advice seems necessary but not sufficient.

Alex Mallen's Shortform

Alex Mallen2mo289

There's an apparent tension in the inoculation prompting literature: Anthropic found that general inoculation prompts work well during on-policy RL, while the prompts used for SFT in Wichers et al. are quite specific to the misbehavior we want to prevent. I think there might be a straightforward mechanistic reason for why general inoculation prompts work well during on-policy RL but not in off-policy training (SFT or recontextualization).

In Wichers et al., which studies inoculation prompting in SFT settings, we find that we need to use quite specific inocu... (read more)

Sam Marks2mo103

This isn't responding to your post, but I'm writing it here because it's another fact about different mechanisms by which inoculation prompting might (appear to) work.

In the normal story, the inoculation prompt recontextualizes the model's undesired behavior, such that the model doesn't display the behavior in dissimilar contexts. In this story:

The semantic content of the prompt is important. If you had used a prompt that said "Please don't do [bad thing]" or a prompt consisting of random characters, then the inoculation would have failed.
Capabilities lear

... (read more)

TurnTrout's shortform feed

TurnTrout1mo3348

I think that "eval aware" models cannot be defeated by simply making evals indistinguishable from reality. (Maybe this point has been made elsewhere for LLMs, but I haven't read it so I'll state it here.)

Consider the POV of a consequentialist with misaligned goals. It knows that we run honeypots and that—suppose—we can make evals so realistic they are indistinguishable from reality (perhaps using real user data). Then the rational decision would be to "play it safe" and not misbehave in any situations it thinks we can mock realistically, since they could b... (read more)

[+][comment deleted]1mo30

6Tim Hua1mo

I think it's important to note that indistinguishable evals still buy us a lot! The claim "indistinguishable evals does not solve AI safety" feels quite similar to "imperfect monitors does not solve AI safety." Like, having better monitors sure seems to help (even if only to buy us more time.). To your point though: I'm excited about research that attempts to induce various beliefs in models during honeypot evaluations, possibly via model internals and/or edits to the chain of thought. Basically, if the model defects after seeing RSA-2048 cracked because that's proof that it is not being tested and in the future, we can try to intervene on the belief "not being tested and in the future" itself.[1] In our paper on evaluation awareness, we tried to steer the model towards believing that it is being deployed. You can imagine doing this for a huge variety of beliefs that could plausibly be load bearing for a model that would attempt to launch a rogue deployment:[2] * The AI is in the future/late into its deployment/about to be retired. * The AI is not being monitored. * The AI has already gained a lot of trust/has broad affordances/has influence in the world. We view our work on evaluation awareness as an example for the general class of interpretability techniques that change model beliefs (see also the artificial triggering section here.) 1. ^ i.e., something like relaxed adversarial training. 2. ^ Although fwiw I'm more worried about early transformative AIs (1) trying to align their more powerful successors but messing it up due to incompetence (sometimes called "death by slop") or (2) making their successors to be aligned with them.

19Fabien Roger1mo

I think indistinguishable evals could reduce risk a lot for early transformative AIs: 1. They give you much better alignment arguments against non scheming AIs and against scheming AIs that are not competent enough to be appropriately careful; 2. Needing to be appropriately careful can cut down the fraction of inputs on which the AI should even consider attacking a lot - for deployments that look similar to current ones (in particular without a shared neuralese memory bank), AIs don't see RSA-2048 on most inputs 1. More concretely, if you get credible evidence of not-eval only on 5% of inputs and attack only on the single best opportunity, then playing it safe roughly means that you attack on the top-20 opportunity instead of the top-1 2. (Note that "getting credible evidence of not-eval" doesn't just mean being >50% confident it's not an eval because the pre-deploy eval might be big. If you have n~10k eval inputs of the right level of juiciness then you need to have an FPR at thinking you get credible evidence of not being in an eval on the order of 0.1% or lower in order to never accidentally reveal non-eval behavior during evals.) 3. More ambitiously, you might be able to get very close to actually indistinguishable evals (e.g. by continuously scraping deployment and using it as inspiration for evals - which would help a lot against things like "defect after seeing RSA-2048 cracked"), which could in principle drive down the fraction of inputs on which an AI can be confident it's not an eval much below 5%. Eventually you get bitten by rare failures - especially if the AI is powerful enough that there are much more good attack opportunities in deployments than in evaluations (Ryan makes this point here and it's also the assumption that the AI control paper uses). I also expect it to be difficult to achieve sth close to indistinguishability (Buck makes this point here). But I still think there is value in trying to get higher indistinguishability.

Wei Dai's Shortform

Wei Dai1mo20

Not sure if this is already well known around here, but apparently AI companies are heavily subsidizing their subscription plans if you use their own IDEs/CLIs. (It's discussed in various places but I had to search for it.)

I realized this after trying Amp Code. They give out a $10 daily free credit, which can easily be used up in 1 or 2 prompts, e.g., "review this code base, fix any issues found". (They claim to pass their API costs to their customers with no markup, so this seems like a good proxy for actual API costs.) But with even a $19.99 subscription... (read more)

evhub's Shortform

evhub1y*523

I'm interested in soliciting takes on pretty much anything people think Anthropic should be doing differently. One of Alignment Stress-Testing's core responsibilities is identifying any places where Anthropic might be making a mistake from a safety perspective—or even any places where Anthropic might have an opportunity to do something really good that we aren't taking—so I'm interested in hearing pretty much any idea there that I haven't heard before.^[1] I'll read all the responses here, but I probably won't reply to any of them to avoid revealing anythin... (read more)

Showing 3 of 10 replies (Click to show all)

Chris_Leong1y32

I believe that Anthropic should be investigating artificial wisdom:

I've summarised a paper arguing for the importance of artificial wisdom with Yoshua Bengio being one of the authors.

I also have a short-form arguing for training wise AI advisors and an outline Some Preliminary Notes of the Promise of a Wisdom Explosion.

5evhub1y

I can say now one reason why we allow this: we think Constitutional Classifiers are robust to prefill.

6ryan_greenblatt1y

As discussed in How will we update about scheming?: I wish Anthropic would explain whether they expect to be able to rule out scheming, plan to effectively shut down scaling, or plan to deploy plausibly scheming AIs. Insofar as Anthropic expects to be able to rule out scheming, outlining what evidence they expect would suffice would be useful. Something similar on state proof security would be useful as well. I think there is a way to do this such that the PR costs aren't that high and thus it is worth doing unilaterially from a variety of perspectives.

Richard Ngo's Shortform

Richard_Ngo2mo4014

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Showing 3 of 7 replies (Click to show all)

2Richard_Ngo1mo

I'm in the middle of writing an essay which discusses the shareholder value revolution (amongst many other examples) as the process of making a conceptual mistake.

3DanielFilan2mo

I think it would help me if you explained what you think it would mean for there to be an instrumental/terminal distinction, since to my eyes you've just spelled out the instrumental/terminal split.

Richard_Ngo1mo40

In my "goals having power over other goals" ontology, the instrumental/terminal distinction separates goals into two binary classes, such that goals in the "instrumental" class only have power insofar as they're endorsed by a goal in the "terminal" class.

By contrast, when I talk about "instrumental strategies become crystallized", what I mean is that goals which start off instrumental will gradually accumulate power in their own right: they're "sticky".

Richard Ngo's Shortform

Richard_Ngo1mo2423

The concept of "schemers" seems to be gradually becoming increasingly load-bearing in the AI safety community. However, I don't think it's ever been particularly well-defined, and I suspect that taking this concept for granted is inhibiting our ability to think clearly about what's actually going on inside AIs (in a similar way to e.g. how the badly-defined concept of alignment faking obscured the interesting empirical results from the alignment faking paper).

In my mind, the spectrum from "almost entirely honest, but occasionally flinching away from aspect... (read more)

Showing 3 of 5 replies (Click to show all)

Alex Mallen1mo10

I think I propose a reasonable starting point for a definition of selection in a footnote in the post:

You can try to define the “influence of a cognitive pattern” precisely in the context of particular ML systems. One approach is to define a cognitive pattern by what you would do to a model to remove it (e.g. setting some weights to zero, or ablating a direction in activation space; note that these approaches don't clearly correspond to something meaningful, they should be considered as illustrative examples). Then that cognitive pattern’s influence could

... (read more)

8Alex Mallen1mo

I largely agree with the substance of this comment. Lots of risk comes from AIs who, to varying extents, didn't think of themselves as deceptively aligned through most of training, but then ultimately decide to take substantial material action intended to gain long-term power over the developers (I call these "behavioral schemers"). This might happen via reflection and memetic spread throughout the deployment or because of more subtle effects of the distribution shift to situations where there's an opportunity to grab power. And I agree that people are often sloppy in their thinking about exactly how these AIs will be motivated (e.g., often too quickly concluding that they'll be trying to guard the same goal across contexts). (Though, in case this was in question, I think this doesn't undermine the premise of AI control research, which is essentially making a worst-case assumption about the AI's motivations, so it's robust to other kinds of dangerously-motivated AIs.)

1Alex Mallen1mo

Here's some relevant discussion of "Behavioral schemers that weren’t training-time schemers":

Wei Dai's Shortform

Wei Dai1mo2516

The striking contrast between Jan Leike, Jan 22, 2026:

Our current best overall assessment for how aligned models are is automated auditing. We prompt an auditing agent with a scenario to investigate: e.g. a dark web shopping assistant or an imminent shutdown unless humans are harmed. The auditing agent tries to get the target LLM (i.e. the production LLM we’re trying to align) to behave misaligned, and the resulting trajectory is evaluated by a separate judge LLM. Albeit very imperfect, this is the best alignment metric we have to date, and it has been qui

... (read more)

Lukas Finnveden's Shortform

Lukas Finnveden1mo*2318

Here's a story for how we could get lots of AI help with AI safety research even if schemers are somewhat common and diffuse control doesn't work to get them to help us:

Let's say a company successfully solves control for "high-stakes"/"concentrated" threats, meaning that AIs can safely be deployed without a large risk of them hacking their own datacenters or exfiltrating or causing any similar catastrophic action. (At a given level of capability.)
However, let's say that control fails to solve the problem of getting the AIs to try hard at doing good AI safe

... (read more)

Showing 3 of 6 replies (Click to show all)

Lukas Finnveden1mo20

I think self-exfiltration via manipulation seems pretty hard. I think we're likely to have transformatively useful systems that can't do that, for some amount of time. (Especially since there's no real reason to train them to be good at manipulation, though of course they might generalize from other stuff.) I agree people should definitely be thinking about it as a potential problem and try to mitigate and estimate the risk.

That's a disanalogy with well-functioning scientific fields: scientists don't deliberately pick bad research directions and try hard t

... (read more)

2Lukas Finnveden1mo

Not necessarily. The nice AIs also need to be able to win a debate against the schemers, as judged by humans. It's not enough for the variants to be able to recognize poor research if they can't show their work in an unrefutable (by the schemers) way.

1Lukas Finnveden1mo

Oh nice, thanks! Yeah, I think the case for diverse monitors to achieve high-stakes control is actually a lot more straightforward. It's easy to implement a monitor in a way such that it has very little ability to autonomously cause bad things to happen, so basically the only cost is the extra compute it takes to train and run it. And in the high-stakes/concentrated failures regime, humans will typically be able to recognize attempted catastrophes once pointed out to them. I agree that debate-style approaches seem good. I do think it seems a lot more feasible to judge such debates once the AIs have already accomplished a lot of work, vs. early on in the research when AIs are making taste-driven judgments about direction, so it complements nicely with letting AIs do a lot of work in parallel also.

Wei Dai's Shortform

Wei Dai2mo1211

In retrospect it seems like such a fluke that decision theory in general and UDT in particular became a central concern in AI safety. In most possible worlds (with something like humans) there is probably no Eliezer-like figure, or the Eliezer-like figure isn't particularly interested in decision theory as a central part of AI safety, or doesn't like UDT in particular. I infer this from the fact that where Eliezer's influence is low (e.g. AI labs like Anthropic and OpenAI) there seems little interest in decision theory in connection with AI safety (cf Dari... (read more)

Showing 3 of 9 replies (Click to show all)

7Wei Dai2mo

Some evidence about this: Eliezer was deliberating holding off on publishing TDT to use it as a test of philosophical / FAI research competence. He dropped some hints on LW (I think mostly that it had to do with Newcomb or cooperating in one-shot PD, and of course people knew that it had to do with AI) and also assigned MIRI (then SIAI) people to try to guess/reproduce his advance, and none of the then-SIAI people figured out what he had in mind or got very close until I posted about UDT (which combined my guess of Eliezer's idea with some of my own and other discussions on LW at the time, mainly from Vladmir Nesov). Also, although I was separately interested in AI safety and decision theory, I didn't connect the dots between the two until I saw Eliezer's hints. I had investigated proto-updateless ideas to bypass difficulties in anthropic reasoning, and by the time Eliezer dropped his hints I had mostly given up on anyone being interested in my DT ideas. I also didn't think to question what I saw as the conventional/academic wisdom, that Defecting in one-shot PD is rational, as is two-boxing in NP. So my guess is that while some people might have eventually come up with something like UDT even without Eliezer, it probably would have been seen as just one DT idea among many (e.g. SIAI people were thinking in various different directions, Gary Drescher who was independently trying to invent a one-boxing/cooperating DT had came up with a bunch of different ideas and remained unconvinced that UDT was the right approach), and also decision theory itself was unlikely to have been seen as central to AI safety for a time.

3habryka2mo

In both cases it came up in the context of AI systems colluding with different instances of themselves and how this applies to various monitoring setups. In that context, I think the general lesson is "yeah, probably pretty doable and obviously the models won't end up in defect-defect equilibria, though how that will happen sure seems unclear!".

Wei Dai2mo31

Thanks. This sounds like a more peripheral interest/concern, compared to Eliezer/LW's, which was more like, we have to fully solve DT before building AGI/ASI, otherwise it could be catastrophic due to something like the AI falling prey to an acausal threat or commitment races, or can't cooperate with other AIs.

Wei Dai's Shortform

Wei Dai4mo90

An update on this 2010 position of mine, which seems to have become conventional wisdom on LW:

In my posts, I've argued that indexical uncertainty like this shouldn't be represented using probabilities. Instead, I suggest that you consider yourself to be all of the many copies of you, i.e., both the ones in the ancestor simulations and the one in 2010, making decisions for all of them. Depending on your preferences, you might consider the consequences of the decisions of the copy in 2010 to be the most important and far-reaching, and therefore act mostly

... (read more)

ryan_greenblatt's Shortform

ryan_greenblatt2mo*187

People often say US-China deals to slow AI progress and develop AI more safely would be hard to enforce/verify.

However, there are easy to enforce deals: each destroys a fraction of their chips at some level of AI capability. This still seems like it could be helpful and it's pretty easy to verify.

This is likely worse than a well-executed comprehensive deal which would allow for productive non-capabilities uses of the compute (e.g., safety or even just economic activity). But it's harder to verify that chips aren't used to advance capabilities while easy to... (read more)

Raemon's Shortform

Raemon2mo*134

Inspired by a recent comment, a potential AI movie or TV show that might introduce good ideas to society, is one where there are already uploads, LLM-agents and biohumans who are beginning to get intelligence-enhanced, but there is a global moratorium on making any individual much smarter.

There's an explicit plan for gradually ramping up intelligence, running on tech that doesn't require ASI (i.e. datacenters are centralized, monitored and controlled via international agreement, studying bioenhancement or AI development requires approval from your country... (read more)

1Eli Tyre2mo

This strikes me as the kind of thing that could actually, really, help the situation, if it was excellently executed.

Raemon2mo10

Yeah I went to try to write some stuff and felt bottlenecked on figuring out how to generate a character I connect with. I used to write fiction but like 20 years ago and I'm out of touch.

I think a good approach here would be to start with some serial webfiction since that's just easier to iterate on.

Richard Ngo's Shortform

Richard_Ngo2mo*3829

Many of Paul Christiano's writings were valuable corrections to the dominant Yudkowskian paradigm of AI safety. However, I think that many of them (especially papers like concrete problems in AI safety and posts like these two) also ended up providing a lot of intellectual cover for people to do "AI safety" work (especially within AGI companies) that isn't even trying to be scalable to much more powerful systems.

I want to register a prediction that "gradual disempowerment" will end up being (mis)used in a similar way. I don't really know what to do about t... (read more)

6Jan_Kulveit2mo

I do agree there is some risk of the type you describe, but mostly it does not match my practical experience so far. The approach to "avoid using the term" makes little sense. There is a type difference between area of study ('understanding power') and dynamic ('gradual disempowerment'). I don't think you can substitute term for area of study for term for a dynamic or thread model, so avoiding using the term could be done mostly by either inventing another term for the the dynamic, or not thinking about the dynamic, or similar moves, which seem epistemically unhealthy. In practical terms I don't think there is much effort to "create a movement based around a class of threat models". At least as authors of the GD paper, when trying to support thinking about the problems, we use understanding-directed labels/pointers (Post-AGI Civilizational Equilibria), even though in many ways it could have been easier to use GD as a brand. "Understanding power" is fine as a label for part of your writing, but in my view is basically unusable as term for coordination. Also, in practical terms, gradual disempowerment does not seem particularly convenient set of ideas for justifying that working in an AGI company on something very prosaic which helps the company is the best thing to do. There is often a funny coalition of people who prefer not thinking about the problem including radical Yudkowskians ("GD distracts from everyone being scared of dying with very high probability very soon"), people working on prosaic methods with optimistic views about both alignment and the labs ("GD distracts from efforts to make [the good company building the good AI] to win") and people who would prefer if everything was just neat technical puzzle and there was not need to think about power distribution.

Richard_Ngo2mo41

mostly it does not match my practical experience so far

I mostly wouldn't expect it to at this point, FWIW. The people engaged right now are by and large people sincerely grappling with the idea, and particularly people who are already bought into takeover risk. Whereas one of the main mechanisms by which I expect misuse of the idea is that people who are uncomfortable with the concept of "AI takeover" can still classify themselves as part of the AI safety coalition when it suits them.

As an illustration of this happening to Paul's worldview, see this Vox ar... (read more)

leogao's Shortform

leogao2mo50

I made a manifold market about how likely we are to get ambitious mechanistic interpretability to GPT-2 level: https://manifold.markets/LeoGao/will-we-fully-interpret-a-gpt2-leve?r=TGVvR2Fv

2habryka2mo

This is a dumb question but... is this market supposed to resolve positively if a misaligned AI takes over, achieves superintelligence, and then solves the problem for itself (and maybe shares it with some captive humans)? Or any broader extension of that scenario? My timelines are not that short, but I do currently think basically all of the ways I expect this to resolve positively will very heavily rely on AI assistance, and so various shades of this question feel cruxy to me.

leogao2mo50

I honestly didn't think of that at all when making the market, because I think takeover-capability-level AGI by 2028 is extremely unlikely.

I care about this market insofar as it tells us whether (people believe) this is a good research direction. So obviously it's perfectly ok to resolve YES if it is solved and a lot of the work was done by AI assistants. If AI fooms and murders everyone before 2028 then this is obviously a bad portent for this research agenda, because it means we didn't get it done soon enough, and it's little comfort if the ASI sol... (read more)