AIs can now often do massive easy-to-verify SWE tasks and I've updated towards shorter timelines

I've recently updated towards substantially shorter AI timelines and much faster progress in some areas. ^[1] The largest updates I've made are (1) an almost 2x higher probability of full AI R&D automation by EOY 2028 (I'm now a bit below 30% ^[2] while I was previously expecting around 15%; my guesses are pretty reflectively unstable) and (2) I expect much stronger short-term performance on massive and pretty difficult but easy-and-cheap-to-verify software engineering (SWE) tasks that don't require that much novel ideation ^[3] . For instance, I expect that by EOY 2026, AIs will have a 50%-reliability...

1Oliver Sourbut7h

Why is TEDAI (operationalised as remote-work-only if I understand) so similar to AI stack + conflict (includes full supply chain self-sufficiency, warfighting, real-time operational tactics) for you? Is this just superexponentials eating up all the gaps? How come TEDAI is less likely later on (but more likely sooner)?

ryan_greenblatt43m20

AI stack + conflict parity requires lots of robots (or crazy novel tech) but doesn't require AIs as capable as TEDAI. TEDAI is a very high capabilities bar. So, in worlds without a software only singularity and especially with slower takeoff, I think you may reach AI stack + conflict parity prior to TEDAI. (It's certainly possible to have great military robots and robot industrial capacity with AIs that are well within the human range on key skills.) TEDAI probably follows reasonably quickly, because economic doubling times are so fast in such a world. In ... (read more)

5ryan_greenblatt19h

This is a good point and I broadly agree with what you're saying. Some possible disagreements: [...] Maybe I should have made this clearer, but I'm not talking about scaling to arbitrary inference costs. Instead, I'm talking about scaling to inference costs that are a moderate fraction of the human task completion cost. (E.g., 1%-100% depending on the task.) I think you'd want to compare the AI performance at some inference budget to human labor with some limit in supply. [...] Yep, it seems reasonably likely that on this alternative notion of time horizon, the horizon length is more like a few days. However, under this alternative definition, relatively small time-horizon values may correspond to much larger (real-world) impact. For example, an AI with a decomposed-time-horizon of ~1 week could potentially speed up AI R&D a lot, while under the original time-horizon notion, a 1-week 50%-reliability time horizon is much less of a big deal. (To be more precise about the decomposed-time-horizon metric: this would correspond to AI having a 50% chance of matching a team of humans where each human gets 1 week before being replaced, with total labor-hours capped at ~100x what the task would normally require to avoid degeneracies from truly arbitrary inference spend.) So the key question becomes: how much can you accomplish if the task must be heavily decomposed but you can apply much more labor? I'm pretty uncertain about this for the tasks we care about. Further, AIs may soon end up being especially good (superhuman?) at working in heavily decomposed contexts—e.g., good at leaving notes to other instances, good at quickly picking up project state from limited context. This makes the correspondence to doing task decomposition with humans more fraught, even though AIs will still do relatively better at tasks that are easier to do incrementally or otherwise decompose. I think it's already the case that AIs are extremely good relative to humans at loading up complex sta

6paulfchristiano14h

I agree. In the more realistic regime you are talking about you have some more complicated quantitative question around how large are the slowdowns from task decomposition into what scale. My main point was that for the tasks we are talking about here, the slowdowns seem like they might not be that large even for modest human time horizons. (In contrast with some of the crazy factored cognition stuff we have sometimes talked about, which involves much shorter horizons, much harder-to-decompose tasks, and much larger slowdowns.) [...] I agree that this could lead to large impacts with relatively short horizons (perhaps even today's horizons, with an appropriately broadened training distribution and a bunch of schlep). That does imply a different picture of AI strengths and weaknesses (e.g. weaker on-the-job learning with performance mostly limited to domains near the training distribution; differential speedup for tasks that are easily decomposed), with a more schlep-heavy singularity, a greater role for tight human involvement later in the process, and probably less alignment concern earlier in the trajectory.

There should be $100M grants to automate AI safety

Marius Hobbhahn

This post reflects my personal opinion and not necessarily that of other members of Apollo Research.

TLDR: I think funders should heavily incentivize AI safety work that enables spending $100M+ in compute or API budgets on automated AI labor that directly and differentially translates to safety.

Motivation

I think we are in a short timeline world (and we should take the possibility seriously even if we don't have full confidence yet). This means that I think funders should aim to allocate large amounts of money (e.g. $1-50B per year across the ecosystem) on AI safety in the next 2-3 years.

I think that the AI safety funders have been allocating way too little funding and their spending has been far too conservative in the past 5 years. So, in my opinion,...

(Continue Reading - 2369 more words)

Charlie Steiner4d30

Datasets might be nice.

Object-level values.
- "What do you like or dislike about my current life?"
- "What kind of actions do you want to take in the next few weeks?"
- "What kind of changes would you make to the world around you if you could?"
- "What are some examples of kindness that you've witnessed?"
- "Come up with a moral dilemma that seems close to you."
- "What would you do in this moral dilemma someone else came up with?"
- etc.
Meta-level values.
- "How would you change yourself if you could?"
- "How do you feel about various ways you expect to grow and change in the futur

LawrenceC

Written quickly as part of the Inkhaven Residency.

At a high level, research feedback I give to more junior research collaborators often can fall into one of three categories:

Doing quick sanity checks
Saying precisely what you want to say
Asking why one more time

In each case, I think the advice can be taken to an extreme I no longer endorse. Accordingly, I’ve tried to spell out the degree to which you should implement the advice, as well as what “taking it too far” might look like.

This piece covers doing quick sanity checks, which is the most common advice I give to junior researchers. I’ll cover the other two pieces of advice in a subsequent piece.

Doing quick sanity checks

Research is hard (almost by definition) and people are often wrong. Every...

(See More - 699 more words)

1Raemon5d

(not 100% sure, but, I think future readers would benefit more from a title like "Doing quick sanity checks in research" than "my most common advice for junior researchers". Or, if I read the latter title I'd expect more like an overview than a description of one particular thing)

LawrenceC5d20

Edited the title!

Judgements: Merging Prediction & Evidence

abramdemski

I recently wrote about complete feedback, an idea which I think is quite important for AI safety. However, my note was quite brief, explaining the idea only to my closest research-friends. This post aims to bridge one of the inferential gaps to that idea. I also expect that the perspective-shift described here has some value on its own.

In classical Bayesianism, prediction and evidence are two different sorts of things. A prediction is a probability (or, more generally, a probability distribution); evidence is an observation (or set of observations). These two things have different type signatures. They also fall on opposite sides of the agent-environment division: we think of predictions as supplied by agents, and evidence as supplied by environments.

In Radical Probabilism, this division is not so strict....

(Continue Reading - 1774 more words)

Wei Dai5d20

I missed this post when it came out and just came across it. Re "Logical Induction as Metaphilosophy" I have the same objection that I recently made against "reflective equilibrium":

Another argument against this form of reflective equilibrium is that it seems to imply anti-realism about normative decision theory, given differing intuitions between people. I think this is plausible but not likely, so it seems bad to bake it into our methodology of doing decision theory.

In other words, in Logical Induction as Metaphilosophy there seems to be nothing groundin... (read more)

Predicting When RL Training Breaks Chain-of-Thought Monitorability

David Lindner

Crossposted from the DeepMind Safety Research Medium Blog. Read our full paper about this topic by Max Kaufmann, David Lindner, Roland S. Zimmermann, and Rohin Shah.

Overseeing AI agents by reading their intermediate reasoning “scratchpad” is a promising tool for AI safety. This approach, known as Chain-of-Thought (CoT) monitoring, allows us to check what a model is thinking before it acts, often helping us catch concerning behaviors like reward hacking and scheming.

However, CoT monitoring can fail if a model’s chain-of-thought is not a good representation of the reasoning process we want to monitor. For example, training LLMs with reinforcement learning (RL) to avoid outputting problematic reasoning can result in a model learning to hide such reasoning without actually removing problematic behavior.

Previous work produced an inconsistent picture of whether RL training hurts CoT monitorability. In some prior...

(Continue Reading - 1253 more words)

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

Motivation

Doing quick sanity checks