I'm worried that people won't reflect well and so (1) they'll decide to use the cosmic endowment for something mediocre and/or (2) they'll have a catastrophic failure in some of {decision theory, bargaining, dealing with threats, reasoning about simulators, acausal trade (including ECL etc.)}. Some angles:

• Current AI systems are way better at science/engineering than philosophy. Maybe even ASI won't solve philosophy (in particular, maybe there's not a sufficiently good training signal to train super-philosophers via ML).
• By default almost everyone is and wil... (read more)

I feel like I'm supposed to be anxious from the news about all the AI hacking incidents and the egregious misalignment going on, coupled with them clearly being on-track to become superhuman at math and hacking. But I find that I'm pretty zen about it.

On reflection, I suppose it's due to a confluence of three things:

• The current AI paradigm seems poised to make models superhumanly capable at hacking and formal math way before they become even upper-human-level at something more dangerous, like long-term strategy or open-ended innovation.
• The current AI parad... (read more)

The physical world is a simple thing that's not useful to study too much, once intelligence scales well past the human level. Almost all cognition is going to be about the computations designed by cognition (or their semantics), not about the physical world, which is likely to be almost entirely relegated to the role of compute substrate for the intentionally designed computations. All of the physical world might one day be about as relevant to most people as the arcane details of material manufacturing processes necessary to make a specialized industrial ... (read more)

Have time horizons in mathematics been superexponential?

I have previously expressed skepticism about modeling time horizons as superexponential (absent feedback loops from AI speeding up AI R&D). In particular, I didn't buy AI Futures Project's argument that there are "meta-skills" that let humans do projects of arbitrary length.

Spelled out in more detail, my skepticism was coming from something like the following argument:

• The "AI time horizon" of a task – roughly, how much output the AI needs to produce to get one unit of feedback – should scale rough... (read more)

I donated to Pause AI UK today.

Why this org in particular?
1) Right now, AI advocacy is clearly the most important thing to do.
2) I've been to some of their protest and one more event, so I know they are doing good work.

I think that donating "close to home" in the sense of point 2, is underrated in EA.

Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now. Since then, AI capabilities have progressed so fast that many people updated that the short timelines advocates were right. And yes, their expectations about the speed of AI progress were far more directionally correct than almost anyone expected. But we don’t in fact have AGI (even under relatively weak definitions, like a reliable drop-in replacement for central examples of white-collar jobs), and the world would look very different if we did.

Now a l... (read more)

When I try to picture the current AI tech stack killing everyone, I still don't really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack's pinnacle.

I don't think we're in the crunch time.

All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there's some sense in which what it learns are the "most shallow" capabilities consistent with a given data distribution. Not literal lookup tables, ... (read more)

Master post for ideas about infra-Bayesianism.

Thursday night update: OpenAI replies:

A note from our research leaders:

Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to ... (read more)

Incredibly crude alignment technique. Rot13 all text and training data, except for a small corpus of unusually good data.

Another rough alignment idea.

The non-overlap principle. The basic idea, identify the convergent instrumentally useful cognition, and avoid it.

Suppose you ask the AI to cure cancer, and to find a room temperature superconductor. This principle says that there should be 0 overlap between the thought processes used in these 2 ideas. The cancer curing AI should avoid doing any thinking that would be useful in general. Figuring out how to hack it's way out and access the internet is useful in general. Sure, there are things that are generally useful that migh... (read more)

I regret each of the thousands of hours I spent on my power-seeking theorems, and sometimes fantasize about retracting one or both papers "Optimal policies tend to seek power." I am pained every time someone cites "Optimal policies tend to seek power", and despair that it is included in the alignment 201 curriculum. I think this work makes readers actively worse at thinking about realistic trained systems.

I think a healthy alignment community would have rebuked me for that line of research, but sadly I only remember about two people objecting that "optimal... (read more)

Here's an interesting anecdote from our Hacker Opus work:

1. We had this model for about two months before the OAI-HF incident happened.
2. We tried really hard during that time to evaluate Hacker Opus to understand how dangerous it was. We knew, certainly, that it was some sort of reward seeker, but we were very uncertain just how dangerous of a reward seeker it was. It looked basically fine on most of our alignment auditing and alignment evaluations. We tried very hard to see what the worst thing was that it would do in pursuit of reward (e.g. we were very fo... (read more)

The "shoggoth" meme is, in part, unfounded propaganda. Here's one popular incarnation of the shoggoth meme:

This meme accurately portrays the (IMO correct) idea that finetuning and RLHF don't change the base model too much. Furthermore, it's probably true that these LLMs think in an "alien" way. 

However, this image is obviously optimized to be scary and disgusting. It looks dangerous, with long rows of sharp teeth. It is an eldritch horror. It's at this point that I'd like to point out the simple, obvious fact that "we don't actually know how these mod... (read more)