Epistemic status: I predict that people who focus on prosaic AI alignment have thought of this before, in some way at least. But I don’t know what they would say in response, so I’m writing this up so I can find out! I’m making it a post instead of an email so that the discussion can be public.

Characterization of prosaic AI alignment: Prosaic AI methods—the sort of methods that we are using today, rather than hypothetical new methods based on a deeper understanding of intelligence—might be sufficient to make human-level AGI in the next two decades or so, and that if this happens we’d better be prepared. Thus we should think about how to take prosaic AI methods and combine them or modify them in various ways to make something which is as competitive, or almost as competitive, as cutting-edge AI. Examples of this approach are debate, imitating humans, preference learning, and iterated distillation and amplification.

Conjecture: Cutting-edge AI will come from cutting-edge algorithms/architectures trained towards cutting-edge objectives (incl. unsupervised learning) in cutting-edge environments/datasets. Anything missing one or more of these components will suffer a major competitiveness penalty.

Example: Suppose that the best way we know of to get general intelligence is to evolve a population of giant neural nets with model-free learning in a part-competitive, part-cooperative, very diverse environment consisting of an ensemble of video games. One year, the systems that come out of this process are at dog level, then two years later they are at chimpanzee level, then two years later they are at IQ-80 human level… It is expected that scaling up this sort of thing will lead, in the next few years, to smarter-than-human AGI.

The Dilemma: Choose plan 1 or plan 2:

Plan 1: Train a system into your scheme from scratch, using cutting-edge algorithms but not cutting-edge environments or objectives. (The environments and objectives are whatever your safety scheme calls for, e.g. debates, imitating humans, a series of moral choice situations, etc.)

Example: We take the best training algorithms and architecture we can find, but instead of training on an ensemble of video games, our AI is trained from scratch to win debates with human judges. We then have it debate on important topics to give us valuable information.
Problem with

...

Prosaic Alignment

Prosaic Alignment