All of Capybasilisk's Comments + Replies

Near the beginning, Daniel is basically asking Jan how they plan on aligning the automated alignment researcher, and if they can do that, then it seems that there wouldn't be much left for the AAR to do.

Jan doesn't seem to comprehend the question, which is not an encouraging sign.

I think I probably didn't quite word that question right, and that's what's explaining the confusion - I meant something like "Once you've created the AAR, what alignment problems are left to be solved? Please answer in terms of the gap between the AAR and superintelligence."

I'd especially like to hear your thoughts on the above proposal of loss-minimizing a language model all the way to AGI.

I hope you won't mind me quoting your earlier self as I strongly agree with your previous take on the matter:

If you train GPT-3 on a bunch of medical textbooks and prompt it to tell you a cure for Alzheimer's, it won't tell you a cure, it will tell you what humans have said about curing Alzheimer's ... It would just tell you a plausible story about a situation related to the prompt about curing Alzheimer's, based on its training data. Ra

... (read more)
4Charlie Steiner1y
Ah, the good old days post-GPT-2 when "GPT-3" was the future example :P I think back then I still thoroughly understimated how useful natural-language "simulation" of human reasoning would be. I agree with janus that we have plenty of information telling us that yes, you can ride this same training procedure to very general problem solving (though I think including more modalities, active leaning, etc. will be incorporated before anyone really pushes brute force "GPT-N go brrr" to the extreme). This is somewhat of a concern for alignment. I more or less stand by that comment you linked and its children; in particular, I said Simulating a reasoner who quickly finds a cure for Alzheimer's is not by default safe (even though simulating a human writing in their diary is safe). Optimization processes that quickly find cures for Alzheimer's are not humans, they must be doing some inhuman reasoning, and they're capable of having lots of clever ideas with tight coupling to the real world. I want to have confidence in the alignment properties of any powerful optimizers we unleash, and I imagine we can gain that confidence by knowing how they're constructed, and trying them out in toy problems while inspecting their inner workings, and having them ask humans for feedback about how they should weigh moral options, etc. These are all things it's hard to do for emergent simulands inside predictive simulators. I'm not saying it's impossible for things to go well, I'm about evenly split on how much I think this is actually harder, versus how much I think this is just a new paradigm for thinking about alignment that doesn't have much work in it yet.

Charlie's quote is an excellent description of an important crux/challenge of getting useful difficult intellectual work out of GPTs.

Despite this, I think it's possible in principle to train a GPT-like model to AGI or to solve problems at least as hard as humans can solve, for a combination of reasons:

  1. I think it's likely that GPTs implicitly perform search internally, to some extent, and will be able to perform more sophisticated search with scale.
  2. It seems possible that a sufficiently powerful GPT trained on a massive corpus of human (medical + other) k
... (read more)
2Vladimir Nesov1y
I think talking of "loss minimizing" is conflating two different things here. Minimizing training loss is alignment of the model with the alignment target given by the training dataset. But the Alzheimer's example is not about that, it's about some sort of reflective equilibrium loss, harmony between the model and hypothetical queries it could in principle encounter but didn't on the trainings dataset. The latter is also a measure of robustness. Prompt-conditioned behaviors of a model (in particular, behaviors conditioned by presence of a word, or name of a character) could themselves be thought of as models, represented in the outer unconditioned model. These specialized models (trying to channel particular concepts) are not necessarily adequately trained, especially if they specialize in phenomena that were not explored in the episodes of the training dataset. The implied loss for an individual concept (specialized prompt-conditioned model) compares the episodes generated in its scope by all the other concepts of the outer model, to the sensibilities of the concept. Reflection reduces this internal alignment loss by rectifying the episodes (bargaining with the other concepts), changing the concept to anticipate the episodes' persisting deformities, or by shifting the concept's scope to pay attention to different episodes. With enough reflection, a concept is only invoked in contexts to which it's robust, where its intuitive model-channeled guidance is coherent across the episodes of its reflectively settled scope, providing acausal coordination among these episodes in its role as an adjudicator, expressing its preferences. So this makes a distinction between search and reflection in responding to a novel query, where reflection might involve some sort of search (as part of amplification), but its results won't be robustly aligned before reflective equilibrium for the relevant concepts is established.