This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program. Where not indicated, ideas are summaries of the documents mentioned, not original contributions. I am thankful for the encouraging approach of the organizers of the program, especially Oliver Zhang.

TL;DR

In the post “The case for aligning narrowly superhuman models”, Ajeya Cotra advocates for performing technical alignment work on currently available large models using human feedback, in domains where (some) humans are already outperformed (in some aspects). The direct benefit of this line of work could be testing out conceptual work, highlighting previously unknown challenges and eliciting new scalable, general solutions, as well as indirectly moving the field of ML in a better direction and building the community and infrastructure/tooling of AI Safety. This post summarizes the original post, some related work and some objections.

Short intro

Progress in ML could allow us to do technical alignment research that is closer to the "real problem" than specific toy examples. In particular, one of the main promising directions (among others such as Interpretability and Truthful and honest AI) is discovering better methods of giving feedback to models more capable than us. This might be easy for some concrete, narrow tasks, such as playing Go (there is an algorithmic way to decide if a model is playing Go better than another model), but for more “fuzzy” tasks (such as “devise a fair economic policy”), we can’t evaluate the model according to such a gold standard.

A couple of examples of work falling into this category (See Example tasks):

Train a language model to accurately explain things about a field that the feedback providers are not familiar with.
Train an RL agent to act well in an environment where the RL agent can observe more information than the feedback providers can.
Train a multilingual model to translate between English and a foreign language that the feedback providers do not know. [Summary after Alignment Newsletter]

These tasks call for more sophisticated ways of generating feedback for the models. Some of the approaches that can be used include debate or decomposing the question into subtasks. See section Possible approaches to alignment.

To test these methods,...

Posts

Wikitag Contributions

Comments