ARC has published a report on Eliciting Latent Knowledge, an open problem which we believe is central to alignment. We think reading this report is the clearest way to understand what problems we are working on, how they fit into our plan for solving alignment in the worst case, and our research methodology.
The core difficulty we discuss is learning how to map between an AI’s model of the world and a human’s model. This is closely related to ontology identification (and othersimilarstatements). Our main contribution is to present many possible approaches to the problem and a more precise discussion of why it seems to be difficult and important.
The report is available here as a google document. If you're excited about this research, we're hiring!
Q&A
We're particularly excited about answering questions posted here throughout December. We welcome any questions no matter how basic or confused; we would love to help people understand what research we’re doing and how we evaluate progress in enough detail that they could start to do it themselves.
Thanks to María Gutiérrez-Rojas for the illustrations in this piece (the good ones, blame us for the ugly diagrams). Thanks to Buck Shlegeris, Jon Uesato, Carl Shulman, and especially Holden Karnofsky for helpful discussions and comments.
(Going to try my hand at Builder, but this is admittedly vague, so I hope you help sharpen it with criticism.)
What if instead of a "reporter", we had a "terrifier", whose adversarial objective is to highlight the additional "sensor" whose observations, assuming the input and actions were held constant, when viewed by a human would maximize the probability of a human reviewer saying the system would not performing as desired. The terrifier would be allowed to run the original predictor model "further" in order to populate whichever new components of the Bayes net are necessarily to populate in order to correctly simulate the output of the sensor. The terrifier could learn what facts would scare the human via multiple rounds of curiosity-driven human questioning (which can probably be done offline/beforehand). Then, we ultimately approve of the model only if the most terrifying thing the AI can show the human is less terrifying than the best-possible answers to the originally-posed questions.
To be clear, "sensor" here is standing in for any possible set of human-understandable observations about the world. Of course, if a human-understandable "sensor" is too restrictive, we can require only that the sensed state be understandable by AI-enhanced humans.
For example, say that the burglar's first action would be to tamper with the camera. But then, there must exist some hypothetical other location that a camera could be placed, that could still see the diamond being stolen - even though that new sensor is never actually revealed to the original simulator. If there are no possible "sensor"s we could add to the environment who results would terrify the human, that seems to me to be indistinguishable by definition from things just being OK. For example, if the thieves replace the diamond with an exact copy down to the nanometer but not molecular level... that's fine? I consider that behavior aligned.
How this defeats the previous counterexample: in some sense, I see the problem with "ontology mismatch" as the human no longer knowing the right question to ask, because the AI and human no longer agree on an ontology that would make the human's question meaningful. But "what question would the human be maximally upset about if they asked" is a pretty straightforward task to expect from an AI that's capable of modeling (most) human preferences.
Why did I frame this as adding a "sensor" instead of proposing a new question? Because we've framed the "reporter" as being learned from examples. If we directly asked the reporter to tell us the most terrifying answer to a question it could think of, it would just tell us something terrifying even if it wasn't "true" in the sense of being a "human translator". However, new "sensors" would be simulated via the original predictor model, and thus we don't have to rely on the translation step.
If the new concern is whether the AI's ability to model human preferences well enough (to know what human-legible sensors would terrify us), that just seems like a much easier problem. It doesn't take a genius AI to understand that a human wouldn't want to live in Cakey's surveillance state.
ARC has published a report on Eliciting Latent Knowledge, an open problem which we believe is central to alignment. We think reading this report is the clearest way to understand what problems we are working on, how they fit into our plan for solving alignment in the worst case, and our research methodology.
The core difficulty we discuss is learning how to map between an AI’s model of the world and a human’s model. This is closely related to ontology identification (and other similar statements). Our main contribution is to present many possible approaches to the problem and a more precise discussion of why it seems to be difficult and important.
The report is available here as a google document. If you're excited about this research, we're hiring!
Q&A
We're particularly excited about answering questions posted here throughout December. We welcome any questions no matter how basic or confused; we would love to help people understand what research we’re doing and how we evaluate progress in enough detail that they could start to do it themselves.
Thanks to María Gutiérrez-Rojas for the illustrations in this piece (the good ones, blame us for the ugly diagrams). Thanks to Buck Shlegeris, Jon Uesato, Carl Shulman, and especially Holden Karnofsky for helpful discussions and comments.