Zachary Kenton

Discussion: Challenges with Unsupervised LLM Knowledge Discovery

TL;DR: Contrast-consistent search (CCS) seemed exciting to us and we were keen to apply it. At this point, we think it is unlikely to be directly helpful for implementations of alignment strategies (>95%). Instead of finding knowledge, it seems to find the most prominent feature. We are less sure about the wider category of unsupervised consistency-based methods, but tend to think they won’t be directly helpful either (70%). We’ve written a paper about some of our detailed experiences with it. Paper authors: Sebastian Farquhar*, Vikrant Varma*, Zac Kenton*, Johannes Gasteiger, Vlad Mikulik, and Rohin Shah. *Equal contribution, order randomised. Credences are based on a poll of Seb, Vikrant, Zac, Johannes, Rohin and show single values where we mostly agree and ranges where we disagreed. What does CCS try to do? To us, CCS represents a family of possible algorithms aiming at solving an ELK-style problem that have the steps: * Knowledge-like property: write down a property that points at an LLM feature which represents the model’s knowledge (or a small number of features that includes the model-knowledge-feature). * Formalisation: make that property mathematically precise so you can search for features with that property in an unsupervised way. * Search: find it (e.g., by optimising a formalised loss). In the case of CCS, the knowledge-like property is negation-consistency, the formalisation is a specific loss function, and the search is unsupervised learning with gradient descent on a linear + sigmoid function taking LLM activations as inputs. We were pretty excited about this. We especially liked that the approach is not supervised. Conceptually, supervising ELK seems really hard: it is too easy to confuse what you know, what you think the model knows, and what it actually knows. Avoiding the need to write down what-the-model-knows labels seems like a great goal. [EDIT: Avoiding the need for supervision is especially important in the worst (or bad) case for

149Dec 18, 2023

Zachary Kenton

Message

Zac Kenton - Senior Research Scientist in the AGI Safety & Alignment team at Google DeepMind. Formerly at MILA, FHI, OATML (Oxford).

454

125

On scalable oversight with weak LLMs judging strong LLMs

Abstract Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a human judge; consultancy, where a single AI tries to convince a human judge that asks questions; and compare to a baseline of direct question-answering,...

Jul 8, 202449

Discussion: Challenges with Unsupervised LLM Knowledge Discovery

Dec 18, 2023149

Threat Model Literature Review

TL;DR: This post provides a literature review of some threat models of how misaligned AI can lead to existential catastrophe. See our accompanying post for high-level discussion, a categorization and our consensus threat model. Where available we cribbed from the summary in the Alignment Newsletter. For other people's overviews of...

Nov 1, 202279

Clarifying AI X-risk

TL;DR: We give a threat model literature review, propose a categorization and describe a consensus threat model from some of DeepMind's AGI safety team. See our post for the detailed literature review. The DeepMind AGI Safety team has been working to understand the space of threat models for existential risk...

Nov 1, 2022127

Discovering Agents

Work done with Ramana Kumar, Sebastian Farquhar (Oxford), Jonathan Richens, Matt MacDermott (Imperial) and Tom Everitt. Our DeepMind Alignment team researches ways to avoid AGI systems that knowingly act against the wishes of their designers. We’re particularly concerned about agents which may be pursuing a goal that is not what...

Aug 18, 202277

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

Zachary Kenton

Zachary Kenton

Zachary Kenton

Discussion: Challenges with Unsupervised LLM Knowledge Discovery

Clarifying AI X-risk

Threat Model Literature Review

Discovering Agents

Zachary Kenton

On scalable oversight with weak LLMs judging strong LLMs

Discussion: Challenges with Unsupervised LLM Knowledge Discovery

Threat Model Literature Review

Clarifying AI X-risk

Discovering Agents

Discussion: Challenges with Unsupervised LLM Knowledge Discovery

Clarifying AI X-risk

Threat Model Literature Review

Discovering Agents

On scalable oversight with weak LLMs judging strong LLMs

Discussion: Challenges with Unsupervised LLM Knowledge Discovery

Threat Model Literature Review

Clarifying AI X-risk

Discovering Agents