x

AI ALIGNMENT FORUM
AF

William_S — AI Alignment Forum

William Saunders

Top postsTop post

William Saunders

Message

I worked at OpenAI for three years, from 2021-2024 on the Alignment team, which eventually became the Superalignment team. I worked on scalable oversight, part of the team developing critiques as a technique for using language models to spot mistakes in other language models. I...

1869

Ω

238

8

247

11y

William Saunders

I worked at OpenAI for three years, from 2021-2024 on the Alignment team, which eventually became the Superalignment team. I worked on scalable oversight, part of the team developing critiques as a technique for using language models to spot mistakes in other language models. I...

Top postsTop post

Prize for Alignment Research Tasks

Can AI systems substantially help with alignment research before transformative AI? People disagree. Ought is collecting a dataset of alignment research tasks so that we can: 1. Make progress on the disagreement 2. Guide AI research towards helping with alignment We’re offering a prize of $200-$2000 for each contribution to this dataset. The debate: Can AI substantially help with alignment research? Wei Dai asked the question in 2019: > [This] comparison table makes Research Assistant seem a particularly attractive scenario to aim for, as a stepping stone to a more definitive [AI Safety] success story. Is this conclusion actually justified? Jan Leike thinks so: > My currently favored approach to solving the alignment problem: automating alignment research using sufficiently aligned AI systems. It doesn’t require humans to solve all alignment problems themselves, and can ultimately help bootstrap better alignment solutions. Paul Christiano agrees: > Building weak AI systems that help improve alignment seems extremely important to me and is a significant part of my optimism about AI alignment. [...] Overall I think that "make sure we are able to get good alignment research out of early AI systems" is comparably important to "do alignment ourselves." Realistically I think the best case for "do alignment ourselves" is that if "do alignment" is the most important task to automate, then just working a ton on alignment is a great way to automate it. But that still means you should be investing quite a significant fraction of your time in automating alignment. Eliezer doesn't: > "AI systems that do better alignment research" are dangerous in virtue of the lethally powerful work they are doing, not because of some particular narrow way of doing that work. If you can do it by gradient descent then that means gradient descent got to the point of doing lethally dangerous work. Asking for safely weak systems that do world-savingly strong tasks is almost everywhe

Machine Learning Projects on IDA

HCH is not just Mechanical Turk

Understanding Iterated Distillation and Amplification: Claims and Oversight

William_S's Shortform

Mar 22, 2023•5

Thoughts on refusing harmful requests to large language models

https://twitter.com/antimatter15/status/1602469101854564352 Currently, large language models (ChatGPT, Constitutional AI) are trained to refuse to follow user requests that are considered inappropriate or harmful. This can be done by training on example strings of the form “User: inappropriate request AI: elaborate apology” Proposal Instead of training a language model to produce “elaborate...

Jan 19, 2023•32

Prize for Alignment Research Tasks

Can AI systems substantially help with alignment research before transformative AI? People disagree. Ought is collecting a dataset of alignment research tasks so that we can: 1. Make progress on the disagreement 2. Guide AI research towards helping with alignment We’re offering a prize of $200-$2000 for each contribution to...

Apr 29, 2022•64

Machine Learning Projects on IDA

TLDR We wrote a 20-page document that explains IDA and outlines potential Machine Learning projects about IDA. This post gives an overview of the document. What is IDA? Iterated Distillation and Amplification (IDA) is a method for training ML systems to solve challenging tasks. It was introduced by Paul Christiano....

Jun 24, 2019•49

Reinforcement Learning in the Iterated Amplification Framework

When I think about Iterated Amplification (IA), I usually think of a version that uses imitation learning for distillation. This is the version discussed in the Scalable agent alignment via reward modeling: a research direction, as "Imitating expert reasoning", in contrast to the proposed approach of "Recursive Reward Modelling". The...

Feb 9, 2019•25

HCH is not just Mechanical Turk

HCH, introduced in Humans consulting HCH, is a computational model in which a human answers questions using questions answered by another human, which can call other humans, which can call other humans, and so on. Each step in the process consists of a human taking in a question, optionally asking...

Feb 9, 2019•42

Understanding Iterated Distillation and Amplification: Claims and Oversight

[Background: Intended for an audience that has some familiarity with Paul Christiano’s approach to AI Alignment. Understanding Iterated Distillation and Amplification should provide sufficient background.] [Disclaimer: When I talk about “what Paul claims”, I am only summarizing what I think he means through reading his blog and participating on discussions...

Apr 17, 2018•36

Load More (7/10)