We’re grateful to our advisors Nate Soares, John Wentworth, Richard Ngo, Lauro Langosco, and Amy Labenz. We're also grateful to Ajeya Cotra and Thomas Larsen for their feedback on the contests.

TLDR: AI Alignment Awards is running two contests designed to raise awareness about AI alignment research and generate new research proposals. Prior experience with AI safety is not required. Promising submissions will win prizes up to $100,000 (though note that most prizes will be between $1k-$20k; we will only award higher prizes if we receive exceptional submissions.)

You can help us by sharing this post with people who are or might be interested in alignment research (e.g., student mailing lists, FB/Slack/Discord groups.)

What are the contests?

We’re currently running two contests:

Goal Misgeneralization Contest (based on Langosco et al., 2021): AIs often learn unintended goals. Goal misgeneralization occurs when a reinforcement learning agent retains its capabilities out-of-distribution yet pursues the wrong goal. How can we prevent or detect goal misgeneralization?

Shutdown Problem Contest (based on Soares et al., 2015): Given that powerful AI systems might resist attempts to turn them off, how can we make sure they are open to being shut down?

What types of submissions are you interested in?

For the Goal Misgeneralization Contest, we’re interested in submissions that do at least one of the following:

Propose techniques for preventing or detecting goal misgeneralization
Propose ways for researchers to identify when goal misgeneralization is likely to occur
Identify new examples of goal misgeneralization in RL or non-RL domains. For example:
1. We might train an imitation learner to imitate a "non-consequentialist" agent, but it actually ends up learning a more consequentialist policy.
2. We might train an agent to be myopic (e.g., to only care about the next 10 steps), but it actually learns a policy that optimizes over a longer timeframe.
Suggest other ways to make progress on goal misgeneralization

For the Shutdown Problem Contest, we’re interested in submissions that do at least one of the following:

Propose ideas for solving the shutdown problem or designing corrigible AIs. These submissions should also include (a) explanations for how these ideas address core challenges raised in the corrigibility paper and (b) possible limitations

...

Posts

Wikitag Contributions

Comments