Here are some of the posts from the AI Alignment Forum writing day. Due to the participants writing 34 posts in less than 24 hours (!), I'm re-airing them to let people have a proper chance to read (and comment) on them, in roughly chronological order.

1) Computational Model: Causal Diagrams with Symmetry by Johns Wentworth

This post is about representing logic, mathematics, and functions with causal models.

For our purposes, the central idea of embedded agency is to take these black-box systems which we call “agents”, and break open the black boxes to see what’s going on inside.

Causal DAGs with symmetry are how we do this for Turing-computable functions in general. They show the actual cause-and-effect process which computes the result; conceptually they represent the computation rather than a black-box function.

I'm new to a lot of this, but to me this seemed like a weird and surprising way to think about math (e.g. the notion that the input $n$ is something that causes the next input $n - 1$ ). Seems like a very interesting set of ideas to explore.

2) Towards a mechanistic understanding of corrigibility by Evan Hubinger

This post builds off of Paul Christiano's post on Worst Case Guarantees. That post claims:

Even if we are very careful about how we deploy ML, we may reach the point where a small number of correlated failures could quickly become catastrophic... I think the long-term safety of ML systems requires being able to rule out this kind of behavior, which I’ll call unacceptable, even for inputs which are extremely rare on the input distribution.

Paul then proposes a procedure built around adversarial search, where one part of the system searches for inputs that produce unacceptable outputs in the trained agent, and talks more about how one might build such a system.

Evan's post tries to make progress on finding a good notion of acceptable behaviour from an ML system. Paul's post offers two conditions about the ease of choosing an acceptable action (in particular, that it should not stop the agent achieving a high average reward and that is shouldn't make hard problems much harder), but Evan's conditions are about the ease of training an acceptable model. His two conditions are:

It must be not that hard for an amplified overseer to verify that a model is acceptable.
It must be not that hard to f

...

List of Links

See Also