I've recently thought of a possibly simpler way of expressing the argument from the Occam's razor paper. Namely:
- Human biases and human preferences contain more combined information than human behaviour does. And more than the full human policy does.
Thus, in order to deduce human biases and preferences, we need more information than the human policy caries.
This extra information is contained in the "normative assumptions": the assumptions we need to add, so that an AI can learn human preferences from human behaviour.
We'd ideally want to do this with as few extra assumptions as possible. If the AI is well-grounded and understands what human concepts mean, we might be able to get away with a simple reference: "look through this collection of psychology research and take it as roughly true" could be enough assumptions to point the AI to all the assumptions it would need.