But is that true? Human behavior has a lot of information. We normally say that this extra information is irrelevant to the human's beliefs and preferences (i.e. the agential model of humans is a simplification), but it's still there.
Look at the paper linked for more details ( https://arxiv.org/abs/1712.05812 ).
Basically "humans are always fully rational and always take the action they want to" is a full explanation of all of human behaviour, that is strictly simpler than any explanation which includes human biases and bounded rationality.
I've recently thought of a possibly simpler way of expressing the argument from the Occam's razor paper. Namely:
Thus, in order to deduce human biases and preferences, we need more information than the human policy caries.
This extra information is contained in the "normative assumptions": the assumptions we need to add, so that an AI can learn human preferences from human behaviour.
We'd ideally want to do this with as few extra assumptions as possible. If the AI is well-grounded and understands what human concepts mean, we might be able to get away with a simple reference: "look through this collection of psychology research and take it as roughly true" could be enough assumptions to point the AI to all the assumptions it would need.