TL;DR: AI which is learning human values may act unethically or be catastrophically dangerous, as it doesn’t yet understand human values. The main idea is simple: a young AI which is trying to learn human values (which I will call a “value learner”) will have a “chicken and egg” problem....