• The problems of embedded agency are due to the notion of agency implicit in reinforcement learning being a leaky abstraction.
  • Machine learning problem statements often makes assumptions that are known to be false, for example, assuming i.i.d. data.
  • Examining failure modes that result from false assumptions and leaky abstractions is important for safety, (at least) because they create additional possibilities for convergent rationality.
  • Attempting to enforce the assumptions implicit in machine learning problem statements is another important topic for safety research, since we do not fully understand the failure modes.
  • In practice, most machine learning research is done in settings where unrealistic assumptions are trivially enforced to a sufficiently high extent that it is reasonable to assume they are not violated (e.g. by the use of a fixed train/valid/test set, generated via pseudo-random uniform sampling from a fixed dataset).
  • We can (and probably should) do machine learning research that targets failure modes of common assumptions and methods of enforcing assumptions by (instead) creating settings in which these assumptions have the potential to be violated.
New Comment
1 comment, sorted by Click to highlight new comments since:

A few more important examples of important leaky abstractions that we might worry about protecting/enforcing:

  • Casual interventions (as "uncaused causes", ala free will).
  • Boxes that don't leak information (BoMAI)

Making a more complete list would be a good project