Previously we derived a regret bound for DRL which assumed the advisor is "locally sane." Such an advisor can only take actions that don't lose any value in the long term. In particular, if the environment contains a latent catastrophe that manifests with a certain rate (such as the possibility of an UFAI), a locally sane advisor has to take the optimal course of action to mitigate it, since every delay yields a positive probability of the catastrophe manifesting and leading to permanent loss of value. This state of affairs is unsatisfactory, since we would like to have performance guarantees for an AI that can mitigate catastrophes that the human operator cannot mitigate on their own. To address this problem, we introduce a new form of DRL where in every hypothetical environment the set of uncorrupted states is divided into "dangerous" (impending catastrophe) and "safe" (catastrophe was mitigated). The advisor is then only required to be locally sane in safe states, whereas in dangerous states certain "leaking" of long-term value is allowed. We derive a regret bound in this setting as a function of the time discount factor, the expected value of catastrophe mitigation time for the optimal policy, and the "value leak" rate (i.e. essentially the rate of catastrophe occurrence). The form of this regret bound implies that in certain asymptotic regimes, the agent attains near-optimal expected utility (and in particular mitigates the catastrophe with probability close to 1), whereas the advisor on its own fails to mitigate the catastrophe with probability close to 1.

Appendix A proves the main theorem. Appendix B contains the proof of an important lemma which is however almost identical to what appeared in the previous essay. Appendix C contains several propositions from the previous essay which we are used in the proof. [Appendices B and C were moved to a separate post because of a length limit in the website.]

Results

We start by formalising the concepts of a "catastrophe" and "catastrophe mitigation" in the language of MDPs.

Definition 1

A catastrophe MDP is an MDP $M$ together with a partition of $S_{M}$ into subsets $S_{M} := S_{M}^{F} ⊔ S_{M}^{D} ⊔ S_{M}^{C}$ (safe, dangerous and corrupt states respectively).

Definition 2

Fix a catastrophe MDP $M$ . Define $A_{M}^{♯} : S_{M} \to 2^{A_{M}}$ by

$A_{M}^{♯} (s) := ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} {a \in A_{M} ∣ supp T (s, a) \subseteq S_{M}^{F}} if s \in S^{F} \end{matrix}$

...