Thanks. I think we mainly agree here.

Look at the paper linked for more details ( https://arxiv.org/abs/1712.05812 ).

Basically "humans are always fully rational and always take the action they want to" is a full explanation of all of human behaviour, that is strictly simpler than any explanation which includes human biases and bounded rationality.

But if you are expecting a 100% guarantee that the uncertainty metrics will detect every possible bad situation

I'm more thinking of how we could automate the navigating of these situations. The detection will be part of this process, and it's not a Boolean yes/no, but a matter of degree.

I agree that once you have landed in the bad situation, mitigation options might be much the same, e.g. switch off the agent.

I'm most interested in mitigation options the agent can take itself, when it suspects it's out-of-distribution (and without being turned off, ideally).

Thanks! Lots of useful insights in there.

So I might classify moving out-of-distribution as something that happens to a classifier or agent, and model splintering as something that the machine learning system does to itself.

Why do you think it's important to distinguish these two situations? It seems that the insights for dealing with one situation may apply to the other, and vice versa.

Cheers! My opinion on category theory has changed a bit, because of this post; by making things fit into the category formulation, I developed insights into how general relations could be used to connect different generalised models.

Did posts on generalised models as a category and how one can see Cartesian frames as generalised models.

Partial probability distribution

A concept that's useful for some of my research: a partial probability distribution.

That's a $Q$ that defines $Q(A\mid B)$ for some but not all $A$ and $B$ (with $Q\left(A\right)=Q(A\mid \Omega )$ for $\Omega$ being the whole set of outcomes).

This $Q$ is a partial probability distribution iff there exists a probability distribution $P$ that is equal to $Q$ wherever $Q$ is defined. Call this $P$ a full extension of $Q$.

Suppose that $Q(C\mid D)$ is not defined. We can, however, say that $Q(C\mid D)=x$ is a logical implication of $Q$ if all full extension $P$ has $P(C\mid D)=x$.

Eg: $Q\left(A\right)$, $Q\left(B\right)$, $Q(A\cup B)$ will logically imply the value of $Q(A\cap B)$.

I like it. I'll think about how it fits with my ways of thinking (eg model splintering).

Cheers; Rebecca likes the "instrumental control incentive" terminology; she claims it's more in line with control theory terminology.

We agree that lack of control incentive on X does not mean that X is safe from influence from the agent, as it may be that the agent influences X as a side effect of achieving its true objective. As you point out, this is especially true when X and a utility node probabilistically dependent.

I think it's more dangerous than that. When there is mutual information, the agent can learn to behave as if it was specifically manipulating X; the counterfactual approach doesn't seem to do what it intended.