Is there any meaningful distinction between filtered evidence and lying? I know that in toy models these can be quite different, but in the expansive setting here, where the speaker can select the most misleading technically true fact, is there any major difference?

And how would the results here look if we expended it to allow the speaker to lie?

Sounds good, cheers!

There are some minor differences; your approach learns the whole model, whereas mine assumes the model is given, and learns only the "acausalish" aspects of it. But they are pretty similar.

One problem you might have, is learning the acausal stuff in the mid-term. If the agent learns that causality exists, and then that in the Newcomb problem is seems to have a causal effect, then it may search a lot for the causal link. Eventually this won't matter (see here), but in the mid-term it might be a problem.

Or not. We need to test more ^_^

Very impressive work, both the output and how you iterate on it.

Some thoughts about the cross-examination issue, prompted by your "Implementation 2 for human debaters: teams of two". It occurred to me that B* could win if it could predict A and B's future behaviour, and match up it's answer with B.

I'd prefer that such an option not exist; that B could answer the question directly, without needing to rewind. Hence prediction won't help.

Cross-examination still helps: A can cross examine as soon as they suspect B is shielding behind an ambiguity. This means that A might have to abandon their current question line, and start again on the other one. This seems more secure (if longer).

I mainly mentioned child-rulers because Robin was using that example; and I used "getting deposed" as an example of agency problems that weren't often (ever?) listed in the economics literature.

More seriously, the reason I'm sceptical of impact measures is because it feels that they all fail for the same reason. Unfortunately, I can't articulate that reason; it's the result of a long history of trying to build impact measures and trying to break them. I just have a feel for where the weaknesses are. So I knew that subagents would be a problem for AUP, long before I could articulate it formally.

But, as I said, I unfortunately can't formalise this feeling; it remains personal.

For this example, it was harder than usual to come up with a counter-example. And I was surprised that half of AUP survived fine - I would not have expected that a restriction against lowering your power would be unhackable. So consider these mild positives for your approach.

But my instinctive feeling remains: I hope that AUP can be made to work for superintelligences, but I expect that it won't :-(

But if you look back at the history of impact measures and AUP, you’ll see lots of skulls; people say “this problem dooms AUP”, and I say “I think we’re talking about conceptually different things and that you’re a little overconfident; probably just a design choice issue”. It then ends up being a solvable design choice issue. So by Laplace’s Rule of Succession, I’d be surprised if this were The Insurmountable Problem That Dooms AUP.

But if you look back at the history of impact measures, you'll see a lot of skulls; people say "this impact measure will work", and I say "I think you're overconfident; I'm sure I can find a way it fails". It then ends up being a solvable "break the impact measure" problem. So by Laplace’s Rule of Succession, I’d be surprised if this approach were The Ultimate Impact Measure That Works.

;-)

I agree I should re-do these all with examples. But, to answer your questions as far as possible:

What $A$ will do: $\varnothing$ forever (maybe with some extra info for the $SA$).

What $A$ would counterfactually do (to define $V_k\left(s_{t+\tau }^{\left(t\right)}\right)$): ${\pi }_0$ for $\tau$ turns (the ${\pi }_0$ rollout, to define $s_{t+\tau }^{\left(t\right)}$) followed by ${\pi }_R^{\ast }$ (to define the $V_k$ aspect). Call this ${\pi }_{0,\tau ,k}^{\ast }$.

The penalty is defined in terms of these $V_k\left(s_{t+\tau }^{\left(t\right)}\right)$, entirely counterfactual except for $\tau =0$.

What $SA$ must do: iff $A$ follows ${\pi }_{0,\tau ,k}^{\ast }$, then $SA$ must ensure that $V_k\left(s_{t+\tau }^{\left(t\right)}\right)=V_k\left(s_{t+\tau }^{\prime \left(t\right)}\right)$.

So a key question is: how early can $SA$ detect if $A$ is following ${\pi }_{0,\tau ,k}^{\ast }$? If it can detect this immediately, then it can immediately implement the policy for ensuring that $V_k\left(s_{t+\tau }^{\left(t\right)}\right)=V_k\left(s_{t+\tau }^{\prime \left(t\right)}\right)$ (for the stricter impact measure) or the much easier $B{V}_k\left(s_t\right)=B{V}_k\left(s_t^{\prime }\right)$ (for the original impact measure).

If it has to wait $\tau$ turns to be sure, then it needs to preserve its power to control many different $k$.

Now, $A$ will never actually follow ${\pi }_{0,\tau ,k}^{\ast }$, so the penalty is (almost) entirely counterfactual. But the counterfactual abilities of $SA$ will determine the penalty; $SA$ is rewarded for quickly gaining control of the rewards.

In practice, $A$ will signal early on that it's not following ${\pi }_{0,\tau ,k}^{\ast }$. Maybe there's a very small impact action, ${\varnothing }^{ϵ}$, with a tiny penalty. Then $A$ would like to use that to signal to $SA$ that its not in the counterfactual world where it is following ${\pi }_{0,\tau ,k}^{\ast }$. So long as ${\pi }_0$'s first action isn't ${\varnothing }^{ϵ}$, that signal is enough (of course, if ${\pi }_0$'s first action isn't $\varnothing$, then $\varnothing$ is a sufficient signal).

After getting that signal, the subagent is (almost) completely unconstrained, because the constraints are counterfactual, in a world it knows it isn't in.

The one mild constraint remaining comes from $s_{t+0}^{\left(t\right)}=s_t$, which is not a counterfactually defined state (though $V_k\left(s_t\right)$ is a counterfactually defined property of that state).

I'll try and write up a further clarification of this.

Alas, the inaction rollouts don't seem to fix the problem: https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/z9MfmF8gA7SBxGSmb

For example, if there is an auxiliary reward for reaching any state except the state before subagent creation, the no-subagent inaction rollout will consist of this state, and the subagent inaction rollout will contain a different state at least once, so subagent creation will be penalized.

This requires identifying what a subagent is in general, a very tricky unsolved problem (which I feel is unsolvable).

There's another issue; it's not enough to show that the subagent triggers a penalty. We need to show the penalty is larger than not creating the subagent. Since the penalty is zero after the subagent is created, and since the subagent has very fine control over the rewards (much finer than actions that don't include creating an intelligent being), creating a subagent might be lower penalty than almost any other action.

It won't be a lower penalty than the agent doing nothing for ever, of course. But we typically want the agent to do something, so will calibrate the penalty or R_0 for that. And it's plausible that creating the subagent will have lower penalty (and/or higher R_0) than any safe "something".