All of Ramana Kumar's Comments + Replies

That's right. A partial function can be thought of as a subset (of its domain) and a total function on that subset. And a (total) function can be thought of as a partition (of its domain): the parts are the inverse images of each point in the function's image.

Blackwell’s theorem says that the conditions under which ${\kappa }_1$ can be said to be more generally useful than ${\kappa }_2$ are precisely the situations where ${\kappa }_1$ is a post-garbling of ${\kappa }_2$.

Are the indices the wrong way around here?

A formalisation of the ideas in this sequence in higher-order logic, including machine verified proofs of all the theorems, is available here.

"subagent [$C$] that could choose $U$" -- do you mean $U\subseteq Ctrl\left(C\right)$ or $U\subseteq Ensure\left(C\right)$ or neither of these? Since $Ctrl$ is not closed under unions, I don't think the controllables version of "could choose" is closed under coarsening the partition. (I can prove that the ensurables version is closed; but it would have been nice if the controllables version worked.)

ETA: Actually controllables do work out if I ignore the degenerate case of a singleton partition of the world. This is because, when considering partitions of the world, ensur... (read more)

I have something suggestive of a negative result in this direction:

Let $C$ be the prime-detector situation from Section 2.1 of the coarse worlds post, and let $p:W\to W$ be the (non-surjective) function that "heats" the outcome (changes any "C" to an "H"). The frame $p^{\circ }\left(C\right)$ is clearly in some sense equivalent to the one from the example (which deletes the temperature from the outcome) -- I am using my version just to stay within the same category when comparing frames. As a reminder, primality is not observable in $C$ but is ob... (read more)

With the other problem resolved, I can confirm that adding an $A=\varnothing$ escape clause to the multiplicative definitions works out.

Using the idea we talked about offline, I was able to fix the proof - thanks Rohin!
Summary of the fix:
When $D_1$ and $D_2$ are defined, additionally assume they are biextensional (take their biextensional collapse), which is fine since we are trying to prove a biextensional equivalence. (By the way this is why we can't take $b_1=b_2$, since we might have $A\supseteq B_1\ne B_2\subseteq A$ after biextensional collapse.) Then to prove $h=h_{f_1}$, observe that for all $b\in B_1$, $b{\bullet }_{1}h\left(b_2^{\prime }\right)=b{\bullet }_{1}h\left(b_2\right)$ which means $b{\star }_{1}h\left(b_2^{\prime }\right)=b{\star }_1{f}_1$, hence $h\left(b_2^{\prime }\right)=$... (read more)

I presume the fix here will be to add an explicit $A=\varnothing$ escape clause to the multiplicative definitions. I haven't been able to confirm this works out yet (trying to work around this), but it at least removes the $null$ counterexample.

How is this supposed to work (focusing on the $h=h_{f_1}$ claim specifically)?

and so

Thus, $(g,h)=(g_{f_1},h_{f_1})$.

Earlier, $h_{f_1}$ was defined as follows:

given by $g_{f_1}\left(b_1\right)=b_1{\cdot }_1{f}_1$ and $h_{f_1}\left(b_2\right)=f_1$

but there is no reason to suppose $f_1=r$ above.

It suffices to establish that $\text{Ensure}\left(C_{T_i}\right)\supseteq \text{Ensure}\left(C_{T_j}\right)$

I think the $T_i$ and $T_j$ here are supposed to be $V$ and $U$

Indeed I think the $A=\varnothing$ case may be the basis of a counterexample to the claim in 4.2. I can prove for any (finite) $W$ with $|W|>1$ that there is a finite partition $V$ of $W$ such that $C$'s agent observes $V$ according to the assuming definition but does not observe $V$ according to the constructive multiplicative definition, if I take $C=null.$

Let $H=\text{hom}(B_1\&1_{R_2\cup R_3},(D_2\&1_{R_1\cup R_3}{)}^{\ast })$

nit: $B_1$ should be $D_1$ here

and let $b_2$ be an element of $b_2$.

and the second $b_2$ should be $B_2$. I think for these $b_1$ and $b_2$ to exist you might need to deal with the $A=\varnothing$ case separately (as in Section 5). (Also couldn't you just use the same $b$ twice?)

UPDATE: I was able to prove ${Assume}_{S_1}\left(C\right)\&{Assume}_{S_2}\left(C\right)\simeq {Assume}_{S_1\cup S_2}\left(C\right)$ in general whenever $S_1$ and $S_2$ are disjoint and both in $Obs\left(C\right)$, with help from Rohin Shah, following the "restrict attention to world $S_1\cup S_2$" approach I hinted at earlier.
 

this is clearly isomorphic to $D_1\&\dots \&D_m$, where $D_i=(B_i,F,{\star }_i)$, where $b{\star }_{i}f=b\star f$. Thus, $C$'s agent can observe $V$ according to the nonconstructive additive definition of observables.

I think this is only true if $V_B$ partitions $W$, or, equivalently, if $v_B$ is surjective. This isn't shown in the proof. Is it supposed to be obvious?

EDIT: may be able to fix this by assigning any $s\in V$ that is not in $V_B$ to the frame $\top$ so it is harmless in the product of $D_i$s -- I will try thi... (read more)

And it seems we do actually need ${Assume}_{S_1}\left(C\right)\&{Assume}_{S_2}\left(C\right)\simeq {Assume}_{S_1\cup S_2}\left(C\right)$ in the proof to justify:

Thus it suffices to show that $\left(D_1\&1_{R_2\cup R_3}\right)\otimes \left(D_2\&1_{R_1\cup R_3}\right)\simeq D_1\&D_2\&1_{R_3}$.

Without it, we have to show $\left(D_1\&1_{R_2\cup R_3}\right)\otimes \left(D_2\&1_{R_1\cup R_3}\right)\simeq {Assume}_{S_1\cup S_2}\left(C\right)\&1_{R_3}$ instead.

Because $S_1$ and $S_2$ are not a partition of the world here.

EDIT: but what we actually need in the proof is ${Assume}_{S_1}\left(C\right)\&{Assume}_{S_2}\left(C\right)\&\cdots \simeq {Assume}_{S_1\cup S_2}\left(C\right)\&\dots$ where the $\dots$ do result in a partition, so I think this will work out the same as the other comment. I'm still not convinced about biextensional equivalence between the frames without the rest of the product.

I haven't yet figured out why it's true under $\simeq$ - I'll keep trying, but let me know if there's a quick argument for why this holds. (Default next step for me would be to see if I can restrict attention to the world $S_1\cup S_2$ then do something similar to my other comment.)

and observe that ${\text{Assume}}_{S_1\cup S_2}\left(C\right)\cong D_1\&D_2$

This cannot be true. I can prove in general ${Assume}_{S_1\cup S_2}\left(C\right)≆{Assume}_{S_1}\left(C\right)\&{Assume}_{S_2}\left(C\right)$ whenever $|Agent\left(C\right)|>1$ by observing that the agent cardinalities on each side differ.

where $D_1={\text{Assume}}_{S_1}$(C), $D_1={\text{Assume}}_{S_1}\left(C\right)$

Presumably two of those indices should be $2$

Let $C\simeq {\text{Assume}}_{S_1}\left(C\right)\&\dots \&{\text{Assume}}_{S_n}\left(C\right)$. Thus, we also have that $C\simeq {\text{Assume}}_{S_1\cup S_2}\left(C\right)\&{\text{Assume}}_{S_3}\left(C\right)\&\dots \&{\text{Assume}}_{S_n}\left(C\right)$

I'm not seeing why this follows. I'll look for a counterexample, but in the meantime maybe there's a simple explanation for why we can combine the product of two assumes as an assume of the union? (I think the only relevant assumption in this context is that the $S_i$s partition the world; but I might be missing some other important assumption.)

EDIT: I can see how maybe this will follow from the definition of observability of a... (read more)

Let $C_1=(A_1,E_1,{\cdot }_1)$, and let $D_1=(B_1,F_1,{\star }_1)=C_1\otimes \cdots \otimes C_n$

Is $D_1$ supposed to be $C_2\otimes \dots$ here, rather than including $C_1$?

Next, assume that $C$'s agent can observe $V$ according to the additive definition. We will show that $C$'s agent can observe $S_1$.

I might be misunderstanding this, but the proof suggests you're actually assuming the assuming definition here, not the additive definition. In which case we may be missing the proof of implication of any of the other definitions from the additive definition.

Definition: We say that $C$'s agent can observe a finite partition $V$ of $W$ if for all functions $f:V\to A$, there exists an element $a_f\in A$ such that for all $e\in E$, $f\left(v\right(a_f\cdot e\left)\right)\cdot e=a_f\cdot e$.

Claim: This definition is equivalent to the definition from subsets.

This doesn't hold in the degenerate case $W=\varnothing$, since then we have an empty function $f$ but no elements of $A$. (But the definition from subsets holds trivially.)

Claim: For any Cartesian frame $C=(A,E,\cdot )$ over $W$ and partition $V$ of $W$, let $v:W\to V$ send each element of $W$ to its part in $V$. If for all $a_0,a_1\in A$ and $e\in E\text{,}$ we have $v(a_0\cdot e)=v(a_1\cdot e)$, then ${\text{External}}_V\left(C\right)\cong C$.

I think this may also need to assume that $A$ is non-empty.

If ${\text{External}}_V=(A^{\prime },E^{\prime },{\cdot }^{\prime })$

The $C$ argument is missing in several places like this from 4.2 onwards.

$h_i^{\prime \prime }:A_{1-i}/B_{1-i}\times E_{1-i}\to A_i/B_i\times E_i$ is given by $h_i^{\prime }(b,e)=\left(f_{1-i}\right(b),h_i(e\left)\right)$

There's a prime missing on $h_i^{\prime }$. I'd also have expected $q$ instead of $b$ as the variable (doesn't affect correctness).

while $h_i^{\prime }:B_{1-i}\times E_{1-i}\to B_i\times E_i$ is given by $h_i^{\prime }(b,e)=({\iota }_{1-i},h_i(e\left)\right)$

${\iota }_{1-i}$ should be applied to $b$ above, I think

Let ${\beta }_i:A_i\to B_i$, send each element of $A_i$ to its part in $B_i$, so $\alpha \left(a\right)=\{a^{\prime }\in A_i|\forall e\in E_i,v(a^{\prime }\cdot e)=v(a\cdot e\left)\right\}$.

Presuming the $\alpha$ here should be a ${\beta }_i$

Oh I see this has also been fixed. Thanks!

This is also suspicious in section 2.2 about Assuming. I think it should be the other way around and about Assume rather than Commit, and I don't think that's equivalent to what's written here. (But I'm not confident about this.)

Claim: For all $S\subseteq W$, ${\text{Commit}}_S\left(C\right){◃}_{+}^{\ast }C$ and ${\text{Commit}}_{\setminus S}\left(C\right){◃}_{+}^{\ast }C$.

Claim: For all $F\subseteq E$, ${\text{Assume}}^F\left(C\right){◃}_{+}^{\ast }C$ and ${\text{Assume}}^{\setminus F}\left(C\right){◃}_{+}^{\ast }C$. 

Are these the wrong way around?

I believe $C{◃}_{+}^{\ast }{Assume}^F\left(C\right)$ is indeed trivial, but the opposite is less obvious.

Similarly, for all $B\subset A$, $\left({\text{Commit}}^B\right(C){)}^{\ast }\cong {\text{Assume}}^B(C^{\ast })$ and $\left({\text{Commit}}^{\setminus B}\right(C){)}^{\ast }\cong {\text{Assume}}^{\setminus B}(C^{\ast })$.

We don't need $B$ to be a proper subset here (i.e., I think $\subset$ is a typo for $\subseteq$). Also in my view all the isomorphisms in this section are actually equalities (but it's also reasonable to never consider equality of frames).

We let $C_2=(A,F,\diamond )$, where $a\diamond f=g\left(a\right)\star f=a\cdot h\left(b\right)$. 

minor typo: I think $b$ should be $f$ above.

I'm not sure why you started using the equivalence symbol on morphisms, e.g.,

 $\varphi \cong {\varphi }_1\circ {\varphi }_0$

in the categorical definition of multiplicative subagent.

I think equality ($=$) is the correct concept to be using here instead, as in the categorical definition of (original) subagent.

and let $(g_3,h_3):D\to (Y,Z,\diamond )$ and $(g_4,h_4):(Y,Z,\diamond )\to D$ compose to something homotopic to the identity in both orders.

Pretty sure the $\diamond$s should be $\bullet$s here (though they're notation for the same function anyway).

In the first claim of 3.4, the last bit of the claim is $D^{\prime }\in C_0⊞C_1$ but the proof actually shows $D^{\prime }\in C_0^{\prime }⊞C_1^{\prime }$.

a few more typos. (do say if these nits aren't wanted.)

$\to (A_0\bigsqcup A_i)\times B$ is the natural bijection

(should be: $A_1$)

while the right hand side is $S$

(should be: in $S$)

To see that some restriction is required here (not imposed by HOL), consider that if $Chu\left(w\right)$ may contain arbitrary Cartesian frames over $w$ then we would have an injection $2^{Chu\left(w\right)}\to Chu\left(w\right)$ that, for example, encodes a set $S\subseteq Chu\left(w\right)$ as the Cartesian frame $C_S$ with $Agent\left(C_S\right)=S$ (the environment and evaluation function are unimportant), which runs afoul of Cantor's theorem regarding the cardinality of $Chu\left(w\right)$.

I wouldn't be surprised if a similar encoding/injection could be made using just the operations... (read more)

Since we have already established commutativity, it suffices to show that $(C_0\otimes C_1)\otimes C_2\cong (C_0\otimes C_2)\otimes C_1$.

For the confused reader, the argument in more detail here is:

$\begin{array}{ccc}& C_0\otimes (C_1\otimes C_2)& \\ & \cong (C_1\otimes C_2)\otimes C_0& \text{comm}\\ & \cong (C_1\otimes C_0)\otimes C_2& \text{lemma}\\ & \cong (C_0\otimes C_1)\otimes C_2& \text{comm, iso}\end{array}$

where $\text{comm}$ is commutativity of tensor, $\text{lemma}$ is the fact claimed to suffice above, and $\text{iso}$ is the implicitly assumed lemma that $C_0\cong C_1$ and $D_0\cong D_1$ implies $C_0\otimes D_0\cong C_1\otimes D_1$ (this is proved later but only for $\simeq$).

minor typo in the indices here:

We will show that $(C_0\otimes C_1)\otimes C_2\cong D$, and since the definition of $D$ is symmetric in swapping $C_1$ and $C_2$, it will follow that $(C_0\otimes C_1)\otimes C_2\cong D$

minor typo:

and take $P$ to be the set of all $p:E\to C$ such that

should have $p:E\to W$

Also I think later in that proof some of the $h$'s (like in $h_0\circ h_1$) should be $g$'s instead.

Do we lose much by restricting attention to finite Cartesian frames (i.e., with finite agent and environment)? I ask because I'm formalising these results in higher-order logic (HOL), and the category $Chu\left(w\right)$ is too big to represent if it really must contain frames with infinite agents and also for any pair of frames the frame whose agents are the morphisms between them. The root problem is probably that I require any category's class of objects to be a set, but it's hard to avoid this requirement in HOL in a nice way. Everything should work out f... (read more)

Counterexample to your claim that $Ctrl(C\oplus D)=Ctrl\left(C\right)\cup Ctrl\left(D\right)$ (with both $C$ and $D$ not $null$):

Take $C$ and $D$ to be the following Cartesian frames over world $\{x,y\}$ with agent and environment both $\{a,b\}$ as the following matrices:

Then $Ctrl\left(C\right)=Ctrl\left(D\right)=\varnothing$ but $Ctrl(C\oplus D)\ni \left\{x\right\}$.

(Sorry for my formatting - I haven't done LaTeX in these comments before.)

I would use 'outcome alignment' for 2 (and agree with 'intent alignment' for 1). In other words, I see the important distinction between 'outcome' and 'intent' being in the first part of the options, not the second.

I'd be inclined to see 3 and 4 as variations on 1 and 2 where what the human wants is for the AI to figure out some notion of their true goals and pursue/achieve that.

Have you seen Erik P Hoel's work on causal abstraction? This post reminded me of it.

When you talk about counterfactuals do you mean interventions? Although I'm guessing the "everything still works" conclusion holds for both interventions and counterfactuals.

Didn't watch the video but would have read the post. Might watch the video only because previous posts have been appetising enough.

the objective of agent-designers is to have the agent collect as many agents as possible

Typo: should say "dollars"?

if the daemon is obfuscated, there is no efficient procedure which takes the daemon circuit as input and produces a smaller circuit that still solves the problem.

So we can't find any efficient constructive argument. That rules out most of the obvious strategies.

I don't think the procedure needs to be efficient to solve the problem, since we only care about existence of a smaller circuit (not an efficient way to produce it).