Theorem 4: Pseudocausal IPOMDP's: If an infra-POMDP has a transition kernel $K:S\times A\stackrel{ik}{\to }O\times S$ which fulfills the niceness conditions, and there's a starting infradistribution $\psi \in \square S$, unrolling it into an infrakernel via $\Theta \left(\pi \right):=p{r}_{(A\times O{)}^{\omega }\ast }(\psi ⋉K_{:\infty }^{\pi })$ has the infrakernel $\Theta$ being pseudocausal and fulfilling all belief function conditions except for normalization.

This takes a whole lot of work to show. The proof breaks down into two phases. The first is showing that the $K_n^{\pi }$ fulfill the niceness conditions (which splits into four subphases of showing 1-Lipschitzness, lower-semicontinuity, compact-shared CAS, and moving constants up/mapping 1 to 1). This is the easy part to do. The second phase is much longer. It splits into five pieces, where we show our list of conditions for $\Theta$.  Our list of conditions to show is: 

1:The $\Theta \left(\pi \right)$ have a uniform upper bound on their Lipschitz constants.

2: $\pi \mapsto \Theta \left(\pi \right)\left(U\right)$ is lower-semicontinuous for all $U$

3: $\Theta \left(\pi \right)$ is supported entirely on histories compatible with $\pi$.

4: All the $\Theta \left(\pi \right)$ agree on the value of 1 (or infinity).

5: $\Theta$ is pseudocausal.

Parts 1 and 4 can be done with little trouble, but parts 2, 3, and 5 split into additional pieces. To show pseudocausality, ie, condition 5, we have to start with our proof goal and keep rewriting it into an easier form, until we get something we can knock out with an induction proof, which splits into a base case and an induction step. To show sensible supports, ie, part 3, we have to split it into three parts, there's one induction proof for the finite $K_{:n}^{\pi }$, showing sensible supports for $K_{:\infty }^{\pi }$, and then finally extending this to the result we actually want, which is fairly easy.

And then there's lower-semicontinuity, part 2, which is an absolute nightmare. This splits into three parts. One is explicitly constructing a sequence of compact sets to use in the Almost-Monotone Lemma for any policy whatsoever, and then there's trying to show that $\pi \mapsto (\psi ⋉K_{:\infty }^{\pi })\left(f\right)$ is lower-semimicontinuous which requires a massive amount of messing around with integrals and limits, and then it's pretty easy to show lower-semicontinuity for $\Theta$ from there.

T4.1 Our first order of business is showing the five niceness conditions for all the $K_n^{\pi }$ infrakernels, so we can take the infinite semidirect product and show that $K_{:\infty }^{\pi }$ inherits the five niceness conditions (from the proof of Theorem 1).

Remember that $K_n^{\pi }$ is defined as

$K_n^{\pi }(s_0,ao{s}_{1:n}):={\delta }_{\pi \left(a{o}_{1:n}\right)}⋉(\lambda a.K(s_n,a\left)\right)$

Obviously it produces inframeasures, since $K$ does. 

T4.1.1 For lower-semicontinuity in the input, we have

${liminf}_{m\to \infty }{K}_n^{\pi }(s_0^m,ao{s}_{1:n}^m)\left(f\right)$

$={liminf}_{m\to \infty }({\delta }_{\pi \left(a{o}_{1:n}^m\right)}⋉(\lambda a.K(s_n^m,a)\left)\right)\left(f\right)$

$={liminf}_{m\to \infty }K(s_n^m,\pi (a{o}_{1:n}^m\left)\right)(\lambda o,s.f(\pi \left(a{o}_{1:n}^m\right),o,s\left)\right)$

and then, as $ao{s}_{1:n}^m$ limits to $ao{s}_{1:n}^{\infty }$, we have $a{o}_{1:n}^m=a{o}_{1:n}^{\infty }$ forever after some finite $m$, because ${\prod }_{i=1}^{i=n}(A\times O)$ is a finite space. So, it turns into

$={liminf}_{m\to \infty }K(s_n^m,\pi (a{o}_{1:n}^{\infty }\left)\right)(\lambda o,s.f(\pi \left(a{o}_{1:n}^{\infty }\right),o,s\left)\right)$

and then, by lower-semicontinuity for K (one of the niceness conditions)

$\ge K(s_n^{\infty },\pi (a{o}_{1:n}^{\infty }\left)\right)(\lambda o,s.f(\pi \left(a{o}_{1:n}^{\infty }\right),o,s\left)\right)$

and then we pack this back up as

$={\delta }_{\pi \left(a{o}_{1:n}^m\right)}⋉(\lambda a.K(s_n^{\infty },a\left)\right)\left(f\right)=K_n^{\pi }(s_0^{\infty },ao{s}_{1:n}^{\infty })\left(f\right)$

and lower-semicontinuity has been shown.

T4.1.2 For 1-Lipschitzness, observe that ${\delta }_{\pi \left(a{o}_{1:n}\right)}⋉(\lambda a.K(s_n,a\left)\right)$ is 1-Lipschitz because $K$ is 1-Lipschitz, and this is the semidirect product of a 1-Lipschitz (all probability distributions are 1-Lipschitz, and this is a dirac-delta distribution) infradistribution with a 1-Lipschitz infrakernel.

T4.1.3 For compact-shared compact-almost-support, observe that $(s_0,ao{s}_{1:n})\mapsto (s_n,\pi (a{o}_{1:n}\left)\right)$ is a continuous function, so any compact subset of $S\times (A\times O\times S{)}^n$ (input to $K_n^{\pi }$) maps to a compact subset of $S\times A$ (corresponding inputs for $K$). Then, we can apply compact-shared compact-almost-support for $K$ (one of the niceness conditions) to get a compact almost-support on $O\times S$, and take the product of that subset with all of $A$ itself, to get a shared-compact almost-support for $K_n^{\pi }$.

T4.1.4 For mapping constants to above constants, we have

$K_n^{\pi }(s_0,ao{s}_{1:n})\left(c\right)=({\delta }_{\pi \left(a{o}_{1:n}\right)}⋉(\lambda a.K(s_n,a)\left)\right)\left(c\right)$

$={\delta }_{\pi \left(a{o}_{1:n}\right)}(\lambda a.K(s_n,a\left)\right(c\left)\right)=K(s_n,\pi (a{o}_{1:n}\left)\right)\left(c\right)\ge c$

By increasing constants for $K$ (one of the niceness conditions). This same proof works with equality when $c=1$, because of the "map 1 to 1" condition on the $[0,1]$ type signature. So all the $K_n^{\pi }$ have the niceness properties, and we can form the infinite semidirect product at all, and the infinite semidirect product inherits these niceness conditions.

T4.2 Now we can get working on showing the belief function conditions.

T4.2.1 For showing a bounded Lipschitz constant, we have

$|\Theta \left(\pi \right)\left(U\right)-\Theta \left(\pi \right)\left(U^{\prime }\right)|=|p{r}_{(A\times O{)}^{\omega }\ast }(\psi ⋉K_{:\infty }^{\pi })\left(U\right)-p{r}_{(A\times O{)}^{\omega }\ast }(\psi ⋉K_{:\infty }^{\pi })\left(U^{\prime }\right)|$

$=|\psi (\lambda s_0.K_{:\infty }^{\pi }(s_0\left)\right(\lambda ao{s}_{1:\infty }.U\left(a{o}_{1:\infty }\right)\left)\right)-\psi (\lambda s_0.K_{:\infty }^{\pi }(s_0\left)\right(\lambda ao{s}_{1:\infty }.U^{\prime }\left(a{o}_{1:\infty }\right)\left)\right)|$

and then since $\psi$ is an infradistribution, it must have a finite Lipschitz constant, ${\lambda }^{\odot }$. So we then get

$\le {\lambda }^{\odot }\cdot {sup}_{s_0}|K_{:\infty }^{\pi }\left(s_0\right)(\lambda ao{s}_{1:\infty }.U(a{o}_{1:\infty }\left)\right)-K_{:\infty }^{\pi }\left(s_0\right)(\lambda ao{s}_{1:\infty }.U^{\prime }(a{o}_{1:\infty }\left)\right)|$

and then, since all the $K_{:\infty }^{\pi }\left(s_0\right)$ are 1-Lipschitz (infinite semidirect product has niceness conditions), we get

$\le {\lambda }^{\odot }{sup}_{ao{s}_{1:\infty }}|U\left(a{o}_{1:\infty }\right)-U^{\prime }\left(a{o}_{1:\infty }\right)|={\lambda }^{\odot }\cdot d(U,U^{\prime })$

and we're done, we showed a Lipschitz bound on the $\Theta \left(\pi \right)$ that is uniform in $\pi$.

T4.2.2 Now for lower-semicontinuity for $\Theta$, which is the extremely hard one.

The way this proof will work is that we'll first show a variant of the Almost-Monotone Lemma that works for all the $K_{:\infty }^{\pi }$ infrakernels simultaneously. Then, using that, we'll show that

$\pi \mapsto (\psi ⋉K_{:\infty }^{\pi })\left(f\right)$ is lower-semicontinuous. And then, it's pretty easy to wrap up and get that

$\pi \mapsto \Theta \left(\pi \right)\left(U\right)$ is lower-semicontinuous, from that.

T4.2.2.1 Anyways, our first goal is a variant of the Almost-Monotone Lemma. The particular variant we're searching for, specialized to our particular case, is:

$\forall C^S\in K\left(S\right),ϵ>0:\exists \overline{C}\in {\prod }_{i=1}^{\infty }K(A\times O\times S):\forall \pi \in \Pi ,n,m\in N,s_0\in C^S,f\in C_B\left(\right(A\times O\times S{)}^{\omega }):$
$K_{:n+m}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+m+1}.{inf}_{ao{s}_{n+m+2:\infty }\in {\prod }_{i=n+m+2}^{\infty }{C}_i}f(ao{s}_{1:n+m+1},ao{s}_{n+m+2:\infty }\left)\right)$
$\ge K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.{inf}_{ao{s}_{n+2:\infty }\in {\prod }_{i=n+2}^{\infty }{C}_i}f(ao{s}_{1:n+1},ao{s}_{n+2:\infty }\left)\right)-4ϵ||f||$

We let our $C^S$ be an arbitrary compact subset of states, and $ϵ$ be arbitrary.

Now, let's assume there's a sequence of compact sets C${}_i^{\top }[C^S,ϵ]$ where, regardless of n or $\pi$, $C_{1:n+1}^{\top }[C^S,ϵ]$ is a shared $ϵ(1-\frac{1}{2^{n+1}})$-almost-support for all the $K_{:n}^{\pi }\left(s_0\right)$ inframeasures (where $s_0\in C^S$).

Then our proof goal becomes

$\forall \pi \in \Pi ,n,m\in N,s_0\in C^S,f\in C_B\left(\right(A\times O\times S{)}^{\omega }):$
$K_{:n+m}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+m+1}.{inf}_{ao{s}_{n+m+2:\infty }\in C_{n+m+2:\infty }^{\top }[C^S,ϵ]}f(ao{s}_{1:n+m+1},ao{s}_{n+m+2:\infty }\left)\right)$
$\ge K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.{inf}_{ao{s}_{n+2:\infty }\in C_{n+2:\infty }^{\top }[C^S,ϵ]}f(ao{s}_{1:n+1},ao{s}_{n+2:\infty }\left)\right)-4ϵ||f||$

The proof of the Almost-Monotone Lemma goes through perfectly fine, the key part where our choice of compact sets matters is in the ability to make the assumption that $C_{1:n+m+1}^{\top }[C^S,ϵ]$ is an $ϵ(1-\frac{1}{2^{n+m+1}})$-almost-support for all the $K_{:n+m}^{\pi }\left(s_0\right)$ with $s_0\in C^S$, as our $s_0$ is. And we assumed our sequence of compact sets fulfilled that property.

So, where we're at is that our variant of the Almost-Monotone-Lemma works, as long as our assumption works out, that there's a sequence of compact sets $C_i^{\top }[C^S,ϵ]$ where, regardless of n or $\pi$ or $s_0\in C^S$, $C_{1:n+1}^{\top }[C^S,ϵ]$ is a $ϵ(1-\frac{1}{2^{n+1}})$-almost-support for $K_{:n}^{\pi }\left(s_0\right)$.

In order to get our variant of the AML to go through, we really need to prove that there is such a sequence, for any $ϵ$ and $C^S$. For this, we'll be looking at a function

$\Xi :(0,\infty )\times A\times K\left(S\right)\to K(O,S)$

What this does is that $\Xi (ϵ,a,C^S)$ is defined to return a compact subset of $O\times S$ which is an $ϵ$-almost-support for all the $K(a,s)$ inframeasures, when $s\in C^S$. There are always compact subsets like this for any input because of shared almost-compact-support for the infrakernel $K$ (one of the assumed niceness conditions), so there does indeed exist a function $\Xi$ with these properties.

Then, we recursively define the $C_i^{\top }[C^S,ϵ]$ as:

$C_1^{\top }[C^S,ϵ]:=A\times {\bigcup }_a\Xi (\frac{ϵ}{2},a,C^S)$

and

$C_{n+1}^{\top }[C^S,ϵ]:=A\times {\bigcup }_a\Xi (\frac{ϵ}{2^{n+1}},a,p{r}_{S_n}(C_{1:n}^{\top }[C^S,ϵ]\left)\right)$

All these sets are compact, by induction. For the first one, since $\Xi$ always returns compact sets, we're taking a finite union of compact sets (is compact), and taking the product with a finite set (is compact). For induction up, since the product of compact sets is compact, $C_{1:n}^{\top }[C^S,ϵ]$ is compact, and then projections of compact sets are compact, so then $\Xi$ returns a compact set, and again, we take a finite union of compact sets and then the product with a compact set to make another compact set.

Now, let's get started on showing that C^{\top}_{1:n+1}[C^{S},\eps] is an \eps\left(1-\frac{1}{2^{n+1}}\right)-almost-support for K^{\pi}_{:n}(s_0), for arbitrary \pi and s_0\in C^{S}. This proof will proceed by induction.

For the base-case, we want that $C_1^{\top }[C^S,ϵ]$ is a $\frac{ϵ}{2}$-almost-support for $K_{:n}^{\pi }\left(s_0\right)$ with $s_0\in C^S$. Well, assume $f,f^{\prime }$ are equal on $C_1^{\top }[C^S,ϵ]$, and $s_0\in C^S$.

$|K_{:0}^{\pi }\left(s_0\right)\left(f\right)-K_{:0}^{\pi }\left(s_0\right)\left(f^{\prime }\right)|=|K_0^{\pi }\left(s_0\right)\left(f\right)-K_0^{\pi }\left(s_0\right)\left(f^{\prime }\right)|$

$=|({\delta }_{\pi \left(\right)}⋉(\lambda a.K(s_0,a)\left)\right)(\lambda a,o,s.f(a,o,s\left)\right)-({\delta }_{\pi \left(\right)}⋉(\lambda a.K(s_0,a)\left)\right)(\lambda a,o,s.f^{\prime }(a,o,s\left)\right)|$

$=|K(s_0,\pi (\left)\right)(\lambda o,s.f(\pi \left(\right),o,s\left)\right)-K(s_0,\pi (\left)\right)(\lambda o,s.f^{\prime }(\pi \left(\right),o,s\left)\right)|$

Now, $\Xi (\frac{ϵ}{2},\pi (),C^S)$ is a $\frac{ϵ}{2}$-almost-support for $K(s_0,\pi (\left)\right)$ because $s_0\in C^S$, so we can apply Lemma 2 from LBIT, and 1-Lipschitzness of $K$ to split this up as

$\le {sup}_{o,s\in \Xi (\frac{ϵ}{2},\pi (),C^S)}|f\left(\pi \right(),o,s)-f^{\prime }\left(\pi \right(),o,s)|+\frac{ϵ}{2}{sup}_{o,s}|f\left(\pi \right(),o,s)-f^{\prime }\left(\pi \right(),o,s)|$

That second term can be upper bounded by $d(f,f^{\prime })$, so we get an upper bound of

$\le {sup}_{o,s\in \Xi (\frac{ϵ}{2},\pi (),C^S)}|f\left(\pi \right(),o,s)-f^{\prime }\left(\pi \right(),o,s)|+\frac{ϵ}{2}d(f,f^{\prime })$

and then we observe that since $f=f^{\prime }$ on $C_1^{\top }[C^S,ϵ]$, and that set is defined as $A\times {\bigcup }_a\Xi (\frac{ϵ}{2},a,C^S)$ , we have that they agree on $\left\{\pi \right(\left)\right\}\times \Xi (\frac{ϵ}{2},\pi (),C^S)$

So that term turns into 0, the two functions are equal on our set of interest, so our net upper bound is

$\le \frac{ϵ}{2}d(f,f^{\prime })$

Showing that $C_1^{\top }[C^S,ϵ]$ is a $\frac{ϵ}{2}$-almost support for all the $K_{:0}^{\pi }\left(s_0\right)$, where $\pi$ is arbitrary and s_0\in C^{S}.

Now for induction. We want to show that $C_{1:n+2}^{\top }[C^S,ϵ]$ is a $ϵ(1-\frac{1}{2^{n+2}})$-almost-support for $K_{:n+1}^{\pi }\left(s_0\right)$ with $s_0\in C^S$, assuming that $C_{1:n+1}^{\top }[C^S,ϵ]$ is a $ϵ(1-\frac{1}{2^{n+1}})$-almost-support for $K_{:n}^{\pi }\left(s_0\right)$ with $s_0\in C^S$.

To begin, assume $f,f^{\prime }$ are equal on $C_{1:n+2}^{\top }[C^S,ϵ]$ and $s_0\in C^S$. Then we can go:

$|K_{:n+1}^{\pi }\left(s_0\right)\left(f\right)-K_{:n+1}^{\pi }\left(s_0\right)\left(f^{\prime }\right)|$

$=|K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.K_{n+1}^{\pi }(s_0,ao{s}_{1:n+1}\left)\right(\lambda a,o,s.f(ao{s}_{1:n+1},a,o,s)\left)\right)$
$-K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.K_{n+1}^{\pi }(s_0,ao{s}_{1:n+1}\left)\right(\lambda a,o,s.f^{\prime }(ao{s}_{1:n+1},a,o,s)\left)\right)|$

and then unpack to get

$=|K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.({\delta }_{\pi \left(a{o}_{1:n+1}\right)}⋉(\lambda a.K(s_{n+1},a\left)\right)\left)\right(\lambda a,o,s.f(ao{s}_{1:n+1},a,o,s)\left)\right)$
$-K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.({\delta }_{\pi \left(a{o}_{1:n+1}\right)}⋉(\lambda a.K(s_{n+1},a\left)\right)\left)\right(\lambda a,o,s.f^{\prime }(ao{s}_{1:n+1},a,o,s)\left)\right)|$

and unpack the semidirect product and substitute the dirac-delta value in to get

$=|K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.K(s_{n+1},\pi \left(a{o}_{1:n+1}\right)\left)\right(\lambda o,s.f(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)\left)\right)$
$-K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.K(s_{n+1},\pi \left(a{o}_{1:n+1}\right)\left)\right(\lambda o,s.f^{\prime }(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)\left)\right)|$

We apply a Lemma 2 from LBIT decomposition, with $C_{1:n+1}^{\top }[C^S,ϵ]$ as an $ϵ(1-\frac{1}{2^{n+1}})$-almost support for $K_{:n}^{\pi }\left(s_0\right)$, which works by induction assumption. Since $K_{:n}^{\pi }\left(s_0\right)$ is also 1-Lipschitz (the niceness conditions we already know), the Lemma 2 upper bound breaks down as

$\le {sup}_{ao{s}_{1:n+1}\in C_{1:n+1}^{\top }[C^S,ϵ]}|K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)(\lambda o,s.f(ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s\left)\right)$
$-K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)(\lambda o,s.f^{\prime }(ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s\left)\right)|$
$+ϵ(1-\frac{1}{2^{n+1}}){sup}_{ao{s}_{1:n+1}}|K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)(\lambda o,s.f(ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s\left)\right)$
$-K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)(\lambda o,s.f^{\prime }(ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s\left)\right)|$

By 1-Lipschitzness of $K$, we get an upper-bound of

$\le {sup}_{ao{s}_{1:n+1}\in C_{1:n+1}^{\top }[C^S,ϵ]}|K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)(\lambda o,s.f(ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s\left)\right)$
$-K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)(\lambda o,s.f^{\prime }(ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s\left)\right)|$
$+ϵ(1-\frac{1}{2^{n+1}}){sup}_{ao{s}_{1:n+1}}{sup}_{o,s}|f(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)-f^{\prime }(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)|$

And then this is upper-boundable by $d(f,f^{\prime })$, so we get

$\le {sup}_{ao{s}_{1:n+1}\in C_{1:n+1}^{\top }[C^S,ϵ]}|K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)(\lambda o,s.f(ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s\left)\right)$
$-K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)(\lambda o,s.f^{\prime }(ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s\left)\right)|+ϵ(1-\frac{1}{2^{n+1}})d(f,f^{\prime })$

Now, for that top term, we know that $s_{n+1}$ comes from $p{r}_{S_{n+1}}\left(C_{1:n+1}^{\top }\right[C^S,ϵ\left]\right)$. We apply another Lemma 2 decomposition on $K$ this time, with the compact set of interest being 

$\Xi (\frac{ϵ}{2^{n+2}},\pi (a{o}_{1:n+1}),p{r}_{S_{n+1}}(C_{1:n+1}^{\top }[C^S,ϵ]\left)\right)$

which is a $ϵ\left(\frac{1}{2^{n+2}}\right)$-almost-support for $K(s_{n+1},\pi (a{o}_{1:n+1}\left)\right)$. Pairing this with 1-Lipschitzness of $K$, we have an upper bound of

$\le {sup}_{ao{s}_{1:n+1}\in C_{1:n+1}^{\top }[C^S,ϵ]}({sup}_{o,s\in \Xi (\frac{ϵ}{2^{n+2}},\pi (a{o}_{1:n+1}),p{r}_{S_{n+1}}(C_{1:n+1}^{\top }[C^S,ϵ]\left)\right)}$
$|f(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)-f^{\prime }(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)|$
$+\frac{ϵ}{2^{n+2}}{sup}_{o,s}|f(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)-f^{\prime }(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)|)$
$+ϵ(1-\frac{1}{2^{n+1}})d(f,f^{\prime })$

That second term is upper-bounded by $d(f,f^{\prime })$ no matter what, so we have

$\le {sup}_{ao{s}_{1:n+1}\in C_{1:n+1}^{\top }[C^S,ϵ]}({sup}_{o,s\in \Xi (\frac{ϵ}{2^{n+2}},\pi (a{o}_{1:n+1}),p{r}_{S_{n+1}}(C_{1:n+1}^{\top }[C^S,ϵ]\left)\right)}$
$|f(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)-f^{\prime }(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)|$
$+\frac{ϵ}{2^{n+2}}d(f,f^{\prime }))+ϵ(1-\frac{1}{2^{n+1}})d(f,f^{\prime })$

Pull out the constant and combine, to get

$={sup}_{ao{s}_{1:n+1}\in C_{1:n+1}^{\top }[C^S,ϵ]}({sup}_{o,s\in \Xi (\frac{ϵ}{2^{n+2}},\pi (a{o}_{1:n+1}),p{r}_{S_{n+1}}(C_{1:n+1}^{\top }[C^S,ϵ]\left)\right)}$
$|f(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)-f^{\prime }(ao{s}_{1:n+1},\pi (a{o}_{1:n+1}),o,s)|)$
$+ϵ(1-\frac{1}{2^{n+2}})d(f,f^{\prime })$

Now, $f,f^{\prime }$ were assumed to agree on $C_{1:n+2}^{\top }[C^S,ϵ]$, which factors as

$C_{1:n+1}^{\top }[C^S,ϵ]\times C_{n+2}^{\top }[C^S,ϵ]$

and then expands as

$C_{1:n+1}^{\top }[C^S,ϵ]\times (A\times {\bigcup }_a\Xi (\frac{ϵ}{2^{n+2}},a,p{r}_{S_{n+1}}(C_{1:n+1}^{\top }[C^S,ϵ]\left)\right))$

which all our $ao{s}_{1:n+1},\pi \left(a{o}_{1:n+1}\right),o,s$ we're feeding in lie within. So those functions are identical for those inputs, and our upper-bound just reduces to

$=ϵ(1-\frac{1}{2^{n+2}})d(f,f^{\prime })$

Which witnesses that $C_{1:n+2}^{\top }[C^S,ϵ]$ is a $ϵ(1-\frac{1}{2^{n+2}})$-almost-support for any $K_{:n+1}^{\pi }\left(s_0\right)$ (arbitrary $\pi$) as long as $s_0\in C^S$. So our induction step goes off without a hitch, and we now know that the $C_i^{\top }[C^S,ϵ]$sequence is suitable to get our modification of the Almost-Monotone-Lemma to work out. We have the result 

$\forall \pi \in \Pi ,n,m\in N,s_0\in C^S,f\in C_B\left(\right(A\times O\times S{)}^{\omega }):$
$K_{:n+m}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+m+1}.{inf}_{ao{s}_{n+m+2:\infty }\in C_{n+m+2:\infty }^{\top }[C^S,ϵ]}f(ao{s}_{1:n+m+1},ao{s}_{n+m+2:\infty }\left)\right)$
$\ge K_{:n}^{\pi }\left(s_0\right)(\lambda ao{s}_{1:n+1}.{inf}_{ao{s}_{n+2:\infty }\in C_{n+2:\infty }^{\top }[C^S,ϵ]}f(ao{s}_{1:n+1},ao{s}_{n+2:\infty }\left)\right)-4ϵ||f||$