A physicalist hypothesis is a pair $(\Phi ,\Theta$), where $\Phi$ is a finite^[4:2] set representing the physical states of the universe and $\Theta \in \square (\Gamma \times \Phi )$ represents a joint belief about computations and physics. [...] Our agent will have a prior over such hypotheses, ranging over different $\Phi$.

I am confused what the state space $\Phi$ is adding to your formalism and how it is supposed to solve the ontology identification problem. Based on what I understood, if I want to use this for inference, I have this prior $\xi \in {\square }^c(\Phi ,\Theta )$, and now I can use the bridge transform to project phi out again to evaluate my loss in different counterfactuals. But when looking at your loss function, it seems like most of the hard work is actually done in your relation $C\in e{l}^{\Gamma }$ that determines which universes are consistent, but its definition does not seem to depend on $\Phi$. How is that different from having a prior that is just over $\xi \in {\square }^c\left(\Gamma \right)$ and taking the loss, if $\Phi$ is projected out anyway and thus not involved?

Last time I checked, you could not teach a banana basic arithmetic. This works for most humans, so obviously evolution did lots of leg work there.

I don't see the problem. Your learning algorithm doesn't have to be "very" complicated. It has to work. Machine learning models don't consist of million lines of code. I do see the problem where one might expect evolution not to be very good at doing that compression, but I find the argument that there would actually be lots of bits needed very unconvincing.

Thanks! Exactly what I was looking for :)

One claim I found very surprising:

To make computationalism well-defined, we need to define what it means for a computation to be instantiated or not. Most of the philosophical arguments against computationalism attempt to render it trivial by showing that according to any reasonable definition, all computations are occurring everywhere at all times, or at least there are far more computations in any complex object than a computationalist wants to admit. I won't be reviewing those arguments here; I personally think they fail if we define computation carefully, but I'm not trying to be super-careful in the present essay.

This sounds very intriguing, as I have encountered this problem "what is computation" in some discussions, but have never seen anything satisfactory so far. I would be very glad for any links to solutions/definitions or resources that might help one to come up with a definition oneself ;).