One of the classic conceptual problems with a Solomonoff-style approach to probability, information, and stat mech is "Which Turing machine?". The choice of Turing machine is analogous to the choice of prior in Bayesian probability. While universality means that any two Turing machines give roughly the same answers in the limit of large data (unlike two priors in Bayesian probability, where there is no universality assumption/guarantee), they can be arbitrarily different before then.

My usual answer to this problem is "well, ultimately this is all supposed to tell us things about real computational systems, so pick something which isn't too unreasonable or complex for a real system". 

But lately I've been looking at Aram Ebtekar and Marcus Hutter's Foundations of Algorithmic Thermodynamics. Based on both the paper and some discussion with Aram (along with Steve Petersen), I think there's maybe a more satisfying answer to the choice-of-Turing-machine issue in there.

Two key pieces:

• The "Comparison against Gibbs-Shannon entropy" section of the paper argues that uncomputability is a necessary feature, in order to assign entropy to individual states and still get a Second Law. The argument says: if there exists a short program which can provably find and output a high-entropy string S, then we can physically instantiate a machine to run that short program. Then, when that physical machine spits out the high-entropy string S, S could be used to erase another copy of S. In other words, there is some high-entropy state (S) which this physical machine + program could steer into a low-entropy state.
• As Aram pointed out, most of the bounds have a constant for the complexity of the laws of physics. If we choose a machine for which the laws of physics have high complexity, then the bounds are quantitatively trash.

The first piece is a part of the theory which can only bind to reality insofar as our chosen Turing machine is tractable to physically implement. The second piece is a part of the theory which can only bind to reality insofar as our physics can be tractably implemented on our chosen Turing machine.

In other words: in order for this thermodynamic theory to work well, we need to choose a Turing machine which is "computationally equivalent to" physics, in the sense that our physics can run the machine without insane implementation size, and the machine can run our physics without insane implementation size.

I'm still wrapping my head around all the pieces here, so hopefully I (or, better yet, someone else) will write up a more clear explainer in the future. But this smells really promising to me. Not just for purposes of Solomonoff thermodynamics, but also as a more principled way to tackle bounded rationality of embedded systems.