Epistemic status: This is an informal explanation of incomplete models that I think would have been very useful to me a few years ago, particularly for thinking about why Solomonoff induction may not be optimal if the universe isn't computable. 

Imagine that a stage magician is performing a series of coin flips, and you're trying to predict whether they will come up heads or tails for whatever reason - for now, lets assume idle curiosity, so that we don't have to deal with any complications from betting mechanisms. 

Normally, coin flips come up heads or tails at 50:50 odds, but this magician is particularly skilled at slight of hand, and for all you know he might be switching between trick coins with arbitrary chances of landing on heads between flips (or simply choosing among his carefully practiced tossing methods, which basically amounts to the same thing). What you want to know is the chances of heads on the next flip - if possible, you would like to predict it deterministically, but realistically the best you can do without precognition is state a probability of heads.

Now, ten of your friends are sitting at the table with you. They have various degrees of expertise about stage magic among them. Before each flip, they're each happy to tell you their guess about the odds of heads.

How should you predict?

Well, that question is currently a bit underspecified. 

Let's assume that exactly one of your friends knows the magician's trick. Now this is basically a solved problem, and you probably know the solution (perhaps pause here and try to explain it precisely). 

Answer: You can use Bayes rule, with the prior weight on each friend decided by your confidence that they know the trick. Actually, setting the prior is (even in this very simple example) hard to do "precisely" - how did you arrive at the knowledge that exactly one friend knows the trick? But roughly speaking, the friends with more expertise should get a larger prior weight. If this seems too unscientific for your preferences, just use a uniform prior. Because as long as the weight on the correct friend isn't crazy tiny, this algorithm has very nice guarantees; you will quickly converge to predicting well.

(Now, to "break the fourth wall"' a bit, we might translate friends -> hypotheses, and we're secretly talking about experimenting to find the true hypothesis. I think this is how laymen often think of the scientific process. But my impression is that scientists - and particularly scientific philosophers - don't think of their work this way. They only believe in finding better and better models, without assuming that any of them will ever be TRUE. With that in mind...)

 Let's make this a little harder. What if the magician is using one trick for odd flips and one trick for even flips, and all you know is that some friend knows the odd trick and some friend knows the even trick - but it is not necessarily the case that any friend knows both?

Answer: I'm not sure this deserves the bold "Answer" anymore, but there is still a very clean solution. You can still use Bayes rule, but now on the larger space of hypotheses that you get from pairing your friends. There is some (ordered) pair with one friend predicting odd flips and one friend predicting even flips which is actually right. It's possible for the pair to be the same guy twice. Now, the coin flips are "really coming from" one of the tricks that you are considering (this assumption is called realizability) and so the Bayesian algorithm still has great convergence guarantees. Things aren't quite as rosy as before though. First off, you've blown up the number of "friends" quadratically. Second, now that they're just "friends" (pairs and not actual guys with more or less trustworthy faces) its harder to intuit what the prior should be, and perhaps even a little confusing what it means. For instance, some pairs might turn out to make identical predictions, even if the individual friends all make different predictions. So the prior is no longer the chance that the pair truly knows the trick - unless perhaps you are able to combine pairs which make identical predictions (but this requires knowing their future predictions) or otherwise are handed a free equality test on pairs.

Here's somewhere I got hung up: You may not want to blow the hypothesis space up quadratically. It seems that each friend can be treated as potentially predicting either odd or even flips. So, you could split him into two "friends," one that uses his prediction on odd flips and just guesses 50:50 on even flips, and one that does the opposite (focusing on even flips). This only doubles the number of "friends" (which should improve the convergence rate). The problem is, it doesn't really work. The 50:50 odds don't properly express total ignorance.^[1] For instance, if your friend Joe makes great predictions on odd bits, but the even bits always come up heads, your "extended friend" Joe-on-odd who always predicts 50:50 on even flips is doing a pretty bad job, particularly if the odd flips are actually pretty close to 50:50. Bayes rule may start ignoring the advice of Joe-on-odd, because if some other friend Carl predicts even flips well, Carl-on-even is very accurate at continues to win posterior weight over Joe-on-odd. Unfortunately, the limit is worse than the Joe-on-odd, Carl-on-even pair! This is true even if the posterior weight ends up divided more evenly between Joe-on-odd and Carl-on-even, since the 50:50 predictions "dilute" the correct predictions.

Things get even more complicated when you only assume that there is a friend who knows each of (say) the trick for prime numbered flips, flip numbers ending in five, and the rest of the flips. Or in the hardcore "prediction with expert advice" setting, we don't assume that any of your friends know what's going on, and the best you can hope for is to not look too much dumber than any of them.  

There are a couple of interesting lessons.

First, if the hypothesis space is very rich (e.g. Solomonoff induction) you might hope it can already simulate the combinations of lots of simpler hypotheses, so you don't have to think about complicated prediction rules with incomplete models. But, this may require the hypothesis space to blow up more than you expected.

Also, it is interesting that the final Bayesian solution doesn't feel Bayesian. The assumption that you are seeking the one true hypothesis (and therefore one of the intuitive justifications for Bayesianism) is gone. In other words, you might not have been able to invent the algorithm through Bayesian thinking - the Bayesian paradigm is not always the right creative instrument. BUT you still want a Pareto-optimal algorithm, and where the coherence arguments for Bayesianism apply, this means that your algorithm should have a Bayesian representation. Finding that representation may be a later step in the process - a correction towards normativity. But it need not be the initial inspiration for your algorithm. And this is not just wild speculation - it is actually how the study of the important Mixing Past Posteriors (MPP) algorithm for prediction with expert advice apparently progressed - see "Putting Bayes to Sleep."  

At the bottom of each page of my research journal, I like to list what 1: I've learned about intelligence that day, 2: how I can apply it to my own reasoning, and 3: how I've been doing at that (in the hopes that I can recursively self-improve).

Part 2 is about extracting pragmatic lessons, which I believe we can also do here: since you are a bounded reasoner who certainly can't afford to combinatorially blow up you the space of models you are tracking, use partial models where it's pragmatically reasonable (how to determine when this is? Well, real-world experts do usually restrict themselves to their domain expertise of expertise, so that suggests a natural default partition). BUT when you have more time to think about something, and when you are unusually clever about it, try to merge those partial models into a more cohesive Bayesian hypothesis space - this should make your learning more sample efficient, that is, allow you so figure out what's going on faster.

Also, don't force all of your thinking about a problem to always start from the Bayesian paradigm.

1. ^{^}

   You could instead treat Joe-on-odd as an imprecise probability distribution - in this case, the (closed convex) set of probability distributions that are consistent with his predictions on odd flips. More work is required to convert this idea into an expert aggregation algorithm though - I'm not sure how it connects with the other approaches in this post.