3. How does that handle ontology shifts? Suppose that this symbolic-to-us language would be suboptimal for compactly representing the universe. The compression process would want to use some other, more "natural" language. It would spend some bits of complexity defining it, then write the world-model in it. That language may turn out to be as alien to us as the encodings NNs use.

The cheapest way to define that natural language, however, would be via the definitions that are the simplest in terms of the symbolic-to-us language used by our complexity-estimator. This rules out definitions which would look to us like opaque black boxes, such as neural networks.

I note that this requires a fairly strong hypothesis: the symbolic-to-us language apparently has to be interpretable no matter what is being explained in that language. It is easy to imagine that there exist languages which are much more interpretable than neural nets (EG, English). However, it is much harder to imagine that there is a language in which all (compressible) things are interpretable.

Python might be more readable than C, but some Python programs are still going to be really hard to understand, and not only due to length. (Sometimes terser programs are the more difficult to understand.)

Perhaps the claim is that such Python programs won't be encountered due to relevant properties of the universe (ie, because the universe is understandable). 

I also don’t believe that insider trading is immoral. Insider trading increases the accuracy of the stock prices available to the public, which is the public good that equity trading provides. For this reason, prediction markets love insider trading. The reason it’s illegal is to protect retail investors, but why do they get privileged over everyone else? Another reason insider trading is immoral is that it robs the company of proprietary information (if you weren’t a limb of The Company, you wouldn’t know the merger is happening). That’s fair, but in that case doing it officially for The Company should be allowed, and it’s not. In this example ChatGPT arguably helped steal information from LING, but did so in service of the other company, so I guess it’s kind of an immoral case—but would be moral if LING is also insider-trading on it.

The problem with insider trading, in my view, is that someone with an important role in the company can short the stock and then do something really bad that tanks the value of the company. The equilibrium in a market that allows insider trading involves draconian measures within the companies themselves to prevent this sort of behavior (or else, no multi-person ventures that can be publicly traded).

This is an instance of the more general misalignment of prediction markets: whenever there's something on a prediction market that is quite improbable in ordinary circumstances but could be caused to happen by a single actor or a small number of people, there's profit to be made by sewing chaos.

1. Sam's theory gives some evidence for  broad commonality between the data-structures different minds would use to represent the same problem (because it proves this happens under some conditions).
2. Sam's theory gives a framework for proving useful things about these structures.
3. Sam's theory of interpretability comes from a kind of usefulness, so there's at least some hope here.

The optimal condensation is not (typically) 1 book per question. Instead, it typically recovers the meaningful latents which you'd want to write down to model the problem. Really, the right thing to do is to work examples to get an intuition for what happens. Sam does some of this in his paper.

Fixed!

Perhaps I'm still not understanding you, but here is my current interpretation of what you are saying:

• The (expected utility) argument that it is valuable for us to get the ASI to entangle its values with ours relies on the assumption of non-nosy-ness.
  • That is: since we are uncertain which values are ours, but whichever thing we value, we're just as happy to impose that thing on versions of ourselves which do not value that thing, we don't see any increase in expected value from Geometric UDT.

I see this line of reasoning as insisting on taking max-expected-utility according to your explicit model of your values (including your value uncertainty), even when you have an option which you can prove is higher expected utility according to your true values (whatever they are).

My argument has a somewhat frequentist flavor: I'm postulating true values (similar to postulating a true population frequency), and then looking for guarantees with respect to them (somewhat similar to looking for an unbiased estimator). Perhaps that is why you're finding it so counter-intuitive?

The crux of the issue seems to be whether we should always maximize our explicit estimate of expected utility, vs taking actions which we know are better with respect to our true values despite not knowing which values those are. One way to justify the latter would be via Knightian value uncertainty (ie infrabayesian value uncertainty), although that hasn't been the argument I've been trying to make. I'm wondering if a more thoroughly geometric-rationality perspective would provide another sort of justification.

But the argument I'm trying to make here is closer to just: but you know Geometric UDT is better according to your true values, whatever they are!

== earlier draft reply for more context on my thinking ==

Perhaps I'm just not understanding your argument here, and you need to spell it out in more detail? My current interpretation is that you are interpreting "care about both worlds equally" as "care about rainbows and puppies equally" rather than "if I care about rainbows, then I equally want more rainbows in the (real) rainbow-world and the (counterfactual) puppy-world; if I care about puppies, then I equally want more puppies in the (real) puppy-world and the (counterfactual) rainbow-world."

A value hypothesis is a nosy neighbor if^[1] it wants the same things for you whether it is your true values or not. So what's being asserted here (your "third if" as I'm understanding it) is that we are confident we've got that kind of relationship with ourselves -- we don't want "our values to be satisfied, whatever they are" -- rather, whatever our values are, we want them to be satisfied across universes, even in counterfactual universes where we have different values.

Maximizing rainbows maximizes the expected value given our value uncertainty, but it is a catastrophe in the case that we are indeed puppy-loving. Moreover, it is an avoidable catastrophe; ...

... and now I think I see your point?

The idea that it is valuable for us to get the ASI to entangle its values with ours relies on an assumption of non-nosyness. 

There is a different way to justify this assumption, 

1. ^{^}

   (but not "only if"; there are other ways to be a nosy neighbor)

Wait, do you think value uncertainty is equivalent/reducible to uncertainty about the correct prior? 

Yep. Value uncertainty is reduced to uncertainty about the correct prior via the device of putting the correct values into the world as propositions.

Would that mean the correct prior to use depends on your values?

If we construe "values" as preferences, this is already clear in standard decision theory; preferences depend on both probabilities and utilities. UDT further blurs the line, because in the context of UDT, probabilities feel more like a "caring measure" expressing how much the agent cares about how things go in particular branches of possibility.

So one conflicting pair spoils the whole thing, i.e. ignoring the pair is a pareto improvement? 

Unless I've made an error? If the Pareto improvement doesn't impact the pair, then gains-from-trade for both in the pair is zero, making the product of gains-from-trade zero. But the Pareto improvement can't impact the pair, since an improvement for one would be a detriment to the other.

When I try to understand the position you're speaking from, I suppose you're imagining a world where an agent's true preferences are always and only represented by their current introspectively accessible probability+utility,^[1] whereas I'm imagining a world where "value uncertainty" is really meaningful (there can be a difference between the probability+utility we can articulate and our true probability+utility).

If 50% rainbows and 50% puppies is indeed the best representation of our preferences, then I agree: maximize rainbows.

If 50% rainbows and 50% puppies is instead a representation of our credences about our unknown true values, my argument is as follows: the best thing for us would be to maximize our true values (whichever of the two this is). If we assume value learning works well, then Geometric UDT is a good approximation of that best option.

1. ^{^}

   Here "introspectively accessible" really means: what we can understand well enough to directly build into a machine.

This reminds me of Ramana’s question about what “enforces” normativity. The question immediately brought me back to a Peter Railton introductory lecture I saw (though I may be misremembering / misunderstanding / misquoting, it was a long time ago). He was saying that real normativity is not like the old Windows solitaire game, where if you try to move a card on top of another card illegally it will just prevent you, snapping the card back to where it was before. Systems like that plausibly have no normativity to them, when you have to follow the rules. In a way the whole point of normativity is that it is not enforced; if it were, it wouldn’t be normative.

I'm reminded of trembling-hand equilibria. Nash equilibria don't have to be self-enforcing; there can be tied-expectation actions which nonetheless simply aren't taken, so that agents could rationally move away from the equilibrium. Trembling-hand captures the idea that all actions have to have some probability (but some might be vanishingly small). Think of it as a very shallow model of where norm-violations come from: they're just random! 

Evolutionarily stable strategies are perhaps an even better model of this, with self-enforcement being baked into the notion of equilibrium: stable strategies are those which cannot be invaded by alternate strategies. 

Neither of these capture the case where the norms are frequently violated, however.

My notion of a function “for itself” is supposed to be that the functional mechanism somehow benefits the thing of which it’s a part. (Of course hammers can benefit carpenters, but we don’t tend to think of the hammer as a part of the carpenter, only a tool the carpenter uses. But I must confess that where that line is I don’t know, given complications like the “extended mind” hypothesis.)

Putting this in utility-theoretic terminology, you are saying that "for itself" telos places positive expectation on its own functional mechanism, or perhaps stronger, uses significant bits of its decision-making power on self-preservation. 

A representation theorem along these lines might reveal conditions under which such structures are usefully seen as possessing beliefs: a part of the self-preserving structure whose telos is map-territory correspondence.