Vladimir_Nesov's Shortform

Vladimir_Nesov

This is a special post for quick takes by Vladimir_Nesov. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

The physical world is a simple thing that's not useful to study too much, once intelligence scales well past the human level. Almost all cognition is going to be about the computations designed by cognition (or their semantics), not about the physical world, which is likely to be almost entirely relegated to the role of compute substrate for the intentionally designed computations. All of the physical world might one day be about as relevant to most people as the arcane details of material manufacturing processes necessary to make a specialized industrial tool they were never aware of.

So I don't think decision theory for agent foundations should be concerned with the kinds of uncertainty that are central to thinking about the physical world. Instead, it should study reasoning about known computations, and influence of decisions on the behavior of known computations. These are not toy examples that in some ways pose unusual difficulties and in other ways are easier to handle, but rather most of what matters.

3jessicata7d

Ok, to make my opinion on this clearer: (a) We don't really know much about what the eventual state looks like, whether it is a singleton, how locked down, how many have access to quantum ops. (b) My personal guess for (a) is that "stochastic Turing machines + Bell nonlocality" is a more relevant model than "deterministic Turing machines", though both are relevant. (c) How do we figure out about (a)? It seems like we need some theory of agent foundations to think about this well. That theory shouldn't start out with strong assumptions like "only deterministic Turing machines are relevant" or "everyone has access to a quantum RNG". But we should also use our personal guesses on this. So I'm thinking about agent foundations with stochastic TMs partially because of my (b) opinion. (d) Part of reasoning about (a) also involves reasoning about intermediate states. These intermediate states can be "messy" and therefore forecasting them involves non-agent-foundations reasoning. But agent foundations is still relevant here, since some intermediate states are smarter-than-human, "economically efficient", etc. (e) I think stochastic TMs are more important for (d) than deterministic TMs, as a model. Sure, both will have some relevance, but quantum RNGs are currently easily available, will become more available in the near future, and have cryptographic applications. Moreover, many RL algorithms, and decision theory (like CDT self-ratification, mixed Nash equilibrium, absent minded driver) assumes access to RNGs, so quantum RNGs are relevant as an idealized case. (f) Overall these considerations make me think stochastic TMs are a better model for agent foundations than deterministic TMs, but in theory both are relevant. My opinion here seems to significantly disagree with yours. (g) I think I have significantly more "dispersion" in my beliefs about the eventual state (a) than you do, such that I don't find the specific scenario you're talking about where agents are locke

4Vladimir_Nesov7d

My stance is "unclear", this subthread is about why it's unclear, why a weird possibility of determinism and a lack of access to quantum observations/randomness/computation could make sense. It's not load bearing for the post or how I think about this; even with quantum observations you still end up with intentionally designed computations in a central sense, rather than mere weather and stars. The backups motivation doesn't imply external observations shouldn't happen at all, only that they should be sufficiently old to already be in the distributed backups by the time they get to be observed on the inside. It's a values question whether creating a vulnerable branch determined by non-backed-up observations is a good thing.

Ah, ok. I agree on "unclear" so we disagree less than I thought. I could see an analogous case of: "Maybe in the future, there will be many low-intelligence agents, because there will be some governance system that forces all agents except one to be low-intelligence, for security reasons. We should be extensively studying low-intelligence agents in agent foundations for this reason." And some but not all reasons why this is not a compelling argument for studying low-intelligence agents (in agent foundations) translate to studying "agents with no quantum tech".