...the problem of how to choose one's IBH prior. (If the solution was something like "it's subjective/arbitrary" that would be pretty unsatisfying from my perspective.)

It seems clear to me that the prior is subjective. Like with Solomonoff induction, I expect there to exist something like the right asymptotic for the prior (i.e. an equivalence class of priors under the equivalence relation where $\mu$ and $\nu$ are equivalent when there exists some $C>0$ s.t. $\mu \le C\nu$ and $\nu \le C\mu$), but not a unique correct prior, just... (read more)

...I'm still comfortable sticking with "most are wide open".

Allow me to rephrase. The problems are open, that's fair enough. But, the gist of your post seems to be: "Since coming up with UDT, we ran into these problems, made no progress, and are apparently at a dead end. Therefore, UDT might have been the wrong turn entirely." On the other hand, my view is: Since coming up with those problems, we made a lot of progress on agent theory within the LTA, which has implications on those problems among other things, and so far this progress seems to only r... (read more)

I'll start with Problem 4 because that's the one where I feel closest to the solution. In your 3-player Prisoner's Dilemma, infra-Bayesian hagglers^[1] (IBH agents) don't necessarily play CCC. Depending on their priors, they might converge to CCC or CCD or other Pareto-efficient outcome^[2]. Naturally, if the first two agents have identical priors then e.g. DCC is impossible, but CCD still is. Whereas, if all 3 have the same prior they will necessarily converge to CCC. Moreover, there is no "best choice of prior": different choices do better in differen... (read more)

The way I see it, all of these problems are reducible to (i) understanding what's up with the monotonicity principle in infra-Bayesian physicalism and (ii) completing a new and yet unpublished research direction (working title: "infra-Bayesian haggling") which shows that IB agents converge to Pareto efficient outcomes^[1]. So, I wouldn't call them "wide open".

1. ^{^}

   Sometimes, but there are assumptions, see child comment for more details.

First, I think that the theory of agents is a more useful starting point than metaphilosophy. Once we have a theory of agents, we can build models, within that theory, of agents reasoning about philosophical questions. Such models would be answers to special cases of metaphilosophy. I'm not sure we're going to have a coherent theory of "metaphilosophy" in general, distinct from the theory of agents, because I'm not sure that "philosophy" is an especially natural category^[1].

Some examples of what that might look like:

• An agent inventing a theory of agents in

I have a question about the conjecture at the end of Direction 17.5. Let $U_1$ be a utility function with values in $[0,1]$ and let $f:[0,1]\to [0,1]$ be a strictly monotonous function. Then $U_1$ and $U_2=f\circ U_1$ have the same maxima. $f$ can be non-linear, e.g. $f\left(x\right)=x^2$. Therefore, I wonder if the condition $u\left(y\right)=\alpha v\left(y\right)+\beta$ should be weaker.

No, because it changes the expected value of the utility function under various distributions.

Moreover, I ask myself if it is possible to modify $U_1$ by a smal

Oops. What if instead of "for any $p$" we go with "there exists $p$"?

I propose the axioms A1-A3 together with

B2. If $A\prec B$ then for any $p\in (0,1)$ we have $A\prec pA+(1-p)B\prec B$
B3. If $A⪯B$ and $A⪯C$, then for any $p\in [0,1]$ we have $A⪯pB+(1-p)C$

I suspect that these imply C4.

I think that step 6 is supposed to say "from 5 and 3" instead of "from 4 and 1"?

Good idea!

Example 1

Fix some alphabet $\Sigma$. Here's how you make an automaton that checks that the input sequence (an element of ${\Sigma }^{\ast }$) is a subsequence of some infinite periodic sequence with period $n$. For every $k$ in $Z/n$, let $A_k$ be an automaton that checks whether the symbols in the input sequences at places $i$ s.t. $i\equiv k\left(\mathrm{mod}n\right)$ are all equal (its number of states is $O\left(n|\Sigma |\right)$). We can modify it to make a transducer $A_k^{\prime }$ that produces its unmodified input sequence if the test passes and $\perp$ if the test fails. It also produces $\perp$ when the input is $\perp$. We then chain $A_0^{\prime },A_1^{\prime }$... (read more)

We received no submissions so far, but I think that such submissions will appear here in the "prize claims" section.

$P\ne NP$ deserves a little more credit than you give it. To interpret the claim correctly, we need to notice $P$ and $NP$ are classes of decision problems, not classes of proof systems for decision problems. You demonstrate that for a fixed proof system it is possible that generating proofs is easier than verifying proofs. However, if we fix a decision problem and allow any valid (i.e. sound and complete) proof system, then verifying cannot be harder than generating. Indeed, let $S_1$ be some proof system and $A$ an algorithm for generating proofs (i.e. an algorithm t... (read more)

I don't think the argument on hacking relied on the ability to formally verify systems. Formally verified systems could potentially skew the balance of power to the defender side, but even if they don't exist, I don't think balance is completely skewed to the attacker.

My point was not about the defender/attacker balance. My point was that even short-term goals can be difficult to specify, which undermines the notion that we can easily empower ourselves by short-term AI.

Of course we need to understand how to define "long term" and "short term" here. O

Thanks for the responses Boaz!

Our claim is that one can separate out components - there is the predictable component which is non stationary, but is best approximated with a relatively simple baseline, and the chaotic component, which over the long run is just noise.In general, highly complex rules are more sensitive to noise (in fact, there are theorems along these lines in the field of Analysis of Boolean Functions), and so in the long run, the simpler component will dominate the accuracy.

I will look into analysis of boolean functions, thank you. How... (read more)

IIUC the thesis of this article rest on several interrelated claims:

1. Long-term planning is not useful because of chaos
2. Short-term AIs have no alignment problem
3. Among humans, skill is not important for leadership, beyond some point
4. Human brains have an advantage w.r.t. animals because of "universality", and any further advantage can only come from scaling with resources.

I wish to address these claims one by one.

Claim 1

This is an erroneous application of chaos theory IMO. The core observation of chaos theory is, that in many dynamical systems with compa... (read more)

Even if we did make a goal program, it's still unknown how to build an AGI that is motivated to compute it, or to follow the goals it outputs.

Actually, it is (to a 0th approximation) known how to build an AGI that is motivated to compute it: use infra-Bayesian physicalism. The loss function in IBP already has the semantics "which programs should run". Following the goal it outputs is also formalizable within IBP, but even without this step we can just have utopia inside the goal program itself^[1].

P.S.

I think that in your example, if a person is given a button that can save a person on a different planet from being tortured, they will have a direct incentive to press the button, because the button is a causal connection in itself, and consciously reasoning about the person on the other planet is a causal^[1] connection in the other direction. That said, a person still has a limited budget of such causal connections (you cannot reason about a group of arbitrarily many people, with fixed non-zero amount of paying attention to the individual details of ... (read more)

I'm curious what is the evidence you see that this is false as a description of the values of just about every human, given that

• I, a human [citation needed] tell you that this seems to be a description of my values.
• Almost every culture that ever existed had norms that prioritized helping family, friends and neighbors over helping random strangers, not to mention strangers that you never met.
• Most people don't do much to help random strangers they never met, with the notable exception of effective altruists, but even most effective altruists only go that

First, you can consider preferences that are impartial but sublinear in the number of people. So, you can disagree with Nate's room analogy without the premise "stuff only matters if it adds to my own life and experiences".

Second, my preferences are indeed partial. But even that doesn't mean "stuff only matters if it adds to my own life and experiences". I do think that stuff only matters (to me) if it's in some sense causally connected to my life and experiences. More details here.

Third, I don't know what do you mean by "good". The questions that I unders... (read more)

and, i'd guess that one big universe is more than twice as Fun as two small universes, so even if there were no transaction costs it wouldn't be worth it. (humans can have more fun when there's two people in the same room, than one person each in two separate rooms.)

This sounds astronomically wrong to me. I think that my personal utility function gets close to saturation with a tiny fraction of the resources in universe-shard. Two people is one room is better than two people in separate rooms, yes. But, two rooms with trillion people each is virtually t... (read more)

There's also the ALTER prize for progress on the learning-theoretic agenda.

Yes, absolutely! The contest is not a publication venue.

The spectrum you're describing is related, I think, to the spectrum that appears in the AIT definition of agency where there is dependence on the cost of computational resources. This means that the same system can appear agentic from a resource-scarce perspective but non-agentic from a resource-abundant perspective. The former then corresponds to the Vingean regime and the latter to the predictable regime. However, the framework does have a notion of prior and not just utility, so it is possible to ascribe beliefs to Vingean agents. I think it makes sense... (read more)

The problem of future unaligned AI leaking into human imitation is something I wrote about before. Notice that IDA-style recursion help a lot, because instead of simulating a process going deep into the external timeline's future, you're simulating a "groundhog day" where the researcher wakes up over and over at the same external time (more realistically, the restart time is drifting forward with the time outside the simulation) with a written record of all their previous work (but no memory of it). There can still be a problem if there is a positive proba... (read more)

I think it's a terrible idea to automatically adopt an equilibrium notion which incentivises the players to come up with increasingly nasty threats as fallback if they don't get their way. And so there seems to be a good chunk of remaining work to be done, involving poking more carefully at the CoCo value and seeing which assumptions going into it can be broken.

I'm not convinced there is any real problem here. The intuitive negative reaction we have to this "ugliness" is because of (i) empathy and (ii) morality. Empathy is just a part of the utility fun... (read more)

This is a fascinating result, but there is a caveat worth noting. When we say that e.g. AlphaGo is "superhuman at go" we are comparing it humans who (i) spent years training on the task and (ii) were selected for being the best at it among a sizable population. On the other hand, with next token prediction we're nowhere near that amount of optimization on the human side. (That said, I also agree that optimizing a model on next token prediction is very different from optimizing it for text coherence would be, if we could accomplish the latter.)

it would be the best possible model of this type, at the task of language modeling on data sampled from the same distribution as MassiveText

Transformers a Turing complete, so "model of this type" is not much of a constraint. On the other hand, I guess it's theoretically possible that some weight matrices are inaccessible to current training algorithms no matter how much compute and data we have. It seems also possible that the scaling law doesn't go on forever, but phase-transitions somewhere (maybe very far) to a new trend which goes below the "irreducible" term.