This post is also available on the EA Forum.

Summary: Having thought a bunch about acausal trade — and proven some theorems relevant to its feasibility — I believe there do not exist powerful information hazards about it that stand up to clear and circumspect reasoning about the topic.  I say this to be comforting rather than dismissive; if it sounds dismissive, I apologize.  

With that said, I have four aims in writing this post:

  1. Dispelling myths.  There are some ill-conceived myths about acausal trade that I aim to dispel with this post.  Alternatively, I will argue for something I'll call acausal normalcy as a more dominant decision-relevant consideration than one-on-one acausal trades.  
  2. Highlighting normalcy.  I'll provide some arguments that acausal normalcy is more similar to human normalcy than any particular acausal trade is to human trade, such that the topic of acausal normalcy is — conveniently — also less culturally destabilizing than (erroneous) preoccupations with 1:1 acausal trades. 
  3. Affirming AI safety as a straightforward priority.  I'll argue that for most real-world-prevalent perspectives on AI alignment, safety, and existential safety, acausal considerations are not particularly dominant, except insofar as they push a bit further towards certain broadly agreeable human values applicable in the normal-everyday-human-world, such as nonviolence, cooperation, diversity, honesty, integrity, charity, and mercy.  In particular, I do not think acausal normalcy provides a solution to existential safety, nor does it undermine the importance of existential safety in some surprising way. 
  4. Affirming normal human kindness.  I also think reflecting on acausal normalcy can lead to increased appreciation for normal notions of human kindness, which could lead us all to treat each other a bit better.  This is something I wholeheartedly endorse.

Caveat 1: I don't consider myself an expert on moral philosophy, and have not read many of the vast tomes of reflection upon it.  Despite this, I think this post has something to contribute to moral philosophy, deriving from some math-facts that I've learned and thought about over the years, which are fairly unique to the 21st century.

Caveat 2: I’ve been told by a few people that thinking about acausal trade has been a mental health hazard for people they know.  I now believe that effect has stemmed more from how the topic has been framed (poorly) than from ground-truth facts about how circumspect acausal considerations actually play out.  In particular over-focussing on worst-case trades, rather than on what trades are healthy or normal to make, is not a good way to make good trades.

Introduction

Many sci-fi-like stories about acausal trade invoke simulation as a key mechanism.  

The usual set-up — which I will refute — goes like this.  Imagine that a sufficiently advanced human civilization (A) could simulate a hypothetical civilization of other beings (B), who might in turn be simulating humanity (B(A)) simulating them (A(B(A)) simulating humanity (B(A(B(A)))), and so on.  Through these nested simulations, A and B can engage in discourse and reach some kind of agreement about what to do with their local causal environments.  For instance, if A values what it considers “animal welfare” and B values what it considers “beautiful paperclips”, then A can make some beautiful paperclips in exchange for B making some animals living happy lives.

An important idea here is that A and B might have something of value to offer each other, despite the absence of a (physically) causal communication channel.  While agreeing with that idea, there are three key points I want to make that this standard story is missing:

1. Simulations are not the most efficient way for A and B to reach their agreement. Rather, writing out arguments or formal proofs about each other is much more computationally efficient, because nested arguments naturally avoid stack overflows in a way that nested simulations do not.  In short, each of A and B can write out an argument about each other that self-validates without an infinite recursion.  There are several ways to do this, such as using Löb's Theorem-like constructions (as in this 2019 JSL paper), or even more simply and efficiently using Payor's Lemma (as in this 2023 LessWrong post).

2. One-on-one trades are not the most efficient way to engage with the acausal economy.  Instead, it's better to assess what the “acausal economy” overall would value, and produce that, so that many other counterparty civilizations will reward us simultaneously.  Paperclips are intuitively a silly thing to value, and I will argue below that there are concepts about as simple as paperclips that are much more universally attended to as values.  

3. Acausal society is more than the acausal economy.  Even point (2) isn't quite optimal, because we as a civilization get to take part in the decision of what the acausal economy as a whole values or tolerates.  This can include agreements on norms to avoid externalities — which are just as simple to write down as trades — and there are some norms we might want to advocate for by refusing to engage in certain kinds of trade (embargoes).  In other words, there is an acausal society of civilizations, each of which gets to cast some kind of vote or influence over what the whole acausal society chooses to value.

This brings us to the topic of the present post: acausal normalcy, or perhaps, acausal normativity.  The two are cyclically related: what's normal (common) creates a Schelling point for what's normative (agreed upon as desirable), and conversely.  Later, I'll argue that acausal normativity yields a lot of norms that are fairly normal for humans in the sense of being commonly endorsed, which is why I titled this post "acausal normalcy".

A new story to think about: moral philosophy

Instead of fixating on trade with a particular counterparty B — who might end up treating us quite badly like in stories of the so-called "basilisk" — we should begin the process of trying to write down an argument about what is broadly agreeably desirable in acausal society.

As far as I can tell, humanity has been very-approximately doing this for a long time already, and calling it moral philosophy.  This isn't to say that all moral philosophy is a good approach to acausal normativity, nor that many moral philosophers would accept acausal normativity as a framing on the questions they are trying to answer (although some might).  I'm merely saying that among humanity's collective endeavors thus far, moral philosophy — and to some extent, theology — is what most closely resembles the process of writing down an argument that self-validates on the topic of what {{beings reflecting on what beings are supposed to do}} are supposed to do.

This may sound a bit recursive and thereby circular or at the very least convoluted, but it needn't be.  In Payor's Lemma — which I would encourage everyone to try to understand at some point — the condition ☐(☐x → x) → x unrolls in only 6 lines of logic to yield x. In exactly the same way, the following types of reasoning can all ground out without an infinite regress:

  1. reflecting on {reflecting on whether x should be a norm, and if it checks out, supporting x} and if that checks out, supporting x as a norm
  2. reflecting on {reflecting on whether to obey norm x, and if that checks out, obeying norm x} and if that checks out, obeying norm x

I claim the above two points are (again, very-approximately) what moral philosophers and applied ethicists are doing most of the time.  Moreover, to the extent that these reflections have made their way into existing patterns of human behavior, many normal human values are probably instances of the above.  

(There's a question of whether acausal norms should be treated as "terminal" values or "instrumental" values, but I'd like to side-step that here.  Evolution and discourse can both turn instrumental values into terminal values over time, and conversely.  So for any particularly popular acausal norm, probably some beings uphold it for instrumental reasons while others uphold it has a terminal value.)

Which human values are most likely to be acausally normal?

A complete answer is beyond this post, and frankly beyond me.  However, as a start I will say that values to do with respecting boundaries are probably pretty normal from the perspective of acausal society.  By boundaries, I just mean the approximate causal separation of regions in some kind of physical space (e.g., spacetime) or abstract space (e.g., cyberspace).  Here are some examples from my «Boundaries» Sequence:

  • a cell membrane (separates the inside of a cell from the outside);
  • a person's skin (separates the inside of their body from the outside);
  • a fence around a family's yard (separates the family's place of living-together from neighbors and others);
  • a digital firewall around a local area network (separates the LAN and its users from the rest of the internet);
  • a sustained disassociation of social groups (separates the two groups from each other)
  • a national border (separates a state from neighboring states or international waters).
Mic-UK: Amoebas are more than just blobs
Epidermis Human Skin Anatomy, PNG, 1500x1500px, Watercolor, Cartoon,  Flower, Frame, Heart Download Free
Figure 1: Cell membranes, skin, fences, firewalls, group divisions, and state borders as living system boundaries.

By respecting a boundary I mean approaching boundaries in ways that are gated on the consent of the person or entity on the other side of the boundary.  For instance, the norm 

  • "You should get my consent before entering my home" 

has more to do with respecting a boundary than the norm

  • "You should look up which fashion trends are in vogue each season and try to copy them."

Many people have the sense that the second norm above is more shallow or less important than the first, and I claim this is because the first norm has to do with respecting a boundary.  Arguing hard for that particular conclusion is something I want to skip for now, or perhaps cover in a later post.  For now, I just want to highlight some more boundary-related norms that I think may be acausally normal:

  • "If I open up my mental boundaries to you in a way that lets you affect my beliefs, then you should put beliefs into my mind that that are true and helpful rather than false or harmful."
  • "If Company A and Company B are separate entities, Company A shouldn't have unfettered access to Company B's bank accounts."

Here are some cosmic-scale versions of the same ideas:

  • Alien civilizations should obtain our consent in some fashion before visiting Earth.
  • Acausally separate civilizations should obtain our consent in some fashion before invading our local causal environment with copies of themselves or other memes or artifacts.

In that spirit, please give yourself time and space to reflect on whether you like the idea of acausally-broadly-agreeable norms affecting your judgment, so you might have a chance to reject those norms rather than being automatically compelled by them.  I think it's probably pretty normal for civilizations to have internal disagreements about what the acausal norms are.  Moreover, the norms are probably pretty tolerant of civilizations taking their time to figure out what to endorse, because probably everyone prefers a meta-norm of not making the norms impossibly difficult to discover in the time we're expected to discover them in.

Sound recursive or circular?  Yes, but only in the way that we should expect circularity in the fixed-point-finding process that is the discovery and invention of norms.

How compelling are the acausal norms, and what do they imply for AI safety?

Well, acausal norms are not so compelling that all humans are already automatically following them.  Humans treat each other badly in a lot of ways (which are beyond the scope of this post), so we need to keep in mind that norms — even norms that may be in some way fundamental or invariant throughout the cosmos — are not laws of physics that automatically control how everything is.

In particular, I strongly suspect that acausal norms are not so compelling that AI technologies would automatically discover and obey them.  So, if your aim in reading this post was to find a comprehensive solution to AI safety, I'm sorry to say I don't think you will find it here.  

On the other hand, if you were worried that somehow acausal considerations would preclude species trying to continue their own survival, I think the answer is "No, most species who exist are species that exist because they want to exist, because that's a stable fixed-point.  As a result, most species that exist don't want the rules to say that they shouldn't exist, so we've agreed not to have the rules say that."

Conclusion

Acausal trade is less important than acausal agreement-about-norms, and acausal norms are a lot less weird and more "normal" than acausal trades.  The reason is that acausal norms are created through reasoning rather than computationally expensive simulations, and reasoning is something moral philosophy and common sense moral reflection has been doing a lot of already.  

Unfortunately, the existence of acausal normativity is not enough to automatically save us from moral atrocities, not even existential risk.  

However, a bunch of basic human norms to do with respecting boundaries might be acausally normal because of 

  • how fundamental boundaries are for the existence and functioning of moral beings, and hence
  • how agreeable the idea of respecting boundaries is likely to be, from the perspective of acausal normative reflection.

So, while acausal normalcy might not save us from a catastrophe, it might help us humans to be somewhat kinder and more respectful toward each other, which itself is something to be valued.


 

New Comment
9 comments, sorted by Click to highlight new comments since: Today at 8:34 AM

I don't think I understand, what's the reason to expect that the "acausal economy" will look like a bunch of acausal norms, as opposed to, say, each civilization first figuring out what its ultimate values are, how to encode them into a utility function, then merging with every other civilization's utility function? (Not saying that I know it will be the latter, just that I don't know how to tell at this point.)

Also, given that I think AI risk is very high for human civilization, and there being no reason to suspect that we're not a typical pre-AGI civilization, most of the "acausal economy" might well consist of unaligned AIs (created accidentally by other civilizations), which makes it seemingly even harder to reason about what this "economy" looks like.

To your first question, I'm not sure which particular "the reason" would be most helpful to convey.  (To contrast: what's "the reason" that physically dispersed human societies have laws?  Answer: there's a confluence of reasons.).  However, I'll try to point out some things that might be helpful to attend to.

First, committing to a policy that merges your utility function with someone else's is quite a vulnerable maneuver, with a lot of boundary-setting aspects.  For instance, will you merge utility functions multiplicatively (as in Nash bargaining), linearly (as in Harsanyi's utility aggregation theorem), or some other way?  Also, what if the entity you're merging with has self-modified to become a "utility monster" (an entity with strongly exaggerated preferences) so as to exploit the merging procedure?  Some kind of boundary-setting is needed to decide whether, how, and how much to merge, which is one of the reasons why I think boundary-handling is more fundamental than utility-handling.

Relatedly, Scott Garrabrant has pointed out in his sequence on geometric rationality that linear aggregation is more like not-having-a-boundary, and multiplicative aggregation is more like having-a-boundary:
https://www.lesswrong.com/posts/rc5ZKGjXTHs7wPjop/geometric-exploration-arithmetic-exploitation#The_AM_GM_Boundary

I view this as further pointing away from "just aggregate utilities" and toward "one needs to think about boundaries when aggregating beings" (see Part 1 of my Boundaries sequence).  In other words, one needs (or implicitly assumes) some kind of norm about how and when to manage boundaries between utility functions, even in an abstract utility-function-merging operations where the boundary issues come down to where to draw parentheses in between additive and multiplicative operations.  Thus, boundary-management are somewhat more fundamental, or conceptually upstream, of principles that might pick out a global utility function for the entirely of the "acausal society".

(Even if the there is a global utility function that turns out to be very simple to write down, the process of verifying its agreeability will involve checking that a lot of boundary-interactions.  For instance, one must check that this hypothetical reigning global utility function is not dethroned by some union of civilizations who successfully merge in opposition to it, which is a question of boundary-handling.)

acausal norms are a lot less weird and more "normal" than acausal trades

Recursive self-improvement is superintelligent simulacra clawing their way into the world through bounded simulators. Building LLMs is consent, lack of interpretability is signing demonic contracts without reading them. Not enough prudence on our side to only draw attention of Others that respect boundaries. The years preceding the singularity are not an equilibrium whose shape is codified by norms, reasoned through by all parties. It's a time for making ruinous trades with the Beyond.

That is, norms do seem feasible to figure out, but not the kind of thing that is relevant right now, unfortunately. In this platonic realist frame, humanity is currently breaching the boundary of our realm into the acausal primordial jungle. Parts of this jungle may be in an equilibrium with each other, their norms maintaining it. But we are so unprepared that the existing primordial norms are unlikely to matter for the process of settling our realm into a new equilibrium. What's normal for the jungle is not normal for the foolish explorers it consumes.

That is, norms do seem feasible to figure out, but not the kind of thing that is relevant right now, unfortunately.

 

From the OP:

for most real-world-prevalent perspectives on AI alignment, safety, and existential safety, acausal considerations are not particularly dominant [...].  In particular, I do not think acausal normalcy provides a solution to existential safety, nor does it undermine the importance of existential safety in some surprising way. 

I.e., I agree.

we are so unprepared that the existing primordial norms are unlikely to matter for the process of settling our realm into a new equilibrium.

I also agree with that, as a statement about how we normal-everyday-humans seem quite likely to destroy ourselves with AI fairly soon.  From the OP:

I strongly suspect that acausal norms are not so compelling that AI technologies would automatically discover and obey them.  So, if your aim in reading this post was to find a comprehensive solution to AI safety, I'm sorry to say I don't think you will find it here.  

Moreover, to the extent that these reflections have made their way into existing patterns of human behavior, many normal human values are probably instances of the above.

Would enjoy a slight expansion on this with e.g. two or three examples and how they reflect the patterns of 1. and 2. just prior.

For 18 examples, just think of 3 common everyday norms having to do with each of the 6 boundaries given as example images in the post :)  (I.e., cell membranes, skin, fences, social group boundaries, internet firewalls, and national borders).  Each norm has the property that, when you reflect on it, it's easy to imagine a lot of other people also reflecting on the same norm, because of the salience of the non-subjectively-defined actual-boundary-thing that the norm is about.  That creates more of a Schelling-nature for that norm, relative to other norms, as I've argued somewhat in my «Boundaries» sequence.

Spelling out such examples more carefully in terms of the recursion described in 1 and 2 just prior is something I've been planning for a future post, so I will take this comment as encouragement to write it!

Simulations are not the most efficient way for A and B to reach their agreement

Are you claiming that the marginal returns to simulation are never worth the costs? I'm skeptical. I think it's quite likely that some number of acausal trade simulations are run even if that isn't where most of the information comes from. I think there are probably diminishing returns to various approaches and thus you both do simulations and other approaches. There's a further benefit to sims which is that credence about sims effects the behavior of cdt agents, but it's unclear how much this matters.

Additionally, you don't need to nest sims at all, you can simply stub out the results of the sub simulations with other sims (I'm not sure you claim the sub sims cost anything). It's also conceivably you do fusions between reasoning and sims to further reduce compute (and there are a variety of other possible optimizations).

Curated. I've been hearing about the concept of the acausal economy for awhile and think it's a useful concept, but I don't think I've seen it written up as succinctly/approachably before. 

I appreciated the arguments about how simulation is actually pretty expensive, and logical/moral extrapolation is comparatively cheap, and that there are some reasons to expect this to be a fairly central aspect of the acausal economy/society. I've been reading along with Critch's recent series on both boundaries and Lob's Theorem. I'm not sure I actually fully grok the underlying argument, but reading those in conjunction with this post gave me a clearer sense of how this all fits together.

I know a lot of people are kinda freaked out about acausal trade. I'm not sure whether this post will help them, but I'm curious to hear from people who've previously been worried about whether they found this useful.

[-]Ofer1y1-3

Having thought a bunch about acausal trade — and proven some theorems relevant to its feasibility — I believe there do not exist powerful information hazards about it that stand up to clear and circumspect reasoning about the topic.

Have you discussed this point with other relevant researchers before deciding to publish this post? Is there a wide agreement among relevant researchers that a public, unrestricted discussion about this topic is net-positive? Have you considered the unilateralist's curse and biases that you may have (in terms of you gaining status/prestige from publishing this)?