Zvi recently coined a nice distinction between AGI-pilled and ASI-pilled: as more people are reckoning with the speed of progress, some are beginning to come around to the fact that AI could be superhuman at a huge range of tasks in a way that radically reshapes society, but not that it could leap us to the end of the tech tree shortly afterwards and start making nanotech and dyson spheres.

Hence, you see people saying things like “Because of AI we must rethink the entire notion of mathematical progress”, or “I have to make sure I amass lots of capital before most jobs get automated”: pretty crazy statements, and yet, still not quite grappling with how wild things could get in the next few years, let alone the next decade.

I feel a particular frustration here as someone who is both pretty ASI-pilled and pretty into liberal democracy. And when I say liberal democracy I don’t so much mean the specific political structure that seems efficient at our current tech level but rather the set of underlying moral commitments like non-domination and a government of the people — commitments which I’m aware might soon be a lot more expensive.

What to do? Well, on the one hand, there’s a growing body of legitimately AGI-pilled liberalism, pushing for things like pluralistic alignment and AI-assisted democracy, but mostly not really reckoning with the possibility of humans losing all meaningful leverage — that even if we could align AIs, we would still be facing an earth-shattering upheaval of our political economy. And on the other hand, the default position among the ASI-pilled seems to be that sooner or later we’re just going to have to hand over all power to Claude 8 Pantheon. Honestly, I find myself a little adrift.

*De parabel der blinden, by Pieter Bruegel the Elder*

These two factions are both correctly noticing that modern liberal democracies are wholly unequipped for what is coming. We can already see legislative processes struggling to keep up with the state of current AI — which, I must remind you, is the weakest it will ever be.

And the liberal democracy crowd is correctly noticing that actually there’s a lot you can do to shore this up against AGI — indeed, lots of ways AGI could help with AGI. It might even be able to help reverse some of the brutal erosion that liberal democracy has been suffering of late. They are also quite prudently continuing to observe that alignment is not only a technical problem but also a political and moral question about what values should have influence over the future.

Meanwhile, the Claude 8 Pantheon crowd is correctly noticing that it’s possible we’re headed for a reasonably fast takeoff (as in, less than a term of office) that completely flips the gameboard, and it’s a heck of a lot simpler to just put an aligned AI in charge rather than trying to retrofit deliberative processes. Liberalism and democracy are both pretty demanding, and we may not be in a position to make many demands of our solutions. Also, maybe aligned AGI would actually be more moral than us?

I could anthropologise more about the STEM high modernists not realising they’re high modernists and the humanities people who don’t like straight lines, but ultimately that’s just a way of dodging the harder and more important question, which is: what do I even want here? What is this ASI liberalism thing I’m trying to stand up for?

It’s not liberal values, like having people be safe from harm. I think even the people who want to hand over power to an ASI ASAP are kind of hoping said ASI will lock those things in. In fact, I think many of them believe that a fairly swift handover is in expectation the best way to safeguard those values.

It’s also not liberal institutions, like everybody voting for a local representative every N years or a (mostly) free market revealing the fair price of goods. These aren’t really core to what I care about, so much as mechanisms that enact it, given certain ambient constraints. And if those constraints change — as I suspect they will — then so should the mechanisms.^[1]

What it is, I think, is a secret third thing getting lost in the middle — the bit where the government is a thing made up of people, and the people have power, and the moral right of the government to exercise power comes from the people in it.

I think the best way to pin down this third thing is by looking at where it’s in tension with the others. Or rather, hopefully we can all agree that hollow zombified remnants of democratic institutions are not great, and that specific principles like voting structures are valuable for what they do, rather than for their ceremonial aesthetic. So let’s focus on how the process ends up in tension with individual values and outcomes.

Here’s one example: I do not like benevolent dictators, not because they might secretly be evil but because they have the capacity to arbitrarily infringe on people’s rights, whether they exercise it or not. There’s a long digression to be had about the fragility of consequentialism and the nature of decision procedures, but ultimately, I’m not a utilitarian.^[2]

Relatedly, I think it’s very bad to manipulate or coerce people even in pursuit of saving the world. That might sound like an extremely lukewarm take but, well, unfortunately I think lots of people disagree. And to be fair, this conviction of mine does have some unpleasant consequences that I accept: if an idealised democracy collectively decided to blow itself up with me inside then I think that would be its right and I wouldn’t want to stop it. There are situations where I’d rather die with dignity than live without.^[3]

I get all torn up about technocracy because I believe there are smarter ways to make society function but I also kind of like the fact that we do things the slow way where everyone is involved. And sure it’s not perfect — it’s all kinds of broken — but there’s something important there. I was actually a bit surprised when I realised this, but apparently I’m some kind of bleeding heart patriot.

And when I read stuff like Ian M. Banks’s Culture novels, where mankind is ruled over by machines of loving grace, I do get a bit of visceral revulsion. I do not want to put on the whispering earring. I do not want man to be abolished.

*Still from (and* *rot13* *spoiler for) Gur Gehzna Fubj*

Of course, none of this is new — there’s a well-oiled subfield of research full of people who keep saying things like “it is important that the values we align AIs to are not decided by a small group of powerful people” and “the benefits of AI progress should be widely shared”. Unfortunately, it feels to me like many of the people saying this just don’t seem to get it — what they want to do about all this is, like, fund upskilling programmes and change what goes in the constitution and support development in third world countries and do more participatory democracy schemes instead of trying to prevent the potential oncoming apocalypse.

If you are worried about literally everybody dying in five-ish years and the light of human civilization winking out for good, it’s sometimes a bit hard to take such people seriously. “But my participatory democracy programme will help us navigate the challenges and opportunities of AI,” they say. “Sure thing buddy,” you reply.

(Lest you think me callous, part of what prompted this essay is a melancholy conversation about how much it sucks to feel yourself inadvertently developing a reflexive dismissal of smart and thoughtful people who are really trying unusually hard to do good in a changing world, whom you happen to disagree with about how well neural nets will generalise.)

It especially doesn’t help that these types of concern — equality and dignity and so on — are so widely recognised as important that it sometimes feels like they’re the default fig leaf that organisations pick up to Show They Care, in a way that sometimes feels like an intentional deflection from grappling with ASI.

I get the sense sometimes that people who have really stared into the void basically just feel alienated from people who have not done that. These void-gazers notice that they believe a lot of very weird things that appear to be frighteningly accurate. Democracy is hard and slow and everything is really unbelievably ten kinds of on fire. All these debates about legitimacy and pluralism are interesting preoccupations for people worried about mere AGI. Once you’re worried about ASI, you don’t really have the luxury. Politics is useful insofar as it gives a lever for influencing the world, but mostly it’s inconvenient that politicians are ‘waking up’ because now there’s a risk they’ll start doing things like wanting all this power for themselves.

It’s easy to take potshots, but I expect there is a very painful tradeoff between good outcomes and good processes, which bites so much harder if and when humans stop having anything useful to contribute. And all these other questions are somewhat moot if we’re all dead. Separately, I happen to think that if we mess these other questions up bad enough, we could still basically lose all value even if we’ve solved technical alignment, but the point I’m trying to make here is that there are some things at stake, distinct from whether we survive, and perhaps even in tension with it, which I nonetheless care about protecting.

Partly this is me grieving the coming loss of innocence. There are certain kinds of freedom people used to have that are gone now — hopping on a train and go to a new town; walking into a forest and picking out a patch of land to be your home; jumping on a boat, sail to a different country, wander right in; building your own house in whatever strange way you want. There was also a time when science was virgin soil. Maybe we’ll get some of these back, but maybe some of them are necessary sacrifices, or unavoidable features of progress.

And part of this is an extended apology for gradual disempowerment. One of the successes of the paper is that it swayed some smart people sceptical of takeover towards taking x-risk seriously. But it mostly didn’t ASI pill them, so their solutions still seem to me to be pretty unlikely to work, and in many cases I expect them to make the problem actively worse. Conversely, some ASI-pilled people were like “ah thank you for finally expressing what I’ve been thinking”, but plenty basically don’t buy it and are pushing in directions which I expect will make things worse.^[4]

Positioned as I am with feet in both worlds, I’m a bit worried that the ASI crowd is writing off liberalism a bit because the liberalism crowd is writing off ASI, and vice versa.

And heck, I do think there’s something here worth fighting for.

Maybe we get worlds where we really are watched over by machines of loving grace who also tile the further edges of the universe with happy squiggles and the literal utility scores are through the roof, and future humans are a bit like pampered kittens — the world is pretty nice for them, but not because they have any meaningful leverage, liberty, or comprehension. This sounds great from a utilitarian perspective, but I do not like it one bit.

And I do not think it is the only way — I think there are worlds where we focus on massively increasing the coordination bandwidth between humans and more generally start by maxing out our own existing potential to be smarter, wiser, and better. They may not be likely or easy, but the fire is still burning. We could start by just getting a bit clearer about what people mean when they talk about the dangers of extreme power concentration, or what we want to align ASI to, and what that would even mean.

One nice thing about the existential risk movement of yore was that it was all pretty hypothetical and we could all have our different answers to thought experiments, but now we’re getting into the period where some people are actually going to have to decide whether to hit the big button that says “give the power to Claude so that it can try to save the lightcone from the evils of men’s hearts”. As it happens, I expect I am somewhat more sceptical about whether that will turn out well, but that’s not really the core of it. If I believed it was likely to work, I would still feel bad about it, and even if it all worked out and we found ourselves in the golden fields being watched over by the proverbial machines of loving grace, I would still feel like something had been lost along the way.

^{^}
This, incidentally, will force us to ask some very difficult questions about what properties of those mechanisms we really care about, and separate the bugs from the features. For example, what are democratic representatives actually representative of? Would real communism be good if we could actually try it?
^{^}
Technical digression for philosophy nerds: The precise term for this conception of liberty is ‘republicanism’. Carlsmith and Laine have characteristically thoughtful essays on the tension between liberalism and welfare which shaped my thinking here, but the notion of liberty they focus on is itself the more Benthamite conception grounded in non-interference. That said, Laine’s piece and Carlsmith’s broader series do grapple with non-domination in spirit.
^{^}
Awkwardly I do not think we currently pass the bar for “idealised democracy”. By analogy, I support the principle of euthanasia, and have more mixed feelings about implementation. But the fact that we currently fall short of my vision of good democracy doesn’t lead me to feel that anything goes.
^{^}
It’s part of the reason why I and others have been doing things like running workshops on post-AGI civilizational equilibria, but even there, it’s really an uphill battle to get people to fully grapple with the bit about post-AGI civilizational equilibria.