In this post,^[1] intended for a broad audience, I will paint a brief picture of what I’m talking about when I talk about “AGI”. It will seem obvious to many people, and obviously wrong to many others! So let’s jump in:

“AI” as most people think of it today	The future “AGI” I’m concerned about
To make AIs better at a task, we need to do R&D—gather more training data, build new training environments, change or scale up the algorithms, etc.—and make a new, better AI.	We can make one AGI design, and we’re done. Many copies of it can autonomously learn to do everything in the global economy—just as many copies of one human brain design, barely changed since the African savannah, built the global economy from scratch.

For example, today, if you want an AI to drive a car, or to control a computer using a mouse, it’s a huge project involving dozens of experts working for years to make a new AI. Whereas if you want a human to do the same, you don’t need to do R&D to breed a new subspecies of human! Instead, you just take an ordinary human—basically the same design from 100,000 years ago—and give them a few hours of practice, and you’re done. Someday we’ll have an AGI design which can do things like that.

Robotics is an area where it’s especially clear that we don’t have AGI yet, because the human brain trounces current AI technology. I’m still waiting for my AI robot butler, alas, despite companies spending billions on R&D. But if you delete the AI software, and get a human teleoperator instead, then a cheap robot today can easily do the laundry, make coffee, and much more. Source.

“AI” as most people think of it today	The future “AGI” I’m concerned about
We’re imagining a tool that humans use.	We’re imagining an agent (or team of agents) that can figure things out, take initiative, get stuff done, make plans, pivot when the plans fail, find and implement out-of-the-box solutions when it gets stuck, autonomously invent new science and technology, …

In this case, many people’s mental image of AI is already transitioning from “tool” towards “agent”, especially in the past year or two, after they’ve watched LLM agents execute on projects. But even those people are usually not going far enough for what I have in mind. Think of things that took a whole society of humans to do over an extended period of time—like inventing language and science from scratch, and developing them all the way into space travel and microchips and skyscrapers. These AGIs will be able to do those kinds of things too, fully autonomously.

“AI” as most people think of it today	The future “AGI” I’m concerned about
Normal-sounding discourse: GDP might go up by X%, unemployment by Y%, various effects on work, school, media, politics…	Sounds like crazy sci-fi stuff: a new intelligent species which will eventually vastly outnumber humans; think much faster than humans; be more insightful, creative, competent, and experienced than humans…

Now, don’t be put off by “crazy sci-fi stuff”—indeed, every technology that exists today was “crazy sci-fi stuff” before it was invented!

Left to right are from Metropolis (1927), Woman in the Moon (1929), and Gowy’s The Fall of Icarus (1636)

So the wrong question is: “Is it sci-fi?”. The right question is: “Is it possible?”

And the answer to that question is “Yes”! And we know this because we have an existence proof. Human brains and bodies can do all these things, and they don’t work by magic, but rather follow the principles of physics, math, and engineering, like everything else.

And whatever engineering principles allow humans to do all those right-column things, we should expect future scientists to sooner or later figure out how to exploit those same principles, in order to accomplish the same things. Even if that might seem impossible today! After all, think of how impossible vision must have seemed 1000 years ago: your eyes provide a magical window through which knowledge of your surroundings enters your soul. But now we understand the big-picture principles that explain how eyes work, and we have our own technology (cameras) based on similar principles. Ditto with hearing (microphones), moving (actuators), digestion (industrial catalysts), and so on.

“AI” as most people think of it today

The future “AGI” I’m concerned about

LLMs of today, and perhaps also the somewhat-better LLMs already under development

It’s controversial:

Maybe “where LLMs are eventually heading”?

Or maybe “a different AI paradigm entirely”?

As it happens, my own opinion is “a different AI paradigm entirely”. But I nevertheless expect AGI to emerge in my lifetime, and maybe even the 2030s.

Remember, new AI paradigms can develop quickly! For example:

Judge for yourself, but from my perspective, it really doesn’t feel like these movies came out a very long time ago. We’re not talking about ancient history here! But think of how much has happened in AI since 2018, to say nothing of 2012. It’s wild!

So by analogy, if you try to project forward in AI by ten years, or even by less than ten years, it’s hard to rule anything out. There might be some AI paradigm that doesn’t even exist today, and yet ten years from now it will have already been subjected to 100,000 person-years of R&D, and trillions of dollars of investment. Or maybe not! We just don’t know.

“AI” as most people think of it today

The future “AGI” I’m concerned about

We’re worried about bad actors, inequality, proliferation of destructive technologies, etc.

We’re worried about bad actors, inequality, proliferation of destructive technologies, etc.

…AND, we’re worried about people accidentally making AGIs which are themselves bad actors!

Here we move into AI concerns. We have all the usual concerns, plus a big new one: the AGIs themselves might be “bad actors”—and bad actors which can think much faster than us, and be more creative and competent, and which can self-reproduce onto any chips they can rent or hack into, and so on.

“AI” as most people think of it today	The future “AGI” I’m concerned about
AI robustness, reliability, and common sense are generally part of the solution.	AI robustness, reliability, and common sense can instead be part of the problem.

If mobsters are trying to kill you, you’re better off if the mobsters all have dementia. By the same token, if an AGI is out to get you, then robustness, reliability, and common sense are all making your prospects worse, not better. The key is instead alignment: What is the AGI trying to do? Is it trying to help, or is it out to get you?

I mentioned above that AGI will be kinda like a new intelligent species on our planet. If so, we’d better make sure it’s a species we want to share the planet with—and that wants to share the planet with us! They could make life great for us humans, or they could kill us all and run the world by themselves. The stakes of alignment could hardly be higher.

Thanks Justis Mills and Linda Linsefors for critical comments on earlier drafts.

^{^}
This post is kinda a revised “version 2” of my post from 2024: “Artificial General Intelligence”: an extremely brief FAQ.