Last week, 27 highly prominent AI researchers and neuroscientists released a preprint entitled Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolution. I think this report is definitely worth reading, especially for people interested in understanding and predicting the long-term trajectory of AI research.

Below, I’ll briefly highlight four passages from the paper that seemed particularly relevant to me.

Doubts about the 'prosaic' approach yielding AGI

The authors write:

The seeds of the current AI revolution were planted decades ago, largely by researchers attempting to understand how brains compute (McCulloch and Pitts 1943). Indeed, the earliest efforts to build an “artificial brain” led to the invention of the modern “von Neumann computer architecture,” for which John von Neumann explicitly drew upon the very limited knowledge of the brain available to him in the 1940s (Von Neumann 2012). The deep convolutional networks that catalyzed the recent revolution in modern AI are built upon artificial neural networks (ANNs) directly inspired by the Nobel-prize winning work of David Hubel and Torsten Wiesel on visual processing circuits in the cat (Hubel and Wiesel 1962; LeCun and Bengio 1995). Similarly, the development of reinforcement learning (RL) drew a direct line of inspiration from insights into animal behavior and neural activity during learning (Thorndike and Bruce 2017; Rescorla 1972; Schultz, Dayan, and Montague 1997). Now, decades later, applications of ANNs and RL are coming so quickly that many observers assume that the long-elusive goal of human-level intelligence—sometimes referred to as “artificial general intelligence”—is within our grasp. However, in contrast to the optimism of those outside the field, many front-line AI researchers believe that major new breakthroughs are needed before we can build artificial systems capable of doing all that a human, or even a much simpler animal like a mouse, can do [emphasis added].

To the degree we take this final comment seriously—that many within the field think that major breakthroughs (plural) are needed before we get AGI—we should probably update in the direction of being relatively more skeptical of prosaic AGI safety. If it appears increasingly likely that AGI won't look like a clean extrapolation from current systems, it would therefore make increasingly less sense to bake the prosaic assumption into AGI safety research.

Researchers eyeing brain-based approaches

While many key AI advances, such as convolutional ANNs and RL were inspired by neuroscience, much of the current research in machine learning is following its own path by building on previously-developed approaches that were inspired by decades old findings in neuroscience, such as attention-based neural networks which were loosely inspired by attention mechanisms in the brain (Itti, Koch, and Niebur 1998; Larochelle and Hinton 2010; Xu et al. 2015). New influences from modern neuroscience exist, but they are spearheaded by a minority of researchers. This represents a missed opportunity. Over the last decades, through efforts such as the NIH BRAIN initiative and others, we have amassed an enormous amount of knowledge about the brain. This has allowed us to learn a great deal about the anatomical and functional structures that underpin natural intelligence. The emerging field of NeuroAI, at the intersection of neuroscience and AI, is based on the premise that a better understanding of neural computation will reveal basic ingredients of intelligence and catalyze the next revolution in AI, eventually leading to artificial agents with capabilities that match and perhaps even surpass those of humans [emphasis in the original]. We believe it is the right time for a large-scale effort to identify and understand the principles of biological intelligence, and abstract those for application in computer and robotic systems.

Again, to the degree that this assertion is taken seriously—that a better understanding of neural computation really would reveal the basic ingredients of intelligence and catalyze the next revolution in AI—it seems like AGI safety researchers should significantly positively update in the direction of pursuing brain-based AGI safety approaches.

As I’ve commented before, I think this makes a great deal of sense given that the only known systems in the universe that exhibit general intelligence—i.e., our only reference class—are brains. Therefore, it seems fairly obvious that people attempting to artificially engineer general intelligence might want to attempt to emulate the relevant computational dynamics of brains—and that people interested in ensuring the advent of AGI goes well ought to calibrate their work accordingly.

Brittleness vs. flexibility & leveraging evolution's 'work'

Modern AI can easily learn to outperform humans at video games like Breakout using nothing more than pixels on a screen and game scores (Mnih et al. 2015). However, these systems, unlike human players, are brittle and highly sensitive to small perturbations: changing the rules of the game slightly, or even a few pixels on the input, can lead to catastrophically poor performance (Huang et al. 2017). This is because these systems learn a mapping from pixels to actions that need not involve an understanding of the agents and objects in the game and the physics that governs them. Similarly, a self-driving car does not inherently know about the danger of a crate falling off a truck in front of it, unless it has literally seen examples of crates falling off trucks leading to bad outcomes. And even if it has been trained on the dangers of falling crates, the system might consider an empty plastic bag being blown out of the car in front of it as an obstacle to avoid at all cost rather than an irritant, again, because it doesn’t actually understand what a plastic bag is or how unthreatening it is physically. This inability to handle scenarios that have not appeared in the training data is a significant challenge to widespread reliance on AI systems.
To be successful in an unpredictable and changing world, an agent must be flexible and master novel situations by using its general knowledge about how such situations are likely to unfold. This is arguably what animals do. Animals are born with most of the skills needed to thrive, or can rapidly acquire them from limited experience, thanks to their strong foundation in real-world interaction, courtesy of evolution and development (Zador 2019). Thus, it is clear that training from scratch for a specific task is not how animals obtain their impressive skills; animals do not arrive into the world tabula rasa and then rely on large labeled training sets to learn. Although machine learning has been pursuing approaches for sidestepping this tabula rasa limitation, including self-supervised learning, transfer learning, continual learning, meta learning, one-shot learning and imitation learning (Bommasani et al. 2021), none of these approaches comes close to achieving the flexibility found in most animals. Thus, we argue that understanding the neural circuit-level principles that provide the foundation for behavioral flexibility in the real-world, even in simple animals, has the potential to greatly increase the flexibility and utility of AI systems. Put another way, we can greatly accelerate our search for general-purpose circuits for real-world interaction by taking advantage of the optimization process that evolution has already engaged in [emphasis added]. (Gupta et al. 2021; Stöckl, Lang, and Maass 2022; Koulakov et al. 2022; Stanley et al. 2019; Pehlevan and Chklovskii 2019).

This excerpt is potent for at least two reasons, in my view:

First, it does a good job of distilling the fundamental limitations of current AI systems with a sort of brittleness-flexibility distinction. Again, the prosaic AGI safety regime is significantly less plausible if current AI systems exhibit dynamics that seem to explicitly fly in the face of the generality we should expect from an artificial, uh, general intelligence. In other words, when monkeying with a few pixels totally breaks our image classifier or our RL model, we will probably want to regard this a suspicious form of overfitting to the specific task specification as opposed to an acquisition of the general principles that would enable success. AGI should presumably behave far more like the latter than the former, which is a strong indication that proasic AI approaches would need to be supplemented by major additional breakthoughs before we get AGI.

Second, this section nicely articulates the fact that the optimization process of evolution has already done a great deal of 'work' in narrowing the search space for the kinds of computational systems that give rise to general intelligence. In other words, researchers do not have to reinvent the wheel—they might instead choose to investigate the dynamics of generally intelligent systems that already tangibly exist. (Note: this is not to say that there won't be other solutions in possible-space-of-generally-intelligent-systems that evolution did not 'find.') Both strategies—'reinvent the brain' & understand the brain—and are certainly available as engineering proposals, but the latter seems orders of magnitude faster, akin to finetuning a pretrained model rather than training one from scratch.

Good response to the 'bird : plane :: brain : AI' analogy

The researchers somewhat epically conclude their paper as follows:

Despite the long history of neuroscience driving advances in AI and the tremendous potential for future advances, most engineers and computational scientists in the field are unaware of the history and opportunities. The influence of neuroscience on shaping the thinking of von Neumann, Turing and other giants of computational theory are rarely mentioned in a typical computer science curriculum. Leading AI conferences such as NeurIPS, which once served to showcase the latest advances in both computational neuroscience and machine learning, now focus almost exclusively on the latter. Even some researchers aware of the historical importance of neuroscience in shaping the field often argue that it has lost its relevance. “Engineers don’t study birds to build better planes” is the usual refrain. But the analogy fails, in part because pioneers of aviation did indeed study birds (Lilienthal 1911; Culick 2001), and some still do (Shyy et al. 2008; Akos et al. 2010). Moreover, the analogy fails also at a more fundamental level: The goal of modern aeronautical engineering is not to achieve “bird-level” flight, whereas a major goal of AI is indeed to achieve (or exceed) “human-level” intelligence. Just as computers exceed humans in many respects, such as the ability to compute prime numbers, so too do planes exceed birds in characteristics such as speed, range and cargo capacity. But if the goal of aeronautical engineers were indeed to build a machine with the “bird-level” ability to fly through dense forest foliage and alight gently on a branch, they would be well-advised to pay very close attention to how birds do it. Similarly, if AI aims to achieve animal-level common-sense sensorimotor intelligence, researchers would be well-advised to learn from animals and the solutions they evolved to behave in an unpredictable world.

I think this is a very concise and cogent defense of why brain-based approaches to AGI and AGI safety are ultimately highly relevant—and why the 'bird : plane :: brain : AI' reply seems far too quick.

Concluding thought

The clear factual takeaway for me is that many of the field’s most prominent researchers are explicitly endorsing 'NeuroAI' as a plausible route to AGI. This directly implies that those interested in preventing AGI-development-catastrophe-scenarios should be paying especially careful attention to this shift of researchers' attention—and calibrate their efforts accordingly.

Practically, this looks to me like updating in the direction of taking brain-based AGI safety significantly more seriously.