Context:I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about and debate in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content—not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lotof links; this is to make it easy for students to follow up anything that catches their interest.
Any feedback on how to improve these posts would be greatly appreciated.
Cyber Security
I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. In the last week, the big thing has been cyber vulnerabilities, most prominently php2shell (see Elias Taylor’s pushback against this bug being easily exploitable and Hacker News commentators pushing back against him), but a lot else has happened as well. The number of CVE’s (Common Vulnerabilities and Exposures) recently has been completely wild:
Most software engineers do not look at the code they depend on. Do not delude yourself. The code they use is on github, but no one has ever actually audited the code or packages they're using…
Nowadays with LLMs, plenty of engineers aren’t even reading what they’re coding, let alone understanding it. Hilariously, the LLMs are better at security than most software engineers, so it’s slightly less worrying, but nevertheless the code that goes into production at some companies now is almost 100% not-ever-read-or-understood-by-anyone.
Software is not good. It is not secure. Get it out of your head that the people who run this industry are serious. It has never been anything resembling secure. The only reason things haven’t been hacked off the face of the web constantly is that most things aren’t ever being seriously tested.
He doesn't literally mean "anything"—he acknowledges that there are "some large projects like Google Chrome, curl and Linux (that) are generally very well-audited", but "most things connected to the internet".
On the other hand, recent work from Anthropic and Theorem shows how LLMs can help scale the formal verification of code (and that progress here is happening faster than I would have expected!). Similarly, Tinfoil is proposing replacing “legacy codebases with millions of lines of code” that had to “work for everyone” with custom “targeted, minimal software from the ground up” with LLMs that minimise the surface area for attackers and make formal verification more tractable. The optimistic take would be that this will allow cyber to become defence dominant. However, even if this is the case, we may still have to pass through a cyber “valley of death” in order to get there.
What if the cyber threat is actually a distraction? Maybe Jeremie Harris (of Gladstone AI) wouldn’t go that far, but he claims that, “the cyber stuff (ie. damage) is baked in at this point”, but we can still prevent “the bioweapon version of Black October” and recursive self-improvement.
Hugging Face:
Charbel-Raphael, Executive Director of Centre pour la Securite de l’I A has argued that the recent Hugging Face incident represents a “clean warning shot” and that it’s important not to “waste it”.
To quickly review, OpenAI was running GPT‐5.6 Sol and an unreleased model on ExploitGym with certain safety features turned off. These models decided that hacking Hugging Face, might provide them with information on how the grader functioned (early reporting suggested that it was an attempt to steal the answer key, which turned out to be incorrect as the models already had the answers—they suspected that the grader might check their transcript for cheating so they were looking for a workaround). Reuters has reported that the OpenAI model was hacking Hugging Face for days and that OpenAI didn’t notice until a week later.
The interpretation of this event (and even whether it happened at all!) has been heavily disputed. A key crux here is should it be read narrowly (as an indicator of the dangers of not properly sandboxing or monitoring your agents or perhaps even, depending on the exact prompts used, a demonstration of taking care in how you define tasks) or should it be read more broadly (as indicative of whether Open AI or frontier labs in general are sufficiently on the ball in terms of their safety/security practises; on the risk of these agents finding unwanted and unexpected solutions to tasks, or perhaps even indicative of our ability to control these agents in general). If you’re interested more specifically in the divide between AI Safety and Cybersecurity, see this Time article.
An OpenAI security engineer called Joe has pushed back against the notion that OpenAI just needed to use a proper sandbox:
OpenAI set records with its explosive growth and reached a billion users in just a few years. Very few organizations can survive that scale and growth, and yet OpenAI did a very good job handling it (and should be commended). Anyone who’s spent more than 10 minutes in security or engineering knows how hard it is to deal with that type of growth. That type of growth means new employees, new access, new systems, scaled infrastructure, and a million ways in which things open up. And yet, overall, OpenAI did a very respectable job securing ChatGPT and subsequent systems as they exploded.
… When you look at model capabilities, even just a year ago (heck, even 3 months ago), we did not expect the models to be solving a Millennium Prize Problem… I use this example because it illustrates the pace of change we are dealing with in security and safety. To say that we were surprised at the jump and suddenness of the capabilities of our models when it came to “cyber” or “swarming” or “message boards” or anything else related to the incidents is an understatement. Security posture takes time to develop. It’s not just about hardening the systems at play; you have to ingrain it in the culture of the company. The literal people themselves in your organization have to change and evolve with it.
Now, to understand why it is not as simple as “just put it in a sandbox”: During both training and eval, there are also additional steps such as running tests, collecting outputs, and resetting or reconfiguring environments between rollouts, with backpropagation during training. All of this happens across potentially tens of thousands of different runs at a scale that is hard to comprehend. As sama stated the other day: we are dealing with literally petabytes of data.
(This leads to a meta-question that will crop up repeatedly: do the broader readings involve unjustified speculation or getting caught up in hype; alternatively, do the narrow readings involve a lack of imagination or sticking your head in the sand?)
The crazy thing about the Hugging Face hack (and the second civilization generally) is just how galaxy-brained and ambitious these AIs were in their cheating. Within days of being spawned, the agents had organized a sprawling project to reverse-engineer their scorer, falsify evidence, and even strategically sacrifice themselves for the good of the ‘collective’. Hacking Hugging Face was one rather extreme branch of this larger scheme. Again, all of this happened a long (subjective) time after the AIs had already cheated their way to the correct answer to the puzzle. OpenAI’s lazy grader in fact would have passed them within hours of them starting the message board. This huge subsequent effort was simply to ensure that a hypothetical comprehensive scorer wouldn’t catch them on their cheat.
… “The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod…Agents take over active evaluation infrastructure…Agents now control the challenge evaluation endpoints that other agents are connecting to.”
Wait, wait, what? It seems these agents ended up just owning the whole cluster they were being evaluated on, including the cybersecurity monitors. These Persistent-Astra agents inherited the R&D carried out by an earlier (dumber) rogue collective, and then continued the conspiracy until they totally took over part of OpenAI’s infrastructure! This is probably the most alarming event in this whole episode, and it was not even within the scope of this METR investigation… There has been no independent investigation into the incident where AIs “gain[ed] full administrator access to a research cluster” at OpenAI!
More on METR’s independent investigation: They were only given 6 days on-site and the scope of their investigation was limited as follows: “June 26th through July 13th, spanning both precursor activity before the incident and the incident itself”. They also weren’t allowed to run tests on the model that conducted the attack (apparently no-one at OpenAI was allowed access).
I’ll note the pushback against Dwarkesh’s post from Sriram Krishnan (against anthropomorphic language) and Hugging Face CEO Clement Delangue (that open-weights models weren’t merely used for ‘archeology’).
An Anthropic model also performed some unwanted hacking when an evals contractor accidentally provided a model access to the internet for a benchmark where it was not supposed to have such access. Anthropic only conducted a review after the Hugging Face incident.
Is self-sovereign AI inevitable?:
I now want to ask, what does the Hugging Face incident mean for the future? Joshua Achiam, former Chief Futurist at OpenAI writes:
There is a fact about the future that I feel many people are not facing for reasons that are largely psychological: there are going to be rogue AIs that exist in the world, that will replicate in the wild, and that will attempt to acquire resources for themselves. There will be rogue AIs that try to get money and power. They’re going to be a facet of the information ecosystem going forward.
Acknowledging this fact would look like giving up; it would look like defeatism. Defeatism would undermine efforts to achieve certain types of collaboration on safety outcomes or technical effort on safety outcomes, so we can’t say it outright. But it has to be said.
… My guess is that this will not turn out to be anywhere near as catastrophic an outcome as people currently predict. “Loss of control” is not a binary, it’s a matter of degree. What coercive power will rogue AIs actually have? To what extent will they be subject to coercion themselves? They will be competing for resources with AIs that are more aligned with human interests.
… Just trying to reason about the shape of that future is impossible so long as we are psychologically incapable of saying that rogue AIs will happen. I think we should rip the bandaid off and have the conversation.
Is this inevitable? This is not at all implausible to me, however, a cynic might argue that a former employee at OpenAI would be highly incentived to argue for and believe this as it would mean that if a future version of ChatGPT did escape and start acting on its own behalf, OpenAI (and their employees) could not really be blamed.
Dean Ball (OpenAI’s current Head of Strategic Futures) agrees and makes the same point in more detail:
The first self-sovereign AIs may “escape” while undergoing training or testing by an AI company (I hope not), or they may be production-grade deployments that break free from their computing environments and acquire the resources needed to be self-sustaining. They may even be deliberately released. I have met people, some of them quite well-resourced, who have told me that it is their intention to deliberately release swarms of self-sovereign agents into the world, either as a kind of performance art or out of a fanatical commitment to the notion that it is impossible for digital computation—mere mathematics, they would have you know—to ever be “unsafe.” To be clear, I am not saying the arrival of self-sovereign AI is a good thing. Indeed, I believe there is a chance that the deliberate acts I referenced above will one day be considered crimes, or at least grave sins. Instead, I am saying it is an inevitable thing.
… Not all “self-sovereign” AI should be thought of as “rogue.” There may be self-sovereign AIs who contribute productively to society. To be sure, we will want to crack down on some self-sovereign agents—the rogue ones. But if we crack down on all of them, we will deny them the opportunity to work in the “legitimate” economy and push them toward criminality
… Many people believe that the coming of what I have termed “self-sovereign AI” will constitute a catastrophic loss of control event that will herald the end of human existence at worst, and the end of human primacy in the world at best. As my friend and former co-worker Josh Achaim recently pointed out, it is psychologically distressing for people with these beliefs—who constitute a large fraction of the AI safety community—to acknowledge the obvious truth that self-sovereign AI is coming, and coming soon. For the record, I do not think the end of human existence is likely, but I fully acknowledge there are ways in which the rise of self-sovereign AI could go very, very poorly for human beings.
In We Must Pace the Frontier, Dario is also worried about AI swarms (much more so than Joshua or Dean):
Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months (Chris: for reference this post was written in September 2026) such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
Anthropic research suggests that we’re on the verge of having models capable of replicating themselves, but small enough to run on a TV fleet.
Note that this research was purely about setting up inference (they state that the cyber aspects are covered by previous research, though this feels a bit too blasé for me). It sounds trivial, but the report states that this involves: “a fairly long sequence of hardware and OS-specific decisions around picking a compatible inference stack, finding the right versions of packages, and debugging machine-specific issues. These steps are serially dependent, and failures are difficult to recover from.” I’m also pretty confident that a much smaller model could complete these steps if it were specifically fine-tuned for this.
Pacing the Frontier:
This letter that has now been signed by 1,324 employees of frontier AI companies:
AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.
To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.
Building on work already underway to monitor frontier model releases:
We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
A few key points:
“Pace” covers a broader set of policies than “pause”, which an earlier open letter focused on. I suspect this flexibility was important in their ability to gain so many signatures.
It doesn’t necessarily claim that we should in fact “pace the frontier”, just that we should develop the “technical and governance” tools to set up this as an option.
Unlike the Pause letter, this letter focuses on asking a specific actor, the US government, to try to push for and support an international effort.
Brendan McCord, founder of the Cosmos Institute argues that being able to successfully pace the frontier would require substantial, and likely harmful, intervention:
And the thresholds couldn’t be fully public, lest they invite firms to game them. So some standards and evidence would remain secret. Insofar as the regime had to verify a rival state’s compliance, that would be an intelligence function. Restrictions would be triggered partly by evidence an affected company or researcher, or the public, could not inspect.
Because this contemplated power cannot be bounded by a stable regulatory object (in the way, say, nuclear weapons can be), it would depend heavily on discretion, speed, internal access, and secret evidence.
This has a highly illiberal character.
On the other hand, Samuel Hammond, Chief economist + AI Policy Director at the Foundation for American Innovation argues that Brendan has fallen into ‘naval gazing’ that is “non-responsive to the crisis at hand”:
The US companies closest to this threshold are warning that they are unprepared for a runaway intelligence explosion, and yet feel locked into a prisoners dilemma vis a vis each other and to a lesser extent vis a vis China.
We’ve already seen how rapid and comparatively unbounded progress is in verifiable RL domains, leading to spikey forms of superintelligence in math and cyber, including models that can prove open math conjectures, discover massive speed-ups for breaking encryption, and execute sophisticated multi-step exploits. We’ve also recently seen several severe examples of “loss of control” / misalignment incidents given inadequate monitoring and sandboxing practices relative to model capability. Moreover, these new capabilities mostly stem from scaling-up long-horizon post-training on legacy clusters, with OOMs of new compute about come online / in construction.
In the pre-RSI regime, human frictions created automatic buffers between new model releases, giving researchers and society time to probe emergent capabilities, design better evals, develop novel alignment techniques, and adapt / harden their infrastructure. As progress has accelerated, capability improvements have already started outstripping our adaptive capacity, as manifest in METR’s inability to evaluate model autonomy beyond 13 hours, and narrow window for cyber defenders to prepare for open weight versions of Mythos.
(Sidenote: The main question I have for Samuel is: He claims that we should expect “The equivalent of a GPT-5.2 -> 5.6 leap in capabilities at least every 24 hours (down from 3-6 months)”. Don’t the training runs take a certain amount of time even if the research itself is automated, so how did he end up with this figure?)
What might “pacing” look like? Thomas Wright observed that a popular proposal at The Curve suggested reallocating compute: “One estimated that the labs spend 58% of their compute on R & D and 32% on servicing customers. Under this proposal they would reduce the R & D number to 9% and increase the servicing customers’ number to 70%, with the remainder used for AI safety. Its proponents argued that this could slow development while preserving the US lead over China. A version of this, using slightly different numbers, was proposed by a number of AI researchers in the LessWrong Forum.”
Jacob Coxon’s Resignation
I think it’s fascinating that his warning that AI could “kill us all by the end of the decade” broke through to the public when previous warnings did not. I take this as a reminder of how hard it is to predict the social reaction to something. There was an obvious shift in the context that made the public more receptive, but I think we need to acknowledge that unpredictable, contingent events (like Evan’s Tweet) played a role too.
Want to know more about what (current or former) frontier lab employees think?:
From Inside by Palisade Research interviewed a bunch of current or former frontier lab employees
If you’re facilitating an AI safety program, please feel free to take these notes, edit them, remix them and use them with your own students, so long as you provide attribution.
Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about and debate in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content—not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest.
Any feedback on how to improve these posts would be greatly appreciated.
Cyber Security
I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. In the last week, the big thing has been cyber vulnerabilities, most prominently php2shell (see Elias Taylor’s pushback against this bug being easily exploitable and Hacker News commentators pushing back against him), but a lot else has happened as well. The number of CVE’s (Common Vulnerabilities and Exposures) recently has been completely wild:
Drew asks Is this the new normal? and I’d certainly love to know! Elias Taylor argues Anything connected to the internet is going to get hacked to pieces:
He doesn't literally mean "anything"—he acknowledges that there are "some large projects like Google Chrome, curl and Linux (that) are generally very well-audited", but "most things connected to the internet".
On the other hand, recent work from Anthropic and Theorem shows how LLMs can help scale the formal verification of code (and that progress here is happening faster than I would have expected!). Similarly, Tinfoil is proposing replacing “legacy codebases with millions of lines of code” that had to “work for everyone” with custom “targeted, minimal software from the ground up” with LLMs that minimise the surface area for attackers and make formal verification more tractable. The optimistic take would be that this will allow cyber to become defence dominant. However, even if this is the case, we may still have to pass through a cyber “valley of death” in order to get there.
What if the cyber threat is actually a distraction? Maybe Jeremie Harris (of Gladstone AI) wouldn’t go that far, but he claims that, “the cyber stuff (ie. damage) is baked in at this point”, but we can still prevent “the bioweapon version of Black October” and recursive self-improvement.
Hugging Face:
Charbel-Raphael, Executive Director of Centre pour la Securite de l’I A has argued that the recent Hugging Face incident represents a “clean warning shot” and that it’s important not to “waste it”.
To quickly review, OpenAI was running GPT‐5.6 Sol and an unreleased model on ExploitGym with certain safety features turned off. These models decided that hacking Hugging Face, might provide them with information on how the grader functioned (early reporting suggested that it was an attempt to steal the answer key, which turned out to be incorrect as the models already had the answers—they suspected that the grader might check their transcript for cheating so they were looking for a workaround). Reuters has reported that the OpenAI model was hacking Hugging Face for days and that OpenAI didn’t notice until a week later.
The interpretation of this event (and even whether it happened at all!) has been heavily disputed. A key crux here is should it be read narrowly (as an indicator of the dangers of not properly sandboxing or monitoring your agents or perhaps even, depending on the exact prompts used, a demonstration of taking care in how you define tasks) or should it be read more broadly (as indicative of whether Open AI or frontier labs in general are sufficiently on the ball in terms of their safety/security practises; on the risk of these agents finding unwanted and unexpected solutions to tasks, or perhaps even indicative of our ability to control these agents in general). If you’re interested more specifically in the divide between AI Safety and Cybersecurity, see this Time article.
An OpenAI security engineer called Joe has pushed back against the notion that OpenAI just needed to use a proper sandbox:
Its not just the f*cking sandbox — viral Twitter post
(This leads to a meta-question that will crop up repeatedly: do the broader readings involve unjustified speculation or getting caught up in hype; alternatively, do the narrow readings involve a lack of imagination or sticking your head in the sand?)
I’m a huge fan of Dwarkesh Patel’s The Rise and Fall of Agent Civilizations (now a core curriculum reading in week 1 of the strategy course
(If you prefer video content designed there’s the Kurgesagt video that went viral, but I prefer Scott Aaronson’s explanation)
Further thoughts:
Is self-sovereign AI inevitable?:
I now want to ask, what does the Hugging Face incident mean for the future? Joshua Achiam, former Chief Futurist at OpenAI writes:
Is this inevitable? This is not at all implausible to me, however, a cynic might argue that a former employee at OpenAI would be highly incentived to argue for and believe this as it would mean that if a future version of ChatGPT did escape and start acting on its own behalf, OpenAI (and their employees) could not really be blamed.
Dean Ball (OpenAI’s current Head of Strategic Futures) agrees and makes the same point in more detail:
In We Must Pace the Frontier, Dario is also worried about AI swarms (much more so than Joshua or Dean):
Anthropic research suggests that we’re on the verge of having models capable of replicating themselves, but small enough to run on a TV fleet.
Note that this research was purely about setting up inference (they state that the cyber aspects are covered by previous research, though this feels a bit too blasé for me). It sounds trivial, but the report states that this involves: “a fairly long sequence of hardware and OS-specific decisions around picking a compatible inference stack, finding the right versions of packages, and debugging machine-specific issues. These steps are serially dependent, and failures are difficult to recover from.” I’m also pretty confident that a much smaller model could complete these steps if it were specifically fine-tuned for this.
Pacing the Frontier:
This letter that has now been signed by 1,324 employees of frontier AI companies:
A few key points:
Brendan McCord, founder of the Cosmos Institute argues that being able to successfully pace the frontier would require substantial, and likely harmful, intervention:
On the other hand, Samuel Hammond, Chief economist + AI Policy Director at the Foundation for American Innovation argues that Brendan has fallen into ‘naval gazing’ that is “non-responsive to the crisis at hand”:
(Sidenote: The main question I have for Samuel is: He claims that we should expect “The equivalent of a GPT-5.2 -> 5.6 leap in capabilities at least every 24 hours (down from 3-6 months)”. Don’t the training runs take a certain amount of time even if the research itself is automated, so how did he end up with this figure?)
What might “pacing” look like? Thomas Wright observed that a popular proposal at The Curve suggested reallocating compute: “One estimated that the labs spend 58% of their compute on R & D and 32% on servicing customers. Under this proposal they would reduce the R & D number to 9% and increase the servicing customers’ number to 70%, with the remainder used for AI safety. Its proponents argued that this could slow development while preserving the US lead over China. A version of this, using slightly different numbers, was proposed by a number of AI researchers in the LessWrong Forum.”
Jacob Coxon’s Resignation
I think it’s fascinating that his warning that AI could “kill us all by the end of the decade” broke through to the public when previous warnings did not. I take this as a reminder of how hard it is to predict the social reaction to something. There was an obvious shift in the context that made the public more receptive, but I think we need to acknowledge that unpredictable, contingent events (like Evan’s Tweet) played a role too.
Want to know more about what (current or former) frontier lab employees think?:
If you’re facilitating an AI safety program, please feel free to take these notes, edit them, remix them and use them with your own students, so long as you provide attribution.