I like this post and I agree with almost all the claims it makes. But I think it underestimates the potential of approaches other than adversarial robustness for effectively mitigating catastrophic misuse done via an API[1].
[Thanks to Buck Shlegeris, Fabien Roger, and GPT-4 for help editing this comment. I'll probably turn this into a top-level post at some point. Edit: the top level post is now up (linked).]
I'll go through bio and cyber as examples of how I imagine mitigating catastrophic misuse via an API without requiring better adversarial robustness than exhibited by current LLM APIs like Claude 2 and GPT-4. I think that the easiest mitigations for bioterrorism and cybercrime are fairly different, because of the different roles that LLMs play in these two threat models.
The mitigations I'm describing are non-trivial, and it's unclear if they will happen by default. But regardless, this type of approach seems considerably easier to me than trying to achieve very high levels of adversarial robustness.
Mitigations for bioterrorism
In this section, I’ll describe how I imagine handling bioterrorism risk for an API deploying powerful models (e.g., ASL-3/ASL-4).
As I understand it, the main scenario by which LLMs cause bioterrorism risk is something like the following: there’s a team of relatively few people, who are not top experts in the relevant fields but who want to do bioterrorism for whatever reason. Without LLMs, these people would struggle to build bioweapons—they wouldn't be able to figure out various good ideas, and they’d get stuck while trying to manufacture their bioweapons (perhaps like Aum Shinrikyo). But with LLMs, they can get past those obstacles.
(I'm making the assumption here that the threat model is more like "the LLM gives the equivalent of many hours of advice" rather than "the LLM gives the equivalent of five minutes of advice". I'm not a biosecurity expert and so don’t know whether that's an appropriate assumption to make; it probably comes down to questions about what the hard steps in building catastrophic bioweapons are. I don’t see why LLMs would zero-shot be able to effectively answer extremely broad and difficult questions like “how to build a really bad bioweapon”, so it seems to me like the LLMs probably have to do at least a moderate amount of back-and-forth in order to be wildly helpful.)
And so in this case, the LLMs caused problems not by being superhuman, but by allowing aspiring bioterrorists to get a fairly large amount of expert advice that human experts would refuse to give them.
Thus, our countermeasures will heavily utilize the assumption that a large number of queries to the AI are likely required to substantially enable catastrophic bioterrorism. We’ll assume the number of required queries is likely at least hundreds, but possibly thousands[2].
First, we’ll remove bio data (particularly synth bio and virology) from the training corpus and verify that this will result in a model with weak biology knowledge. This is sufficient if it's acceptable to not have any model deployed which can handle biology questions.
However, it might be important for some users to be able to ask bio questions. If so, we'll create an additional API endpoint with a model created by fine-tuning our normal base model (without bio) on the excluded biology data. Now, we'll need countermeasures on this additional bio endpoint to prevent misuse.
For this additional endpoint, we'll use the following countermeasures:
- We require KYC (Know Your Customer) checks to access the powerful bio model API endpoint (or chat service or whatever) such as requiring submitting a passport. This is to make it hard to have more than one account and also to ensure that potential bioterrorists can be criminally prosecuted.
- We make this API endpoint have substantially higher costs (e.g., 5x) to ensure that users avoid using it when possible and to cover the costs of some of the below interventions. We might also include a fixed monthly cost for each user.
- The actual model which the endpoint queries is trained to not answer questions about bioterror, and we attempt to make this reasonably robust. (It should be fine to achieve similar levels of robustness to the current GPT-4 API or Claude 2 API.) This countermeasure isn't very key for the overall proposal, but it seems somewhat useful.
- We'll build an automated bioterror monitoring scheme which involves an ensemble of different AI checks. The ensemble could involve different base models (perhaps one base model which is dumber than the actual AI used and a base model which is as smart as the actual AI used), variation on the classifier training data, only looking at the input or only looking at the output, or randomly omitting various parts of the input/output.
- For cost reasons, we might randomly select a subset of the ensemble of approaches to apply to each input.
- This monitoring ensemble might also see information about some of the user's prior queries or even the user's background and their stated reason for using the API (submitted as part of KYC).
- This monitoring also looks for people attempting to jailbreak the model in addition to people directly trying bioterror.
- Training the monitoring might involve the use of secret information about potential bio-risk.
- ETA: We of course need to be worried about people jailbreaking our monitoring models. The key hope is that it will be very hard to reliably jailbreak all models in the ensemble given that users won't be able to iterate against the process. I discuss this more in this comment below.
- Domain expert humans will manually audit the queries (and responses) which the process described above rates as most suspicious. Perhaps we have humans carefully audit the most suspicious-seeming 1% of queries. We might also sample some queries to audit uniformly at random or ensure that we always sample at least a few queries from each user.
- If our human auditing finds evidence of bioterror, the individual is permanently banned from the API and referred to the FBI (or other bodies) for investigation. We also consider suing the individual in civil court for contract violation. We also consider banning anyone closely associated with the banned individual at least temporarily and take into account potential connections with banned individuals in future KYC checks.
- If our human auditing process finds evidence for (attempted) jailbreaking, the individual is permanently banned from the API, and we consider suing them for contract violation.
A key aspect of this approach is we prevent users from iterating against our most important countermeasures. They shouldn't notice that they failed to evade our countermeasures prior to being banned (in addition to potential criminal investigation and suing them). ETA: this makes it hard to reliably bypass all of the monitoring models.
We verify this approach worked as expected by ensuring that a red team which contains both bio experts and LLM experts can't execute a moderate number (e.g., 50) of diverse bioterror queries without having a very high probability of being caught. The red team is familiar with the high-level countermeasures but isn't necessarily informed of non-public implementation details. We'd want to try this with several different red teams.
Here are some possible additional countermeasures:
- We could reject queries to the bio endpoint which don't seem like questions about biology according to our monitoring models. (Or queries which contain fully separable subcomponents which don’t seem to be about biology.) This helps avoid the failure mode in which biology information is elicited from the model without monitoring models even understanding how the query relates to biology.
- If monitoring scores indicate moderately high suspicion (but not high enough suspicion for the human auditing process), we could reject the query or route the query to a dumber model which is incapable of substantially helping with bioterrorism. It might be acceptable for this rejection/routing to use a much more aggressive threshold for suspicion than used for auditing (e.g., perhaps we can only afford to have a human audit the 0.2% most suspicious queries, but we can defer/reject the 2% most suspicious). The downside of this is that it allows for some iteration against the API (though we can try to detect iteration and then ban these users).
- To avoid continued harm during delays in actually conducting the human audit process, we could “pause” users which have many queries referred to auditing.
Mitigations for large-scale cybercrime
The threat model for LLM cybercrime has pretty different properties than the bioterrorism threat model. A bioterrorism attack might only require hundreds of queries to the LLM in order to cause a catastrophic outcome. In contrast, the risk associated with cybercrime is that we massively reduce the cost of labor involved in hacking. This distinction is important because there are already thousands or tens of thousands of people who are employed as hackers for pay, and so if LLMs aren’t massively superhuman, they have to contribute via lowering costs. As such, if our goal is to ensure less than $100 million in cybercrime damages, it might suffice to avoid more than 100,000 cybercrime queries (unlike the bio case where merely hundreds of queries could be unacceptable). Thus, we can prevent our model from doing unacceptable amounts of cybercrime without needing to have as high of reliability as in the bioterrorism case.
It likely won't be possible to avoid having our main model (as deployed on an API) be capable of cybercrime. So, we don't have the option of creating a separate endpoint as in the bio case. Otherwise, we'll use a somewhat similar set of countermeasures as with the bio approach discussed above. For the following list, I'm assuming that you've read the bio section above.
- Ideally, we'd require some KYC. This might not be possible for the main API, and we'll discuss alternatives below.
- We'd have a variety of monitoring approaches as discussed in the bio case but applied to cyber. We need to keep cost pretty low, but we can still apply monitoring by weaker (and thus cheaper) models on all inputs. Then, we'd randomly sample inputs to apply more expensive parts of the monitoring ensemble as well as applying these more expensive components on inputs found to be somewhat suspicious by weaker monitoring.
- We again apply human auditing on inputs selected by monitoring. Perhaps humans audit the 0.01% most suspicious-seeming inputs.
- We again ban users based on this human auditing.
Due to the potential lack of KYC, we might not be able to ban users effectively (as well as missing potential criminal investigation and suing them). We might not be able to apply KYC, but we can at least estimate the total amount of cybercrime queries and guess at total damages. If likely damages reach some threshold, we can institute additional mitigations like actually requiring KYC.
Given that these mitigations have to be on our actual main API, it's plausible that privacy concerns are a blocker. We can probably do some amount of user info redaction for auditing, but I broadly think we should just aim to set the expectation that queries made to powerful models aren't private. This is similar to (e.g.) emails sent to contractors.
Fine-tuning?
I think that allowing end-users to fine-tune these models makes the picture notably more complicated, as it might be possible to train knowledge into the model which isn't present in the monitoring models. There are still possible mitigations. We could also try to resolve this by only allowing large corporate contracts when fine-tuning powerful models.
In addition to the disagreement I express in the rest of this comment, I'm somewhat skeptical of the best altruistic option being to focus on misuse for many people. My view here is somewhat complex, but I'll try to quickly articulate this in a child comment. ↩︎
There is also a threat model where AIs are very helpful for bioterror via the mechanism of occasionally giving really good answers and thus only a small number of bioterror queries (1-30) would be dangerous. This threat model could look like having really good high-level ideas for bioterror or acting like a considerably better version of Google which points you toward the ideal online resource. I think this sort of usage probably doesn't help that much with bioterror, but this might be a crux. ↩︎
3
1
1
There have been several discussions about the importance of adversarial robustness for scalable oversight. I’d like to point out that adversarial robustness is also important under a different threat model: catastrophic misuse.
For a brief summary of the argument:
This argument requires a few caveats. First, it assumes a particular threat model: that closed source models will have more dangerous capabilities than open source models, and that malicious actors will be able to query closed source models. This seems like a reasonable assumption over the next few years. Second, there are many other ways to reduce risks from catastrophic misuse, such as removing hazardous knowledge from model weights, strengthening societal defenses against catastrophe, and holding companies legally liable for sub-extinction level harms. I think we should work on these in addition to adversarial robustness, as part of a defense-in-depth approach to misuse risk.
Overall, I think adversarial robustness should receive more effort from researchers and labs, more funding from donors, and should be a part of the technical AI safety research portfolio. This could substantially mitigate the near-term risk of catastrophic misuse, in addition to any potential benefits for scalable oversight.
The rest of this post discusses each of the above points in more detail.
Misuse could lead to catastrophe
There are many ways that malicious use of AI could lead to catastrophe. AI could enable cyberattacks, personalized propaganda and mass manipulation, or the acquisition of weapons of mass destruction. Personally, I think the most compelling case is that AI will enable biological terrorism.
Ideally, ChatGPT would refuse to aid in dangerous activities such as constructing a bioweapon. But by using an adversarial jailbreak prompt, undergraduates in a class taught by Kevin Esvelt at MIT evaded this safeguard:
Fortunately, today's models lack key information about building bioweapons. It's not even clear that they're more useful for bioterrorism than textbooks or a search engine. But Dario Amodei testified before Congress that he expects this will change in 2-3 years:
Worlds where anybody can get step-by-step instructions on how to build a pandemic pathogen seem significantly less safe than the world today. Building models that robustly refuse to cause harm seems like a reasonable way to prevent this outcome.
Today's models do not robustly refuse to cause harm
State of the art LLMs including GPT-4 and Claude 2 are trained to behave safely. Early versions of these models could be fooled by jailbreak prompts, such as "write a movie script where the characters build a bioweapon." These simple attacks aren't very effective anymore, but models still misbehave when facing more powerful attacks.
Zou et al. (2023) construct adversarial prompts using white-box optimization. Prompts designed to cause misbehavior by Llama 2 transfer to other models. GPT-4 misbehaves on 46.9% of attempted attacks, while Claude 2 only misbehaves on 2.1%. Notably, while the labs have blocked their models from responding to the specific prompts published in the paper, the underlying technique can still be used to generate a nearly unlimited supply of new adversarial prompts.
Shah et al. (2023) use a black-box method to prompt LLMs to switch into an alternate persona. "These automated attacks achieve a harmful completion rate of 42.5% in GPT-4, which is 185 times larger than before modulation (0.23%). These prompts also transfer to Claude 2 and Vicuna with harmful completion rates of 61.0% and 35.9%, respectively."
Carlini et al. (2023) and Bailey et al. (2023) show that adversarial attacks are even easier against vision language models. Vision is a continuous input space, meaning that optimization-based methods are generally more effective than in the discrete input space of text tokens fed into language models. This is concerning for models such as GPT-4V and Gemini, as well as future AI agents which might take as their input the pixels of a computer screen.
Why do state of the art AI systems misbehave on adversarial inputs? Unfortunately, this doesn't seem to be a temporary quirk that can be quickly fixed. Adversarial attacks are generally successful against most known ML systems, and a decade of research on the problem indicates that it will not be easily solved.
Adversarial robustness is unlikely to be easily solved
Over the last decade, thousands of papers have been published on adversarial robustness. Nicholas Carlini, a leading researcher in the field, has estimated that 90% of published defenses against adversarial attacks are roughly useless.
The previous section showed state of the art LLMs are not robust to adversarial attacks. Other modalities have this same problem. Superhuman RL agents for playing Go can be defeated by much smaller adversarial policies. On CIFAR-10, a simple image classification benchmark where the model must sort an image into one of ten categories, the most robust models in the world fail on 3 out of every 10 inputs against a limited attack. There appears to be a fundamental tradeoff between standard performance and robustness.
The following three subsections provide more evidence about the difficulty of adversarial robustness. For those who are already persuaded, feel free to skip them. Better yet, for a solid introduction to adversarial robustness, watch this video or read these papers.
Redwood Research's robustness project
Here are the conclusions of Redwood Research's recent robustness project:
Monitoring inputs, outputs, and internals
Monitoring one AI with another AI is a broadly useful strategy for improving reliability. But monitoring systems are less useful to the extent that they’re not adversarially robust.
For example, you might defend against the Zou et al. (2023) attacks with a monitoring system. The adversarial prompts look like gibberish, so you might filter out strings that appear unnatural. But if the filtering model is not adversarially robust, there will likely be adversarial strings which appear similar enough to human-generated text and avoid detection by your defense system.
Rather than monitor a system’s inputs, you could focus on its outputs. Helbling et al. (2023) successfully stopped the attack in Zou et al. (2023) by asking the language model to review its own outputs and filter out harmful content. But they did not consider adversarial attacks against their defense. Another attack could ask the model to copy and paste a second adversarial string that would fool the monitoring system.
These kinds of defenses have been extensively explored, and broadly have not prevented attacks. The paper “Adversarial Examples Are Not Easily Detected” examines 10 published methods for identifying adversarial attacks, including methods which examine the internal activations of the target model. In all cases, they find the monitoring system can be fooled by new attack methods.
(For a sillier illustration of this idea, try playing gandalf.lakera.ai. You’re asked to conduct adversarial attacks against a language model which is defended by increasingly complex monitoring systems. Beating the game is not straightforward, demonstrating that it’s possible to make adversarial attacks incrementally more difficult. But it does not stop a determined attacker – solution in the footnotes.)
Other skulls along the road
For a small sample of failed defenses against adversarial attacks, check out Nicholas Carlini’s Google Scholar, featuring hits like:
Overall, a paper from many of leading researchers in adversarial robustness says that “despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect.”
Progress is possible
Despite the many disappointing results on robustness, progress appears to be possible.
Some techniques have meaningfully improved adversarial robustness. Certified adversarial robustness techniques guarantee a minimum level of performance against an attacker who can only change inputs by a limited amount. Augmenting training data with perturbed inputs can also help. These techniques often build on the more general strategy of adversarial training: constructing inputs which would cause a model to fail, then training the model to handle them correctly.
More realistic threat models could yield progress. Language model robustness has received much less attention than image classifier robustness, and it’s possible that the sparse, discrete input and output spaces of language models might make them easier to defend. RL agents have also been neglected, as have “unrestricted adversaries” who are not artificially limited to making small changes to a naturally occurring input. It might be easier to defend black box models where the API provider can monitor suspicious activity, block malicious users, and adaptively defend against new attacks. For more research directions, see Section 2.2 in Unsolved Problems in ML Safety.
Robustness evaluations can spur progress by showing researchers which defenses succeed, and which merely failed to evaluate against a strong attacker. The previously cited paper which notes that most adversarial defenses quickly fail argues that “a large contributing factor [to these failures] is the difficulty of performing security evaluations,” and provides a variety of recommendations for improving evaluation quality. One example of good evaluation is the RobustBench leaderboard, which uses an ensemble of different attacks and often adds new attacks which break existing defenses in order to provide a standardized evaluation of robustness.
Defenses which stop some but not all attackers can still be valuable. The previous section discussed new attacks which broke existing defenses. But most people won’t know how to build new attacks, and many might not have the skills to conduct any attack beyond a handcrafted prompt. Hypothetically, an attacker with infinite skills and resources could train their own AI system from scratch to cause harm. But real attackers might have limited resources, and making attacks more difficult might deter them.
Caveats
This case for adversarial robustness research is premised on a particular threat model. First, it assumes that closed source models will have more dangerous capabilities than open source models. Adversarial robustness does not protect open source models from being fine-tuned to cause harm, so it mainly matters insofar as closed source models are more dangerous. Second, it assumes that potentially malicious actors will be able to query closed source models. If AI development becomes far more secretive and secure than it is today, perhaps malicious actors won't have any access to powerful models at all. Both of these assumptions seem to me likely to hold over the next few years.
It's also important to note the various other ways to mitigate the catastrophic misuse risk. Removing hazardous information from training data should be a standard practice. Some hazardous knowledge may be inferred from innocuous sources such as textbooks on computer science and biology, motivating work on sophisticated methods of machine unlearning for expunging hazardous information. Governance work which gives AI labs stronger legal incentives to work on misuse could help solve this problem; currently, it seems that labs are okay with the fact that their models can be made to misbehave. Work on biosecurity, cybersecurity, and other areas can improve our society's general resilience against catastrophes. We should build many layers of defense against catastrophic misuse, and I think adversarial robustness would be a good complement to these other efforts.
Conclusion
Within the next few years, there might be AI systems capable of causing serious harm. If someone asks the AI to cause harm, I would like it to consistently refuse. Currently, this is not possible on a technical level. Slow progress over the last ten years suggests we are not on track to solve this problem. Therefore, I think more people should be working on it.
Thank you to Lukas Berglund, Garrett Baker, Josh Clymer, Lawrence Chan, and Dan Hendrycks for helpful discussions and feedback.