Context: Simeon Campos and I wanted to understand how other industries handle catastrophic risks, then write pieces about how any standards from them could be applied to AI for policy readers. This is a draft slightly modified for LessWrong. 

Epistemic status: Very shallow dive, though I checked my understanding with people with more biosecurity and policy expertise, like David Manheim (thanks!).

Related posts I recommend: Six Dimensions of Operational Adequacy in AGI ProjectsHigh Reliability Orgs, and AI Companies“Carefully-Bootstrapped Alignment” is organizationally hard

 

Tl;dr: 

Argument: We do a lot to mitigate pandemic risk from dangerous pathogen research, requiring pathogens like Ebola only be studied in BSL-4 labs subject to extensive standards. Given that experts think extinction risk from AI should be a global priority alongside pandemics, we should hold high-risk AI projects to similar if not higher standards. 
Example standard of BSL-4 labsExample application to high-risk AI projects
Research needs project-specific approval and can only happen in designated BSL-4 labs, set up specifically to follow stringent standards. Governments require high-risk AI projects to only take place in licensed labs, with project-specific approval.  
Personnel needs to go through FBI and other security risk assessments and extensive safety training before they can access the lab. These check things like criminal record, mental stability, susceptibility to coercion, knowledge of safety practices, etc. Before being able to participate in high-risk AI projects, personnel must go through reliability screening and training. Labs cultivate a culture of prioritizing safety. 
Labs record every incident and plan for emergencies.Labs are required to record all incidents that suggest heightened research risk (eg. AI developing unexpected capabilities). Labs design mechanisms for shutting down unsafe projects quickly. 
Some officer or body is responsible for maintaining safety and reporting accidents, and they’re empowered to shut down research projects they determine to be unsafe.Labs have safety officers accountable to external oversight bodies to whom they report incidents. 
Physical and information security are prioritized.Labs invest more in securing infrastructure eg. via air gapping, reducing visitor access, reducing publishing, etc. 
Labs have extensive oversight from governments and independent auditors.Auditors check the reliability of labs’ facilities, personnel, plans, and AI models, including via dangerous capability evaluations.

.


How biosafety could inform AI standards

Leading AI executives and researchers recently signed a statement saying “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” 

If we take this seriously, it’s worth looking at how much we do to mitigate pandemic and nuclear risks. (Spoiler: a lot.) 

This post addresses the first half of that, outlining my understanding of biosafety standards and how they could inform standards for high-risk AI research.  

 

Biosafety & its relevance

The field of biology has comprehensive standards for maintaining safety while working with potentially dangerous biological agents, such as viruses that could escape the lab and infect many people. 

Biosafety standards are applied in a tiered approach, where the riskiest research can only occur in laboratories taking the most stringent precautions. The levels of laboratory precautions range from biosafety level 1 (BSL-1, the lowest) to biosafety level 4 (BSL-4, the highest). BSL standards also govern the storage and transportation of biological agents, access to these labs, and similar. 

BSL-1 has minimal standards; it allows places like high school labs to easily experiment. BSL-4 has very restrictive standards; there are only around 50 BSL-4 labs in the world. BSL-4 labs are the only places where easily-transmissible and lethal diseases like Ebola can be studied. 

By enforcing tiered standards according to research risk at all labs, biology has been relatively successful at mitigating the two types of risks it faces: 

  • accident risks, such as a virus escaping the lab because it was accidentally mishandled, and 
  • misuse risk, such as a virus being intentionally taken out of the lab by someone who intends to cause harm with it.

There are notable parallels to AI here. Different AI research poses different risks, and most are easily manageable. However, research on the most powerful systems poses significant accident and misuse risks that are difficult to manage (in some cases, we don’t even know how we could manage them): powerful systems could get out of developers' control or they could be used to cause harm. Accordingly, labs conducting that research should probably be held to particularly high standards of reliability and security. 

Below I outline some standards for US biosafety level 3 and 4 labs, which handle pathogens that pose catastrophic risks. Other countries' standards at these levels are very similar, so I haven’t taken the time to spell out differences. Then, I outline how they could be applied in AI. I’m not going to defend these particular applications here; they’re just examples. That said, I've chosen examples I’m pretty sympathetic to.

 

Selected biosafety standards & potential applications in AI 

1. High-risk research must be conducted in designated labs subject to stringent standards.  

The USDA maintains a list of biological agents that pose severe public health and safety threats (henceforth select agents). Any lab wishing to work with a select agent must be licensed by the Federal Select Agent Program (FSAP). These facilities are continuously monitored for compliance. 

Additionally, each BSL-4 research project must get specific approval before it can be conducted. Researchers must submit detailed research plans and undergo a thorough review by their institutional committee and sometimes by government agencies, overseen by a government-approved official. The review process assesses the potential risks associated with the select agent being researched, as well as the suitability of the researcher and the lab. 

Research that is less risky is subject to correspondingly less severe restrictions, which range from BSL-1 laboratories in grade schools, homes, and some high schools, to high-school and hobbyist BSL-2 biology laboratories with minimal oversight, to BSL-2 or -3 clinical or research laboratories with full-time biosafety officers and institutional biosafety reviews.

 

Potential application in AI: 

A government agency would determine various risk levels for AI research, and set out standards for each level. For high-risk research, eg. research on large, general purpose AI systems, research would only be allowed to be conducted in designated labs, using designated compute providers. Governments would provide licenses to designated labs and compute providers once they have demonstrated compliance with the most restrictive security and research standards, as well as continuously monitor their compliance. For especially large training runs expected to pose the greatest risk, labs would additionally be required to apply for project-specific approval from the government. Governments would block research projects too risky for any lab to carry out, eg. scaling AI systems significantly beyond current levels, until sufficient precautions are developed. It would be illegal to conduct research outside licensed facilities or without project-specific approval. 

 

2. Personnel must be trained and screened for reliability. 

All personnel with access to listed select agents must successfully undergo FBI Security Risk Assessments and be approved by either the CDC or the USDA’s select agent program. These resemble or can include DOD security clearances, Department of Energy reliability checks, or Office of Personnel Management “Public Trust” investigations. 

Security risk assessments check for risk factors for intentional subversion of rules or irresponsible behaviors. This can include whether a candidate has been involved in certain crimes, has used or possessed certain drugs, is affiliated to potentially-hostile nations, has been shown themselves to be mentally unstable; or has been discharged from the Armed Services of the United States under dishonorable conditions.

Personnel assessments check for “emotional stability, capacity for communication and cooperation, integrity, capacity to resist external pressure, acceptance and capacity to follow instructions, active approach to safety and security, mental alertness, mental and emotional stability, trustworthiness, freedom from unstable medical conditions, dependability in accepting responsibilities, effective performance, flexibility in adjusting to changes, good social adjustment, ability to exercise sound judgment in meeting adverse or emergency situations, freedom from drug/alcohol abuse or dependence, compliance with requirements, positive attitude toward PRP” as well as “whether the candidate can be “blackmailed, coerced, or otherwise manipulated.”

Some assessments are ongoing. Personnel need to self-report “medical matters... prescription and over-the-counter medications used, alcohol abuse, legal actions, public record court actions (eg, separation, divorce, lawsuit), financial problems or concerns, or any other activity that may influence the staff member’s day-to-day reliability”. Peers and supervisors also need to report “any behaviors or events they suspect are affecting an individual’s day-to-day reliability”

There are extensive requirements of credentials and training for the use of BSL-3 and -4 laboratories. It if effectively impossible to work at such a lab before having thoroughly studied safety, security, and accident procedures.

 

Potential application in AI research: 

In order to work on a high-risk AI research project, candidates would be required to go through training and screening to ensure they deeply understand and are committed to safety. Training might cover risks from misalignment and misuse, the limits of current safety and security techniques, and the limits of current understanding of AI systems. Screening might check for a candidate's understanding of known risks and limitations, ability to identify new risks, psychological stability, integrity, commitment to the common good, freedom from conflicts of interest and other potential burdens on decision making, and ability to make good decisions under pressure.  

 

3. Labs record and respond to every incident and plan for emergencies. 

Labs must perform an initial risk assessment before any work is started. Risk assessments must also be reviewed whenever a change occurs in factors that may significantly impact risk such as personnel changes, relocations or renovations, aging of equipment, new biological agents, new animal species, and new routes models of administration.

Risk assessments must also be performed after every incident, even if it was a near-miss. The norm is for labs to make concrete changes that would robustly prevent similar accidents in the future each time this occurs. Most accidents are non-punishable, incentivizing open reporting. This follows best practice for all high reliability organizations, such as nuclear power, aviation, and similar.

Labs also prepare thoroughly for emergencies, such as an earthquake compromising physical security. They generally maintain emergency egress routes and backup generators, design buildings and barriers to withstand impact, and have procedures in case of suspected release of a select agent. 

 

Potential application in AI research: 

Every lab and data center involved in high-risk AI would be required to monitor for, report, and address any incidents that suggest heightened research risk. For instance, they would be required to report whenever an AI model develops unexpected or emergent capabilities; shows tendencies towards deception, power-seeking, avoiding shutdown; shows signs of situational awareness, agency, or autonomy; or outputs harmful content, such as text that offends users or information that would assist people in carrying out acts of violence. They would also be required to develop mechanisms in case of security emergencies, such as mechanisms to rapidly shut down compute clusters or destroy weights. 

 

4. Someone at each lab is responsible for safety, and they are empowered to shut projects down if they determine them to be unsafe. 

A biosafety officer is required for managing biosafety at laboratories. Their duties include, but are not limited to, “developing the safety policy and procedures…; orientation and training of all laboratory staff in biosafety; providing safety advice; ensuring staff compliance with safety policies and procedures by performing regular safety inspections, and documenting and submitting reports of such inspections to the Laboratory Manager for review and action; [and] investigating laboratory accidents and documenting accident reports.” 

The biosafety officer requirement is universal across different safety levels, but the work involved differs depending on the laboratory and safety requirements. BSL-4 laboratories will frequently have a group or departments rather than a single individual in charge of this function. BSL-3 and BSL-4 labs handling select agents are also required to have a Responsible Official, who is not only in charge of ensuring compliance, but is legally required to be on-site, has authority to make decisions on the part of the organization, and ensures reporting is performed - including by ensuring that whistleblowers have a secure way to report issues directly to the relevant federal inspector general.

 

Potential application in AI research: 

Every lab and data center involved in high-risk AI research would be required to have an officer or body responsible for ensuring safety. Safety officers would be accountable to the national or international agency or external oversight body. To that body, safety officers be required to report all incidents (eg. “the model demonstrated deceptive and power-seeking tendencies” or “a staff member tried to download model weights onto their personal laptop”) and anticipated risks (eg. “the proposed fine-tuning seems too risky given the limitations of current safety techniques” or “our security system may soon be outdated,”). In addition to reporting, they would be empowered to act on behalf of the organization to stop unsafe projects until these issues are addressed. 

 

5. Physical and information security are prioritized. 

BSL-4 levels take extensive physical security precautions to minimize the risk of select agents infecting people and leaving the lab, both by accident and misuse. 

For perimeter security, labs generally have security guards, metal detectors, fences and barriers that can withstand impact, limited entries/exits, mantraps, and biometric readers and keycards. To control the working environment, labs generally airlock workspaces, ensure airflow in one direction, and sanitize water and air before it exits the facility. To protect staff, labs generally require researchers to wear positive pressure personal suits inside the lab, take a chemical and regular shower before exiting the lab, and avoid using the restroom until outside the lab. While working in the lab, researchers are always watched; scientists outside communicate via radio, operate microscopes remotely, etc. Throughout the lab, sensors are set up to detect select agents in the air that may have been accidentally released.

Labs also have strict access controls. The number of people given access to high-risk areas and information is kept to the minimum needed. Accordingly, silo-ing within research institutes is standard. For instance, Boston University's National Emerging Infectious Diseases Laboratories (the NEIDL) contains BSL-2, BSL-3, and BSL-4 containment spaces. Only a small, designated set of their researchers are allowed to access the BSL-4 space.  Even outside the BSL-4 lab, visitors are kept to a minimum and may not be unattended.

There are also strict rules for how information can be accessed and shared, to minimize the risk of research insights being used by outside actors to cause harm. 

 

Potential application in AI research: 

Labs would invest drastically more in securing their AI model (eg. to prevent weights from being leaked or stolen or to prevent AIs from getting unintended access to the internet) and secure research insights (to reduce the spread of information that enables others to build high-risk systems). Unlimited access that allows for model-parroting and models-extraction attacks would need to be restricted.

Computers and networks involved in high-risk research would be air-gapped (as is already standard for some military research and recommended or required for high-security applications like critical infrastructure or cryptocurrency storage by depository institutions). Mechanisms for detecting anomalies in AI models during training and deployment would need to be improved, and continually revised and updated over time. Finally, access to AI models and labs would be limited, and oversight would be increased throughout the research process. 

 

6. Labs have extensive oversight from governments and independent auditors. 

Regular audits are conducted by teams of experts from the CDC, other government agencies, and external consultants and inspectors, who are contracted by the CDC or hired directly by the lab. Auditors thoroughly review a lab's procedures, inspect physical infrastructure and equipment, observe the handling of select agents, and interview personnel. At the conclusion of the audit, they provide detailed reports outlining their findings and recommendations. The lab is generally required to address any concerns identified in the report.

 

Potential application in AI research: 

AI labs would be overseen by external bodies that evaluate the reliability of their facilities, personnel, research plans, and AI models. Questions evaluators may ask include: do the AI models have dangerous capabilities, how robustly do safety techniques being used mitigate risks, are the researchers trustworthy, and are the facilities secure? Government bodies, internal evaluators, and internal governance would all be involved in these assessments.

 

Concluding thoughts 

Frontier AI research mostly happens in private companies like OpenAI, Google DeepMind, and Anthropic. Though they invest in safety, these places are far from BSL-4 labs. If their own leaders believe AI poses risks comparable to or worse than pandemics, they have a long way to go.

I haven’t researched enough to confidently comment on biosafety standards’ effectiveness, but I will briefly remind COVID-19 may have leaked from a BSL-4 lab. 

Also, in the introduction, I suggested there are four tiers of biosafety standards: BSL-1 to BSL-4. Actually, in practice, there is a tier beyond BSL-4: restricted experiments. Restricted experiments are those considered too dangerous to conduct even in BSL-4 settings, and they are effectively banned. I expect there is some threshold beyond which research is unmanageably risky in AI, too (eg. scaling far beyond current levels). I think governments should be coordinating to make sure they can prevent research beyond that threshold (eg. via a moratorium). 

I raise these points because I expect AI needs standards like biosafety’s, I expect AI companies need to be more like BSL-4 labs, and I expect we will need to go even further. BSL-4 isn’t foolproof, and AI may pose risks worse than anything they study. We have a long way to go. 

21

New Comment
3 comments, sorted by Click to highlight new comments since: Today at 1:15 PM

There’s lots of interesting stuff I didn’t get into here. I nudge others to follow-up on: 

  • the history of how biosafety standards arose (tldr: military started because of biowarfare; academia integrated upon)
  • why labs uphold biosafety standards, even though there’s not that much enforcement and oversight detail written into law (tldr: researchers learn a culture of safety in academia, and there’s stronger personal incentives to be safe since you might get infected)  
  • how many labs, government agencies, and independent contractors interacting in a complex systems somehow functions (don’t know) 
  • how biosafety standards fail (don’t know) 
  • how effective each standard is, and why it works (don’t have a tldr answer)  
     

I generally don't find writeups of standards useful, but this piece was an exception. Below, I'll try to articulate why:

I think AI governance pieces-- especially pieces about standards-- often have overly vague language. People say things like "risk management practices" or "third-party audits", phrases that are generally umbrella terms that lack specificity. These sometimes serve as applause lights (whether the author intended this or not): who could really disagree with the idea of risk management?

I liked that this piece (fairly unapologetically) advocates for specific things that labs should be doing. As an added benefit, the piece causes the reader to realize "oh wow, there's a lot of stuff that our civilization already does to mitigate this other threat-- if we were actually prioritizing AI x-risk as seriously as pandemics, there's a bunch of stuff we'd be doing differently." 

Naturally, there are some areas that lack detail (e.g., how do biolabs do risk assessments and how would AI labs do risk assessments?), but the reader is at least left with some references that allow them to learn more. I think this moves us to an acceptable level of concreteness, especially for a shallow investigation.

I think this is also the kind of thing that I could actually see myself handing to a policymaker or staffer (in reality, since they have no time, I would probably show them a one-pager or two-pager version of this, with this longer version linked).

I'll likely send this to junior AI governance folks as a solid example of a shallow investigation that could be helpful. In terms of constructive feedback, I think the piece could've had a TLDR section (or table) that directly lists each of the recommendations for AI labs and each of the analogs in biosafety. [I might work on such a table and share it here if I produce it]. 

Thanks. Good suggestion, added table