In May and June of 2023, I (Orpheus) had about 50-70 meetings about AI risks with congressional staffers. I had been meaning to write a post reflecting on the experience and some of my takeaways, and I figured it could be a good topic for a LessWrong dialogue. I saw that hath had offered to do LW dialogues with folks, and I reached out.

In this dialogue, we discuss how I decided to chat with staffers, my initial observations in DC, some context about how Congressional offices work, what my meetings looked like, lessons I learned, and some miscellaneous takes about my experience.

Context

hath

Hey! In your message, you mentioned a few topics that relate to your time in DC.

I figured we should start with your experience talking to congressional offices about AI risk. I'm quite interested in learning more; there don't seem to be many public resources on what that kind of outreach looks like.

How'd that start? What made you want to do that?

Orpheus16

In March of 2023, I started working on some AI governance projects at the Center for AI Safety. One of my projects involved helping CAIS respond to a Request for Comments about AI Accountability that was released by the NTIA.

As part of that work, I started thinking a lot about policy ideas for frontier AI systems. For instance: if I could set up a body to oversee frontier AI systems or assign certain tasks to existing agencies, what would that like? Where in the US government would it be housed? What information would I want it to assess?

I began to wonder how actual policymakers would react to these ideas. I was also curious to know more about how policymakers were thinking about AI extinction risks and catastrophic risks.

I started asking other folks in AI Governance. The vast majority had not talked to congressional staffers (at all). A few had experience talking to staffers but had not talked to them about AI risk. A lot of people told me that they thought engagement with policymakers was really important but very neglected. And of course, there are downside risks, so you don't want someone doing it poorly.

After consulting something like 10-20 AI governance folks, I asked CAIS if I could go to DC and start talking to congressional offices. The goals were to (a) raise awareness about AI risks, (b) get a better sense of how congressional offices were thinking about AI risks, (c) get a better sense of what kinds of AI-related priorities people at congressional offices had, and (d) get feedback on my NTIA request for comment ideas.

CAIS approved, and I went to DC in May-June 2023. And just to be clear, this wasn't something CAIS told me to do– this was more of an "Akash thing" that CAIS was aware was happening.

hath

Whoa, that's really interesting. A couple random questions:

And of course, there are downside risks, so you don't want someone doing it poorly.

How does one go about doing it non-poorly? How does one learn to interact with policymakers?

Also, what's your background? Did you do policy stuff before this?

Orpheus16

Yeah, great question. I'm not sure what the best way to learn is, but here are some things I tried:

Talk to people who have experience interacting with policymakers. Ask them what they say, what they found surprising, what mistakes they made, what downside risks they've noticed, etc.
Read books. I found Master of the Senate and Act of Congress to be especially helpful. I'm currently reading The Devil's Chessboard to better understand the CIA & intelligence agencies, and I'm finding it informative so far.
Do roleplays with policymakers you already know and ask them for blunt feedback.
Get practice in lower-stakes meetings, and use those experiences to iterate.

I hadn't done much policy stuff before this. In college, I wrote for the Harvard Political Review and was involved in the Institute of Politics, but that's a lot more academic than the "real-world policy engagement" stuff.

Arriving in DC & initial observations

hath

That all makes sense. What did you do once you arrived in DC?

Orpheus16

I cold-emailed congressional offices, as well as some executive branch folks. I also continued working on the NTIA Request for Comment (which was due June 6).

The initial plan was to have a few meetings, assess how they were going, and then have a bunch more if I thought they were going reasonably well.

In total, I ended up having about 50-70 meetings with congressional staffers (as well as some with folks at think tanks and folks in executive branch agencies, but I'll focus this post on the meetings with congressional staffers).

hath

I take it they went reasonably well, then?

Orpheus16

According to me, yes! One thing I'll note is that these can be somewhat hard to assess– like, staffers are supposed to be kind to people, and they're not going to say things like "I thought you were an idiot" or "you wasted my time" or "I now have a worse impression of AI safety."

With that caveat in mind:

I was surprised at how open-minded staffers seemed. The Overton Window has shifted so much recently, but at the time, I didn't really know if people would be like "ha! Extinction risks? That sounds like sci-fi."
The dominant vibe was "AI is really important, and I'm a busy staffer with 100 priorities, so I haven't had time to learn about it. I'm really really excited to talk to someone who can tell me stuff about AI– I've been desperate to get up to speed on stuff."
Staffers expressed a lot of gratitude for the opportunity to meet someone who was willing to answer basic questions about AI (e.g., what is a large language model and how is it different than other kinds of AI? How many companies do frontier AI?)
There were some "tangible" signals that things were going well. For instance, some staffers introduced me to other people they knew, some people sent me work that their offices were drafting, and a few people even introduced me to congresspeople (two in total).

Hierarchy of a Congressional office

hath

That's really interesting!

As a brief aside, could you paint me a picture of what a Congressional office looks like in terms of staffing hierarchy? Like, who did you typically speak to, and what relationship did they typically have to the congressperson?

Orpheus16

Great question! So, my understanding is that a Congressional office typically has the following roles, listed from "most influential" to "least influential":

Chief of Staff
Legislative Director
Legislative Assistant
Legislative Correspondent
Interns and fellows

There are some other roles too, but these tend to matter most from a legislative perspective.

Note that each office has its own vibe. Someone once told me "each Congressional office is its own start-up, and each Congressperson gets to run their office however they want."

So in some offices, interns and fellows might actually have a lot of influence (e.g., if a Congressperson or a Legislative Director trusts the intern to be the subject matter expert on a specific topic). But in general, I think this hierarchy is pretty common.

I think I mostly spoke with people at the legislative assistant/legislative correspondent level. I also spoke with a few legislative directors.

Outreach to offices

hath

Okay, that all makes sense. So, how did you end up going from a few meetings to 60-80?

Orpheus16

I sent a mass email to tech policy staffers, and I was pretty impressed by the number who responded. The email was fairly short, mentioned that I was at CAIS, had 1-2 bullets about what CAIS does, and had a bullet point about the fact that I was working on an NTIA request for comment.

I think it was/is genuinely the case that Congressional staffers are extremely interested in AI content right now. Like, I don't think I would've been able to have this many meetings if I was emailing people about other issues.

There was some sense that "AI is very hot right now, but no one really knows much about AI". I think it's unclear how long that will last (especially the "people don't know much & offices haven't made up their minds") part.

I would go as far as to say something like "I think this was an opportunity that the AIS community as a whole could've/should've taken advantage of more." Like, Congressional staffers were (and I think are still) extremely interested in engaging with people about AI– it's hard to imagine a better opportunity for folks in the AIS community to be able to come in and serve as advisors/advocates.

hath

That makes sense.

How might one be able to start engaging with Congressional staffers? What should one do to get into that space/what orgs might be well-positioned to deploy people for this?

Orpheus16

This is going to be a pretty vague answer, but I think it depends a lot on the person, their skills, and their policy objectives.

Also– and I mentioned this above but it's important to reiterate– there are definitely risks from people doing this kind of work poorly. On the other hand, there's a risk of being too "inaction-biased", or something like that, and leaving a lot of value on the table.

This is genuinely hard and confusing. I mentioned earlier that I consulted 10-20 AI governance people. Most of them were like "this seems important and neglected, but idk man, seems confusing." Several of them were like "yeah, I totally think you should do this, especially if you employ XYZ tactics." One fairly prominent AI governance person told me explicitly that they did not want me to do this. I found it hard to balance this conflicting feedback.

I also think a lot of my advice would depend on what exactly someone wants to say– like, for example:

What is their pitch going to be? If a meeting starts and the staffer says "so, what did you want to talk about?", what's the initial response?
Are they the kind of person that is good at asking questions and being curious about other people's worldviews?
Are they going to sound alarmist?
Do they know a lot of facts about AI? When they don't know something, are they going to be able to recognize that and hedge appropriately?

With all that in mind, if someone reading this was interested in engaging with Congressional staffers (or having someone in their org do this), and they valued my opinions, I recommend they reach out to me through LW. I'd be able to provide better advice with more context.

A typical meeting

hath

Yeah, that all makes sense. Could you walk me through a typical meeting you might have had? Like, how would you first get in contact with a staffer, where would you meet them, what did the actual conversation look like, how would you follow up or otherwise figure out whether it was helpful?

Orpheus16

Meeting Logistics

Would get in contact via email
Would usually meet them at the congressional office (there are basically 4 main buildings in DC that have all the congressional offices) or via Zoom

How meetings went

The conversation would typically start by me asking if they had any questions about AI or wanted me to share stuff I was working on. Usually, they wanted me to start.
I would start by introducing myself and CAIS. Once the CAIS statement came out, I would reference the CAIS statement. I'd tell them that I was focusing on global security risks from advanced AI. I'd also tell them that I was working on an NTIA response, and I'd tell them about some of the high-level ideas I was considering.
Then, I'd pause to see if they had any questions.
Often, they'd either ask more about extinction risks, or they'd ask miscellaneous questions about AI (e.g., do you have any thoughts on how to handle deepfakes?), or they'd bring up some high-level questions about regulation (e.g., how do we regulate without stifling innovation? How do we regulate without losing to China?)
In some of the best meetings, I'd hear about some of the AI-related stuff that the office is working on. Most offices don't have capacity/interest to take the lead on AI stuff. About 10% of offices were like "yea my congressperson is super interested in this and we're thinking about introducing legislation or being a core part of someone else's legislation."
Lots of people asked me if I had draft legislation. Apparently, if you have regulatory ideas, people want to see that you have a (short) version of it written up like a bill.

Following-up

I sent a follow-up to everyone I met with once the NTIA request for comment response was finished. When I had an especially good meeting (e.g., a staffer expressed strong interest in AI risks or told me that they wanted to send me something they were working on), I would send a personalized follow-up. I think the most tangible sign of helpfulness came from instances when people continued to send me questions/thoughts, introduced me to colleagues, or wanted to work with me on proposals. (To be clear, this happened in a minority of cases, but I think it's where the majority of the impact came from).

Staffer attitudes toward AI risk

hath

Lots of people asked me if I had draft legislation.

What kind of issues were they looking for legislation on? Any bits of legislation that you suggested?

Orpheus16

Staffers often wanted to know if I had draft legislation to describe the licensing regime that I was writing about in my NTIA response.

hath

Ah, okay. More generally, what kinds of priors did people have on AI risk? Do you think you typically caused a significant change in how they approached the topic?

Orpheus16

It seemed like most people didn't have strong priors about AI risk. I was expecting peoples' priors to be more skeptical (like "what? the end of the world? Really?"). But I think a lot of people were like "yeah, I can totally see how AI could cause global security risks" or even "yeah, I'm actually quite worried about SkyNet-like AIs, and I'm glad other people are working on that."

Often, people seemed genuinely concerned about extinction risks from AI but also didn't have any plans to work on it. I was reminded that it's actually a very rare thing to be like "X is an existential risk" --> "Therefore I should seriously consider working on X." A lot of people were just like "I'm glad someone else is thinking about this [but I'm not going to and I don't expect my congressperson will]."

In terms of my effect– I think I mostly just got them to think about it more and raised it in their internal "AI policy priorities" list. I think people forget that staffers have like 100 things on their priority list, so merely exposing and re-exposing them to these ideas can be helpful.

I also met a few staffers who seemed to care a lot about AI risks and who seemed like strong allies in the AI policy space. I'm still in touch with several, and I introduced a bunch of them to others. If there's a bill that I want to see passed, I think I have a much better understanding of which specific people I'd try to get in touch with. It seems likely to me that most of the impact of the entire DC trip was in figuring out who the allied staffers are & forming initial relationships with them.

One final thing is that I typically didn't emphasize loss of control//superintelligence//recursive self-improvement. I didn't hide it, but I included it in a longer list of threat models, and it was rarely the main thing I was trying to convey. If I was doing this over again, I would probably emphasize these threat models more.

hath

It seems likely to me that most of the impact of the entire DC trip was in figuring out who the allied staffers are & forming initial relationships with them.

Ah, okay! Any traits that were good predictors on whether staffers would be sympathetic to the cause? E.g. specific regions, political leaning, other policies.

Orpheus16

Any traits that were good predictors on whether staffers would be sympathetic to our cause?

Not really. The sample size is pretty small. Like, in total, there were probably ~4 staffers who I would put in the "cares a lot about extinction risk & in a position where they could be pretty helpful in moving forward legislation".

hath

Ah, gotcha. Did the discussions you were having (before the CAIS statement came out) affect anything (phrasing, outreach, etc.) regarding the statement?

Orpheus16

The discussions didn't affect the statement; the statement was written before I left for DC. (Fun fact: I was one of the people involved in drafting the CAIS statement. It is a bit weird to think that contributing to a good sentence was like >100X more impactful than many other things I do, but hey, sometimes it works out that way).

hath

(Fun fact: I was one of the people involved in drafting the CAIS statement. It is a bit weird to think that contributing to a good sentence was like >100X more impactful than many other things I do, but hey, sometimes it works out that way).

Damn. Weird world we live in. Well done on that, by the way.

Orpheus16

Thanks! It has definitely been weird to see how much the statement mattered.

I think it was also pretty humbling– when I first heard about the statement (at the time we were calling it an open letter), I remember being all doomy and being like "meh, what is an open letter going to do? We already had the FLI Pause letter."

This was a useful reminder that sometimes you might not be able to predict the impact of something in advance. In hindsight, it's pretty clear (at least to me) that the CAIS statement was useful & the theory-of-change is very solid. But at the time, it didn't feel like a backchained master plan. It felt like it was just one project on a list of like 20 projects, and it had a kind-of-fuzzy theory-of-change, and it was just another bet that seemed worth taking.

Lessons Learned

hath

Anything you would do differently if you were going about this process again? What were the main things that surprised you/that you feel like you've learned from this?

Orpheus16

I think I would've come with draft legislation (assuming the organization I was with was comfortable with that). It seems like people take you more seriously if you have draft legislation.

I also would've written a much shorter NTIA response– we ended up writing a paper that was like 20+ pages. I would've optimized much more for having shorter materials.

Ah, speaking of which, I would've come with a printed-out 1-pager that explained what CAIS is & summarized the regulatory ideas in the NTIA response. I ended up doing this halfway through, and I would've done this sooner.

Also, I would've come with business cards. People seem to love business cards!

hath

Yeah, that all makes sense, though I definitely wouldn't have guessed it in advance.

Final Takes

hath

I believe I've run out of questions to ask: do you have anything else you want to say? Feel free to ramble.

Orpheus16

Here are a few miscellaneous takes:

My experience in DC made me think that the Overton Window is extremely wide. Congress does not have cached takes on AI policy, and it seems like a lot of people genuinely want to learn. It's unclear how long this will last (e.g., maybe AI risk ends up getting polarized), but we seem to be in a period of unusually high open-mindedness & curiosity.
However, it's also very hard to get Congress to do anything. Like, for rather boring reasons, not many bills get passed. There are a bunch of steps in the process where bills can die. This is even more true when things need to be bipartisan (which currently they do, since we have a Democratic Senate and a Republican House). This mostly updates me toward "wow, the status quo usually results in nothing happening, and there's a lot of work that will need to be done to get any kind of meaningful legislation." With that in mind, I do think we're in a pretty unique situation with AI safety (there aren't that many things that actually pose extinction risks and all sorts of other catastrophic risks; also there aren't that many things that become a priority for the Senate Majority Leader, inspire international summits with world leaders, or become the focus of entire Executive Orders).
Many people are overestimating the amount of "inside game" in DC, especially when it comes to congressional engagement. There are a few sort-of-secret things happening, but for the most part, I don't think anyone has the ball.
I want to see more coordination around specific policy visions. For a while, you were in the Cool Kids Club simply for caring about xrisk. I think the Overton Window has moved a bunch, and we're at the point where it's no longer enough to "care about xrisk." What matters is what specific policies people support & are willing to advocate for.
With that in mind, I also think there are benefits to having the broader AI risk community. Poorly-implemented coordination can lead to nothing getting done because you never get consensus (which currently favors the leading labs and unregulated scaling). Too little coordination can lead to lack of coalition-building and unnecessary conflicts. I think I've moved from "coordination is good" to "coordination is good when done properly but it actually takes skill and tact and effort to do coordination well."
I generally think more people should be writing their views up publicly. It is hard to coordinate when I don't know what people believe. I think the community should be less willing to praise people who have not come out with any particular stances.
I learned a lot about the folks working on AI security issues (I'm referring mostly to folks who are doing AI safety work motivated by a desire to avert xrisk or societal-scale catastrophes.)
1. TLDR: It's complicated. I think the top 10% of thinkers were quite talented and pursuing reasonable theories of change. On the other hand, there were also a lot of people who claim to be interested in AI policy but who don't have a basic understanding of various AI safety threat models. There are also (genuine and reasonable) fears that socially-inept and politically-incompetent newcomers might enter the space in ways that threaten or weaken existing efforts.
2. On balance, I felt like the dominant culture was too dismissive of new policy efforts. I hope this changes as the AI policy conversation continues to move forward and attract new groups of people. I would be excited about a community that had a reaction more like "ah, new people are interested! Let's give you some tips/pointers and point out specific experiences we've had and discuss concrete models of downside risks." The status quo often felt less concrete and (in my opinion) overly protectionistic toward new efforts. I found that the culture made it harder for me to think clearly or perform advocacy, especially what I might call "high-directness advocacy" (where you are EG trying predominantly to convey your internal world-state to people, as opposed to trying predominantly to convey a set of beliefs that will land well with your audience). I think there are serious debates to be had about how "direct" various advocacy efforts should be (and I think some DC folks would actually lose some of their influence/"seriousness points" if they were fully direct), but I was still surprised at the magnitude of the effect– the degree to which the culture seemed to disincentivize me and my peers from being direct. I believe this culture has slowed down new policy efforts considerably, and continues to threaten/weaken/stymy new policy efforts in ways that I think are bad for the world. As with many things, I think the high-level concerns are correct, but there are issues in exactly how these high-level concerns are applied/implemented
3. It's also difficult to evaluate the track record of various people/plans. This is partly because some information is secret, partly because things like "we have good relationships with important stakeholders" is a useful instrumental step but doesn't necessarily translate into impact, and partly because a lot of theories of change are hits-based and take time to yield direct impact (e.g., if someone has developed good relationships with X, maybe at some point X will become extremely relevant for AI regulation, but maybe there's only a 1-10% chance of that being true.) With that said, I think coordination would be easier if people ended up being more explicit about what they believe, more explicit about specific policy goals they are hoping to achieve, and more explicit about their legible wins (and losses). In the absence of this, we run the risk of giving too much power and too many resources to people who "play the game", develop influence, but don't end up using their influence to achieve meaningful change. (see also this Dominic Cummings podcast).
Relatedly, this comment from Oliver Habryka resonates with me a lot. I have found that I often think more clearly when I get some distance from "mainstream" thinkers in the space. There are a lot of antibodies and subtle cultural pressures that can prevent me from thinking about certain ideas and can atrophy my ability to take directed action in the world. (Of course, I don't think the solution is "never interact with mainstream thinkers"– but I do think that people are likely underestimating the negative effects the community has on thinking well & achieving difficult things. I certainly was.)
For people interested in donating, feel free to reach out and I'd be happy to give you my thoughts on various organizations.
If you want to talk to me, feel free to reach out on LessWrong. I enjoy talking to people who are working in AI policy. I'm also open to being pitched on impactful things that I could be doing or others I know could be doing.

hath

Wow, okay. Thanks for doing this dialogue!