I apologize for not posting this closer to Petrov Day. It’s been a busy month and there was much to think about.

You can view the EA Forum’s retrospective here.

This year was the third Petrov Day celebration on LessWrong in which the site was endangered, and the first year we joined together with the EA Forum. In case you missed it, neither site was taken down, despite 200 people being issued codes that would allow them to do so [1][2]. Huzzah!

Although neither site went down (and thus there's no need for a blow-by-blow analysis of whodunit and why), there are some interesting things to review. In particular, there were some substantial criticisms of the Petrov Day ritual this year and last year that I want to address.

Why Petrov Day

The annual Petrov Day post recounts the basic story of Petrov Day, yet given the questions that were asked this year about what Petrov Day should be, I think it’s right to first revisit why we celebrate Petrov Day in the first place. The following is my own personal take, the one from which I’ve acted, but it is Not Official.

We find ourselves at what may be one of the most critical periods in the history of humanity and the universe. This is kind of crazy–though I’ll refer you to the writings of Holden Karnofsky for a compelling argument for why believing anything else is equally crazy. In the next few decades, we might go extinct (or worse), or we might commence an explosion in progress and productivity that propels us to the stars, allowing us to take the seemingly barren universe and fill it with value.

Petrov Day is a celebration of not going extinct. It’s a commemoration of not taking actions that would destroy the world. It’s about how Petrov chose not to follow policy and relay his alarm because, in his personal estimation, it was probably a false alarm. If he had relayed the alarm, there’s a chance his superiors would have chosen to launch nuclear missiles at the US, and history would be very different.

We can identify two virtues worth applauding in the story:

Choosing actions that don’t destroy the world
Even in the face of pressures otherwise, using one’s judgment to not destroy the world

On September 26th, we celebrate these virtues and attempt to enshrine them in our community. We say to ourselves and others I accept the virtue of not destroying the world, even when there’s pressure to do it! We don’t do this for idle spiritual fulfillment–we do it because there’s a real chance that we or our community may soon face actual choices that resemble Petrov’s. Be it AI, bio, or general policy, our community is represented and our influence is real. As such, the values we take as our own matter.

In addition to the virtues directly displayed by Petrov, we can add others that are important for not destroying the world:

3. Not taking unilaterally taking large (and irreversible) action

4. Cooperating / being the kind of person who can cooperate / being the kind of community that cooperates with itself, especially when the stakes are high

Virtues 2 and 3 are in some tension and there’s probably a meta-virtue of judging which to apply. The default principle might be like “use your own judgment to avoid destructive actions; don’t rely only on your judgment alone to take [potentially] destructive actions.”

Ritual

Eliezer posted about Petrov Day first in 2007 and in 2014, Jim Babcock wrote a ritual guide for a ceremony that people could conduct in small gatherings. At some point, a red button that would end the ceremony was introduced to the tradition. You’d be a real jerk to press it, thereby ending the Petrov Day celebration for everyone.

In 2019, the LessWrong team decided to create a Petrov Day ritual for the entire community by doing something with the website.

I wasn’t involved in Petrov Day that year, but I believe the team then wanted to celebrate all the four virtues I listed above (and maybe others too) as part of a general let’s celebrate the virtues involved in not ending the world. Unfortunately, it’s quite tricky to symbolize 2. (using your own judgment against incentives) within a game.

In addition to celebrating the four virtues above, LessWrong organizers wanted to further use Petrov Day as an opportunity to test (and hopefully prove) the trustworthiness and ability to cooperate of our community. Symbolism is powerful and it’s meaningful if you can get a large group of people to go along with your ritual. From that arose the challenge of finding N people who wouldn’t press the button. The higher the N we could find who don’t press the button, the more people we would have who are bought into our community–all of them treated the value of the trust-building symbolic exercise as more important than having fun or objecting or financial incentive or anything.

I feel pride and reassurance if I imagine truthfully saying “we have 1000 people that if we give them the chance to be a troll or a conscientious objector or a something–they don’t take it, they hold fast in not taking a destructive action”. The LessWrong frontpage is a big deal to the LessWrong team, and putting it on the line was a way of buying some gravitas for the ritual.

It’s because having N people who don’t press the button is such a powerful idea that people regard the ritual seriously and look poorly upon anyone who’d damage that. We succeeded in 2019 with no one pressing the button, yet failed in 2020. 2021 was to be a high-stakes tie-breaker involving another community.

Although the button(s) wasn’t pressed this year, I actually feel that we failed. We were unable to find 200 people (100 for each forum) who wanted to be part of our community of people who don’t take destructive actions. I don’t know that we failed by a lot, but I think we did. This is our failure as organizers as much as anyone else–we were responsible for choosing people and for designing the ritual.

An Aside About Community Buy-In

There has been criticism that the LessWrong team unilaterally designed and deployed the community Petrov Day ritual, deciding for the community at large what was going to be celebrated and how. I think this is a fair charge.

There are historical explanations for why the Petrov ritual evolved the way that it did, and, separately, principles and policies that can speak to whether that's good or bad.

Historically, building A Big Red Button That Takes Down The Site felt like a pretty straightforward evolution of the tradition people were already enacting in their homes and parties. It didn't seem like the sort of step that required public discussion or vetting, and that still seems like the correct decision for 2019

Additionally, the team prepared its Petrov Day ritual somewhat at the last minute, and found itself in a position where a big discussion wasn't really a viable option.

Given the choice between a LessWrong team (and an overall community) where people are willing to try ambitious and potentially-cool things on their own judgment, or one where people err toward doing nothing without discussion and consensus, it seems clearly better for 2019 LW to have forged bravely ahead.

(This is actually a good place to distinguish the Petrov Day moral of "don't take irreversible and destructive actions on your own authority" from a more general moral of "don't do anything on your own authority." The latter is no good.)

That being said, though: community rituals are for the community, and LessWrong is closer to being something like a public utility than it is to being the property of the LessWrong team. At this stage, it feels right and proper for the community to have greater input and a greater say than in 2019, and without having specific plans, I expect us to put real effort into making that happen well in advance of Petrov Day 2022. This feels especially important given both that Petrov Day now seems like it's going to be an enduring piece of our subculture, and also that we want it to be.

Not Getting Opt-In

Speaking of consulting the community, the 2021 ritual consisted of making people part of the game involuntarily by sending them launch codes. I see a few different complaints here.

The first is that launch codes are hazardous. Because the Petrov Day ritual is treated seriously (more on this below), someone who enters them (or just talks about entering them!) is subject to real social sanction, up to and including it affecting their job prospects. Our community takes character judgments seriously, and it's not at all clear what aspects of something like Petrov Day are "off limits" when it comes to evaluating people's cooperativeness, trustworthiness, impulsiveness, and general judgment.

In a world where the letter containing the codes was unambiguous about the cultural significance and the stakes of the Petrov Day ritual, I think receiving the launch codes would only endanger the highly impulsive and those with poor reading comprehension (and those should reasonably affect your job prospects). However, I think the way I wrote this year’s letter could be interpreted as a “Murder Mystery” invitation by someone not aware of the Petrov Day context. Plus, the letter didn’t explain the cultural significance to people who hadn’t been following along the LessWrong Petrov Day celebrations in last two years, which especially seems like a misstep when reaching out to a whole new subculture (i.e., the EA Forum).

I screwed up on that account and I’m sorry to anyone I put at risk. If you had pressed the button, it would have been on me.

The second–and I think more serious–complaint around lack of opt-in is that it leaves people who object to the ritual with no good option. If you don’t press the button, you are tacitly cooperating with a ritual you object to; if you do press it, you’ll have destroyed value and be subject to serious social sanction.

Moreover, the organizers (me, EA Forum staff) have declared by fiat what the moral significance of people’s symbolic actions are. This goes beyond just deciding what the ritual is and into deciding what’s good and bad symbolic behavior (with strong social consequences). While the Petrov Day ritual might be innocuous, it is a scary precedent if LessWrong/EA Forum organizers freely shape the moral symbolic landscape this way, without the checks and balances of broader community discussion.

I think this is fair. and this makes me realize that the LessWrong team has more power (and therefore more responsibility) than we previously credited oursevles with. We set out to build culture, including ritual and tradition, but it’s another matter to start defining the boundaries of good and bad. I think possibly this should be done, but again probably with more community consultation.

Why So Serious

Related to both complaints is the fact that Petrov Day has been treated increasingly seriously. It’s because it’s serious that people will sanction you if you press the button. And it’s because you believe it’s too serious that you might want to object/boycott the ritual (well, that’s one reason).

I think the degree of seriousness that the ritual is treated with is one of the questions that should be reconsidered next year in consultation with the community. It's possible, for instance, that Petrov Day should be a place where some amount of mischievousness is considered fair game, and not representative of someone's global character.

Notwithstanding, I personally want to defend the position that a very high degree of seriousness is appropriate: a serious ritual for a serious situation. The stakes we find ourselves facing in this century are genuinely high–astronomical value vs extinction–and it makes sense to me to have a ritual that we treat with reverence, to celebrate and encourage values that we treat as somewhat sacred. Or in short, things matter, so let’s act like they do. I don’t know that this argument will win out on net, but I think seriousness should be considered.

Aside from a general position that Petrov Day should not be serious, some have argued in particular the most recent Petrov Day ritual should be lighthearted because the only thing at stake is the LessWrong/EA Forum page going down. My response to that is sadness. There is understandably inferential distance between the LessWrong team and others about how valuable LessWrong is and what it means to take the site down for a day. As I wrote in the Petrov Day post:

One of the sites [LessWrong, EA Forum] going down means hundreds to thousands of people being denied access to important resources: the destruction of significant real value. What's more it will damage trust between the two sites...For the rest of the day, thousands of people will have a hard time using the site, some posts and comments will go unwritten.

LessWrong is not a mere source of entertainment. It’s a site whose content shapes how people think about their lives and make major decisions. If there was a person who was going to have their life changed by LessWrong (and this happens to many) who fails to because the site is down, that’s a tragic loss.

LessWrong is also used as a major communication tool between researchers. LessWrong being offline is not so different from removing a day from a major research conference. Or, to change tack: the operating budget of the LessWrong website has historically been ~$600k, and this budget is artificially low because the site has paid extremely below-market salaries. Adjusting for the market value of the labor, the cost is more like $1M/year, or $2,700/day. If I assume LessWrong generates more value than the cost required to run it, I estimate that the site provides at least $2,700/day in value, probably a good deal more.

Still, if we want stakes for the ritual/exercise/game, probably better to use something with lower inferential distance. It’s on me as an organizer to mistakenly think that just because I think something is valuable, that will be transparent to others, and given that, I accept that it’s on me that not everyone thought the last Petrov Day iteration should be a big deal.

I could imagine it being better if there’s $5-10k that simply gets burned if someone presses the button rather than going to some worthy cause. Either way, this debate has clearly not properly taken place.

For an idea of what next year could look like, see these notes from Ben Pace

An Aside: Repeating Mistakes

Many of the issues pointed out this year were pointed out last year. It’s a real failure to not have addressed them. This is my (Ruby’s) fault. I took over organizing Petrov Day this year (inviting the EA Forum to join LessWrong) but didn’t go back and re-read through the previous year’s comments. Had I done so, I could have avoided repeating some of the previous mistakes.

I do think that repeating mistakes is quite bad and am quite sorry for that.

Wrapping Up

Stanislav Petrov was on duty at a particularly fraught time in history. I think we are, too. This makes it imperative to think about the kinds of decisions we might face and prepare ourselves for them. It makes it crucial that know and practice our values and principles, so that we can rely on them even when temptations are strong or matters are unclear.

Rituals and traditions are what keep people true to their values. Having them or not might be the difference between us being a community that can succeed at its ambitious goals vs not–the difference between colonizing the stars and annihilation.

I regret the flaws of Petrov Day rituals so far, but I’m excited to keep iterating and innovating so we can make these essential values part of our community, cultures, and selves.

[1] We apologize for sending codes to some people who did not want the opportunity/responsibility/involvement of receiving them.

[2] On the day itself, 70 out of 100 EA Forum recipients opened the email, but only 30 out of 100 LessWrong recipients (perhaps due to emails getting eaten by spam).