Epistemic status: All of the western canon must eventually be re-invented in a LessWrong post. So today we are re-inventing federalism.

Once upon a time there was a great king. He ruled his kingdom with wisdom and economically literate policies, and prosperity followed. Seeing this, the citizens of nearby kingdoms revolted against their leaders, and organized to join the kingdom of this great king.

While the kingdom's ability to defend itself against external threats grew with each person who joined the land, the kingdom's ability to defend itself against internal threats did not. One fateful evening, the king bit into a bologna sandwich poisoned by a rival noble. That noble quickly proceeded to behead his political enemies in the name of the dead king. The flag bearing the wise king's portrait known as "the great unifier" still flies in the fortified cities where his successor rules with an iron fist.

Once upon a time there was a great scientific mind. She developed a new theoretical framework that made large advances on the hardest scientific questions of the day. Seeing the promise of her work, new graduate students, professors, and corporate R&D teams flocked into the field, hungry to tackle new open problems and make their mark on the world. Within ten years, a vibrant new academic field had formed, with herself among its most respected members.

While the field's ability to make progress on the hard problems increased with each new researcher who joined the field, the field's ability to defend itself against the institutional incentives of the broader academic ecosystem did not. Low-quality researchers, seeing lucrative new opportunities for publication, began producing flashy results on the easier problems adjacent to her field with low attention to scientific rigor. Seeing their success, others began to join them, attracted to the social and financial rewards. Being conflict averse and not seeing it as her job to prosecute these people, a growing fraction of the field became careerists.

Twenty years later, her scientific field had become so diluted by uninteresting or irrelevant work that the great original problems remained unsolved, mired in bureaucracy, respectability politics, and academic warfare. Most of the scientists who joined early, attracted by the promise of great progress, stopped being scientists altogether and moved to industry. Almost nobody remembers her name in the history books.

Once upon a time there was a great advocate. She built a social movement around the protection of the rights of a marginalized group, and after many years of hard work, saw the day that the most severe forms of discrimination against the group had been outlawed, and wide social consensus had moved in favor of respecting the members of this group.

But in the success of the movement's aims, she also lost most of her authority. No longer having a compelling vision to offer the members of this movement, others who did became more influential. While she remained the acknowledged founder of the movement, she was no longer treated by the general public as its spokesperson. The press would always talk to the new, charismatic leaders of the movement who had the strongest and most unyielding views. She couldn't afford to make enemies in the movement that she considered hers, so she would publicly endorse the perspectives of these new leaders even when she privately disagreed with them.

Ten years later, her social movement had become so focused on purity and removing any remaining trace of its original enemy that it had begun causing substantially more harm than the original problem it was founded to address. In the history books, she would be briefly mentioned as one of the people who laid the groundwork for the new dark age.

Once upon a time emperor Marcus Aurelius (himself a great general and a great leader) died in 180 AD, and was succeeded by his son Commodus. Commodus, whom historian Cassius Dio described as "a greater curse to the Romans than any pestilence or any crime", turned out to be interested in gladiator fighting much more than in governing the Roman Empire. The Pax Romana began its long descent into the Crisis of the Third Century, and marked the start of the eventual collapse of the Roman Empire.

Once upon a time the French revolution swept across France, bringing the people liberty and executing the corrupt French aristocracy in an unprecedented flurry of violence. Within a decade the idealistic leaders of the revolution would mostly all be dead, executed by the political machine they themselves had created. And within another few years, Napoleon Bonaparte would claim power and proceed to wage aggressive war across all of continental Europe for another decade.

Once upon a time Lee Kuan Yew built modern Singapore out of what was, at the time, a small regional trading post in Southeast Asia. Under his leadership, Singapore's GDP per capita grew 30x over 30 years. But Lee Kwan Yew is dead and his son just handed over power to Lawrence Wong, not a member of the Lee family. While Singapore continued to thrive under his son's leadership, I find myself very worried about what happens once the Singapore story depends on a third generation of leaders, and wonder if Singapore has in fact already peaked.

Once upon a time George Washington retired. George Washington, the Continental Army general who defeated the British army and successfully established the United States of America as an independent nation, and later the first United States president, served his two terms as president and then voluntarily relinquished power. King George III of Great Britain called him "the greatest man in the world" upon hearing the news. Some say this decision singlehandedly saved American democracy.

Do not conquer what you cannot defend.

At the heart of classical liberalism, a philosophy I have much sympathy for, is the belief that allowing many individuals to act freely and autonomously (especially when they are empowered by markets, democratic processes, and the scientific method) will tend to produce outcomes that are better than the outcomes that can be produced by central authorities.

Maybe the most important way ambitious, smart, and wise people leave the world worse off than they found it is by seeing correctly how some part of the world is broken and unifying various powers under a banner to fix that problem — only for the thing they have built to slip from their grasp and, in its collapse, destroy much more than anything previously could have.

I sometimes consider quitting. When I do, my friends and colleagues often react with bafflement. "How can you think that what you've done is bad for the world? Do you not think that you are steering this boat we are in together into a good direction? Do you really think a world without the AI Safety movement, without LessWrong, without Effective Altruism would be better?".

And in their heads when they visualize the alternative, I can only imagine that they see a great big emptiness where rationality and EA and AI Safety is. And they compare our current community against nothingness, and come to the conclusion that even if its leadership is kind of broken, and the incentives are kind of messed up, that this is still clearly better than no one in the world working on the things we care about.

But what I am worried about, is that we conquered much more than we can defend. That the alternative to the work of me and others in the space is not nothingness, but a broken and dysfunctional and confusing patchwork of metaphorical city-states that barely does anything, but at least when any part of it fails, it doesn't all go down together, and in its distributed nature, promises much less nourishing food to predators and sociopaths.

In grug language: Smart man sees big problem. Often state of nature is many small things. Smart man make one big thing out of many small things to throw at big problem. But then evil man take big thing from smart man and make more problem. Or big thing grow legs and beat smart man without making problem go away. This is bad. Maybe better to throw small things at big problem and not make big thing, even if solve problem less. Or before make big thing have plan for how to not have big thing do evil.

But Moloch, in whom I sit lonely

"But what about Moloch" you say!

"Your principle betrays itself. If we want to have good things, we need to coordinate and work together. And death comes for us all, eventually, so nothing we build can truly be defended. Do you not see how one company owning one lake will produce more fish than 20 companies each polluting the commons until all fish are dead? Do you not see how having 20 AI companies all racing to the precipice is worse than having one clearly in the lead, even if the one that raced to the top might stray from the intentions of its creators?"

And you know, fair enough. Coordination problems are real. I am not saying that you should not centralize power.

Here I am arguing for a much narrower principle. Much has been written, and will continue to be written, about the tradeoff between freedom and justice. About small vs. big government. I am not trying to cover all of that.

Here I am just trying to highlight a single principle that seems robust across a wide range of tradeoffs: "If you make a plan that involves concentrating a bunch of power, especially in the name of goodness and justice, really actually think about whether you can defend that power from corruption and adversaries".

And if you can, then go ahead! When George Washington stepped down, he traded off direct power in favor of a system that would actually be able to defend the principles he cared about for much longer, birthing much of Western democracy. I am glad the US exists and covers almost all of the north American continent. Its leaders and founders did have a plan for defending what they conquered, and the world is better off for it.

But if your plan involves rallying a bunch of people under the banner of truth and goodness and justice, and your response to the question of "how are you going to ensure these people will stay on the right path?" is "they will stay on the right path because they will be truthseeking, good, and just people", or if as a billionaire your plan for distributing your wealth is "well, I'll hire some people to run a foundation for me to distribute all of my money according to my goals", then I think you are in for a bad time.