Many thanks to Brandon Goldman, David Langer, Samuel Härgestam, Eric Ho, Diogo de Lucena, and Marc Carauleanu, for their support and feedback throughout.

Most alignment researchers we sampled in our recent survey think we are currently not on track to succeed with alignment–meaning that humanity may well be on track to lose control of our future.

In order to improve our chances of surviving and thriving, we should apply our most powerful coordination methods towards solving the alignment problem. We think that startups are an underappreciated part of humanity’s toolkit, and having more AI-safety-focused startups would increase the probability of solving alignment.

That said, we also appreciate that AI safety is highly complicated by nature^[1] and therefore calls for a more nuanced approach than simple pro-startup boosterism. In the rest of this post, we’ll flesh out what we mean in more detail, hopefully address major objections, and then conclude with some pro-startup boosterism.

Expand the alignment ecosystem with startups

We applaud and appreciate current efforts to align AI. We could and should have many more. Founding more startups will develop human and organizational capital and unlock access to financial capital not currently available to alignment efforts.

“The much-maligned capitalism is actually probably the greatest incentive alignment success in human history” – Insights from Modern Principles of Economics

The alignment ecosystem is limited on entrepreneurial thinking and behavior. The few entrepreneurs among us commiserate over this whenever we can.

We predict that many interested in alignment seem to do more to increase P(win) if they start thinking of themselves as problem-solvers specializing in a particular sub-problem first, deploying whatever approaches are appropriate in order to solve the smaller problem. Note this doesn’t preclude scaling ambitiously and solving bigger problems later on.^[2]

Running a company that is targeting a particular niche of the giant problem seems like one of the best ways to go about this transition, unlocking a wealth of best practices that could be copied. For example, we’ve seen people in this space raise too little, too late, resulting in spending unnecessary time in the fundraising stage instead of doing work that advances alignment. We think this is oft...

Startups