Articles:

There should be more AI safety organisations (Marius Hobbhahn, 2023-09-21)

Why does the AI Safety Community need help founding projects? (Ryan Kidd, 2024-07-12)

AI Assurance Tech Report (2024)

AI Safety as a YC Startup (Lukas Peterson, 2025-01-08)

Alignment can be the ‘clean energy’ of AI (Cameron Berg, Judd Rosenblatt, and Trent Hodgeson, 2025-02-22)

AI Tools for Existential Security (Lizka Vaintrob and Owen Cotton-Barratt, 2025-03-14)

What makes an AI Startup "Net Positive" for Safety (Jacques Thibodeau, 2025-04-18)

AI Safety Undervalues Founders (Ryan Kidd, 2025-11-16)

Incubation Programs:

Catalyze Impact: Catalyze is an AI Safety incubator helping individuals and teams start new AI Safety organizations. Accepts pre-idea entrepreneurs.

Seldon Accelerator: “We take on AGI risk with optimism”

Constellation Incubator: Supports pre-idea entrepreneurs and early-stage organizations

AE Studio has expressed interest in helping seed neglected approaches, incl. ideas that take the form of business ideas

Fifty Years: "We help AI researchers build startups for a safe and aligned future"

D/acc residency: "This will be a first-of-its-kind residency for 15 leading builders to turn decentralized & defensive acceleration from philosophy into practice."

BlueDot AGI Strategy Course: The BlueDot AGI Strategy course seems to effectively be an incubation course, though not necessarily limited to the founding of new orgs. Graduates can apply for funding or to attend an incubation week.

Halycon Futures: “We’re an entrepreneurial nonprofit and VC fund dedicated to making AI safe, secure and good for humanity.” They do new project incubation, grants, investments, etc.

AIS Friendly General Program:

Founding to Give: General EA incubator

Entrepreneur First - They run def/acc, so this could be a decent alternative

Brains Accelerator: "Brains is an accelerator that helps talented scientists and technologists execute on ambitious research visions that are beyond the scope of individual academic labs, startups, or large companies. These visions range from upending the way we deal with carbon to how we understand chronic disease or observe the universe."

Communities:

Entrepreneurship Channel EA Anywhere

AI Safety Founders: Website (with links to resources), Discord, LinkedIn, collaboration request

Funding:

AI Safety Seed Funding Network

Lionheart Ventures: “We invest in the wise development of transformative technologies.”

Juniper Venture: Invests in companies “working to make AI safe, secure and beneficial for humanity”. Published the AI Assurance Tech Report

Polaris Ventures: “We support projects and people aiming to build a future guided by wisdom and compassion for all”

Mythos Ventures: “an early-stage venture capital firm investing in prosocial technologies for the transformative AI era.”

Metaplanet: Founded by Jaan Tallinn. “We invest in deep tech that benefits humanity in the long run. We also make grants to fund projects that don’t fit the venture model... We also love projects that reduce existential risks from AI and other advanced technologies. We tend to skip well-known immediate risks and remedies that get ample attention and investment"

Anthology Fund: Backed by Anthropic. One of the five key areas is “trust and safety tooling”

Menlo Ventures: Lead the GoodFire round (article)

Babuschkin Ventures (contact): "Supports AI safety research and backs startups in AI and agentic systems that advance humanity and unlock the mysteries of our universe"

Safe AI Fund: "an early-stage venture fund dedicated to supporting startups developing tools to enhance AI safety, security, and responsible deployment. The fund provides both financial investment and mentorship"

Shift Grants: "Shift Grants are designed to support scientific and technological breakthrough projects that align with d/acc philosophy: decentralized, democratic, differential, defensive acceleration."

Organizational Support:

Ashgro: fiscal sponsorship

Berkeley Existential Risk Initiative (BERI): fiscal sponsorship

Factor Omega: Access to GPUs, but also potentially other non-capital resources like advising

Rethink Priorities Special Projects: provides fiscal sponsorship or incubation

Impact Ops: "We provide consultancy and hands-on support to help high-impact organizations upgrade their operations."

Other:

Funding options spreadsheet

Hackathon for AI Safety Startups: Ran once by Apart Research, may run again

Constellation Residency: Year-long position

Nonlinear: Free coaching for people who are running an AI safety startup or who are considering starting one

Dustbin - Things that previously existed

Def/acc at Entrepreneur First - this is a new program, which focuses on "defensive" tech, which will produce a wide variety of startups, some of which may help with reducing x-risk, but others may include AI applications like autonomous weapons or diagnostic medical software, which, while they generally won't increase x-risk, may not reduce it either.

Longtermist Entrepreneurship Project - Retrospective

Posts

Wikitag Contributions

Comments