Also available on the EA Forum.
Followed by: Encultured AI, Part 2 (forthcoming)

Hi! In case you’re new to Encultured AI, we’re a for-profit start-up with a public benefit mission: developing technologies promoting the long-term survival and flourishing of humanity and other sentient life. However, we also realize that AI poses an existential risk to humanity if not developed with adequate safety precautions. Given this, our goal is to develop products and services that help humanity steer toward the benefits and away from the risks of advanced AI systems. Per the “Principles” section of our homepage:

Our current main strategy involves building a platform usable for AI safety and alignment experiments, comprising a suite of environments, tasks, and tools for building more environments and tasks. The platform itself will be an interface to a number of consumer-facing products, so our researchers and collaborators will have back-end access to services with real-world users. Over the next decade or so, we expect an increasing number of researchers — both inside and outside our company — will transition to developing safety and alignment solutions for AI technology, and through our platform and products, we’re aiming to provide them with a rich and interesting testbed for increasingly challenging experiments and benchmarks.

In the following, we’ll describe the AI existential safety context that motivated us to found Encultured, and go into more detail about what we’re planning to do.

What’s trending in AI x-safety?

The technical areas below have begun to receive what we call “existential attention” from AI researchers, i.e., attention from professional AI researchers thinking explicitly about the impact of their work on existential safety:

Trustworthiness & truthfulness — ensuring AI systems are telling us the truth and doing the things they and their creators say they’re going to do.
Preference learning — enabling AI systems to learn what humans want.
Interpretability — enabling humans to understand what AI systems are thinking and doing.
Robustness & risk management — ensuring AI systems continue functioning well in novel situations, and quantifying the risk that they won’t.

In other words, the topics above lie in the intersection of the following Venn diagram:

See Appendix 1 for examples of research in these areas. More research in th...

Posts

Wikitag Contributions

Comments