x

AI ALIGNMENT FORUM
AF

DragonGod — AI Alignment Forum

Cinera Verinia

Top postsTop post

Cinera Verinia

Message

Theoretical Computer Science Msc student at the University of [Redacted] in the United Kingdom.

I'm an aspiring alignment theorist; my research vibes are descriptive formal theories of intelligent systems (and their safety properties) with a bias towards constructive theories.

I think it's important that our theories of...

2498

Ω

106

86

835

5

9y

Cinera Verinia

Theoretical Computer Science Msc student at the University of [Redacted] in the United Kingdom.

I'm an aspiring alignment theorist; my research vibes are descriptive formal theories of intelligent systems (and their safety properties) with a bias towards constructive theories.

I think it's important that our theories of...

Top postsTop post

Why The Focus on Expected Utility Maximisers?

Epistemic Status Unsure[1], partially noticing my own confusion. Hoping Cunningham's Law can help resolve it. Related Answer Confusions About Arguments From Expected Utility Maximisation Some MIRI people (e.g. Rob Bensinger) still highlight EU maximisers as the paradigm case for existentially dangerous AI systems. I'm confused by this for a few reasons: 1. Not all consequentialist/goal directed systems are expected utility maximisers * E.g. humans 2. Some recent developments make me sceptical that VNM expected utility are a natural form of generally intelligent systems 1. Wentworth's subagents provide a model for inexploitable agents that don't maximise a simple unitary utility function 1. The main requirement for subagents to be a better model than unitary agents is path dependent preferences or hidden state variables 2. Alternatively, subagents natively admit partial orders over preferences 1. If I'm not mistaken, utility functions seem to require a (static) total order over preferences 1. This might be a very unreasonable ask; it does not seem to describe humans, animals, or even existing sophisticated AI systems 3. I think the strongest implication of Wentworth's subagents is that expected utility maximisation is not the limit or idealised form of agency 2. Shard Theory suggests that trained agents (via reinforcement learning[2]) form value "shards" 1. Values are inherently "contextual influences on decision making" 1. Hence agents do not have a static total order over preferences (what a utility function implies) as what preferences are active depends on the context 1. Preferences are dynamic (change over time), and the ordering of them is not necessarily total 2. This explains many of the observed inconsistencies in human decision making 2. A multitude of value shards do not admit analysis as a simple unitar

118Dec 27, 2022

Tracr: Compiled Transformers as a Laboratory for Interpretability | DeepMind

Beren's "Deconfusing Direct vs Amortised Optimisation"

DragonGod's Shortform

Beren's "Deconfusing Direct vs Amortised Optimisation"

Preamble I heavily recommend @beren's "Deconfusing Direct vs Amortised Optimisation". It's a very important conceptual clarification that has changed how I think about many issues bearing on technical AI safety. Currently, it's the most important blog post I've read this year. This sequence (if I get around to completing it)...

Apr 7, 2023•52

Tracr: Compiled Transformers as a Laboratory for Interpretability | DeepMind

Abstract Interpretability research aims to build tools for understanding machine learning (ML) models. However, such tools are inherently hard to evaluate because we do not have ground truth information about how ML models actually work. In this work, we propose to build transformer models manually as a testbed for interpretability...

Jan 13, 2023•62

Why The Focus on Expected Utility Maximisers?

Epistemic Status Unsure[1], partially noticing my own confusion. Hoping Cunningham's Law can help resolve it. Related Answer Confusions About Arguments From Expected Utility Maximisation Some MIRI people (e.g. Rob Bensinger) still highlight EU maximisers as the paradigm case for existentially dangerous AI systems. I'm confused by this for a few...

Dec 27, 2022•118

DragonGod's Shortform

May 31, 2022•4