x

AI ALIGNMENT FORUM
AF

David Scott Krueger (formerly: capybaralet) — AI Alignment Forum

David Scott Krueger

Top postsTop post

David Scott Krueger

Message

https://twitter.com/DavidSKrueger
https://www.davidscottkrueger.com/
https://therealartificialintelligence.substack.com/p/the-real-ai-deploys-itself

2699

Ω

599

75

477

1

11y

David Scott Krueger

https://twitter.com/DavidSKrueger
https://www.davidscottkrueger.com/
https://therealartificialintelligence.substack.com/p/the-real-ai-deploys-itself

Top postsTop post

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Full version on arXiv | X Executive summary AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal. However, we argue that even an incremental increase in AI capabilities, without any coordinated power-seeking, poses a substantial risk of eventual human disempowerment. This loss of human influence will be centrally driven by having more competitive machine alternatives to humans in almost all societal functions, such as economic labor, decision making, artistic creation, and even companionship. A gradual loss of control of our own civilization might sound implausible. Hasn't technological disruption usually improved aggregate human welfare? We argue that the alignment of societal systems with human interests has been stable only because of the necessity of human participation for thriving economies, states, and cultures. Once this human participation gets displaced by more competitive machine alternatives, our institutions' incentives for growth will be untethered from a need to ensure human flourishing. Decision-makers at all levels will soon face pressures to reduce human involvement across labor markets, governance structures, cultural production, and even social interactions. Those who resist these pressures will eventually be displaced by those who do not. Still, wouldn't humans notice what's happening and coordinate to stop it? Not necessarily. What makes this transition particularly hard to resist is that pressures on each societal system bleed into the others. For example, we might attempt to use state power and cultural attitudes to preserve human economic power. However, the economic incentives for companies to replace humans with AI will also push them to influence states and culture to support this change, using their growing economic power to shape both policy and public opinion, which will in t

188Jan 30, 2025

An Update on Academia vs. Industry (one year into my faculty job)

"Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities)

112Mar 18, 2023

Causal confusion as an argument against the scaling hypothesis

Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development

Full version on arXiv | X Executive summary AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal. However, we argue that even an incremental increase in AI...

Jan 30, 2025•188

A Sober Look at Steering Vectors for LLMs

We thank Madeline Brumley, Joe Kwon, David Chanin and Itamar Pres for their helpful feedback. Introduction Controlling LLM behavior through directly intervening on internal activations is an appealing idea. Various methods for controlling LLM behavior through activation steering have been proposed. Most steering methods add a 'steering vector' (SV) to...

Nov 23, 2024•40

Is there any rigorous work on using anthropic uncertainty to prevent situational awareness / deception?

AI systems up to some high level of intelligence plausibly need to know exactly where they are in space-time in order for deception/"scheming" to make sense as a strategy. This is because they need to know: 1) what sort of oversight they are subject to and 2) what effects their...

Sep 4, 2024•20

[Link Post] "Foundational Challenges in Assuring Alignment and Safety of Large Language Models"

We’ve recently released a comprehensive research agenda on LLM safety and alignment. This is a collaborative work with contributions from more than 35+ authors across the fields of AI Safety, machine learning, and NLP. Major credit goes to first author Usman Anwar, a 2nd year PhD student of mine who...

Jun 6, 2024•70

"Publish or Perish" (a quick note on why you should try to make your work legible to existing academic communities)

This is a brief, stylized recounting of a few conversations I had at some point last year with people from the non-academic AI safety community:[1] Me: you guys should write up your work properly and try to publish it in ML venues. Them: well that seems like a lot of...

Mar 18, 2023•112

What organizations other than Conjecture have (esp. public) info-hazard policies?

I believe Anthropic has said they won't publish capabilities research? OpenAI seems to be sort of doing the same (although no policy AFAIK). I heard FHI was developing one way back when... I think MIRI sort of does as well (default to not publishing, IIRC?)

Mar 16, 2023•20

A (EtA: quick) note on terminology: AI Alignment != AI x-safety

I think the terms "AI Alignment" and "AI existential safety" are often used interchangeably, leading the ideas to be conflated. In practice, I think "AI Alignment" is mostly used in one of the following three ways, and should be used exclusively for Intent Alignment (with some vagueness about whose intent,...

Feb 8, 2023•46

Load More (7/23)