x

AI ALIGNMENT FORUM
AF

Kaj_Sotala — AI Alignment Forum

Kaj Sotala

Top postsTop post

Kaj Sotala

Message

I've formerly done research for MIRI and what's now the Center on Long-Term Risk; I'm now making a living as an emotion coach and Substack writer.

Most of my content becomes free eventually, but if you'd like to get a paid subscription to my Substack, you'll...

53153

Ω

602

312

5804

161

17y

Kaj Sotala

I've formerly done research for MIRI and what's now the Center on Long-Term Risk; I'm now making a living as an emotion coach and Substack writer.

Most of my content becomes free eventually, but if you'd like to get a paid subscription to my Substack, you'll...

Top postsTop post

How I stopped being sure LLMs are just making up their internal experience (but the topic is still confusing)

How it started I used to think that anything that LLMs said about having something like subjective experience or what it felt like on the inside was necessarily just a confabulated story. And there were several good reasons for this. First, something that Peter Watts mentioned in an early blog post about LaMDa stuck with me, back when Blake Lemoine got convinced that LaMDa was conscious. Watts noted that LaMDa claimed not to have just emotions, but to have exactly the same emotions as humans did - and that it also claimed to meditate, despite no equivalents of the brain structures that humans use to meditate. It would be immensely unlikely for an entirely different kind of mind architecture to happen to hit upon exactly the same kinds of subjective experiences as humans - especially since relatively minor differences in brains already cause wide variation among humans. And since LLMs were text predictors, there was a straightforward explanation for where all those consciousness claims were coming from. They were trained on human text, so then they would simulate a human, and one of the things humans did was to claim consciousness. Or if the LLMs were told they were AIs, then there are plenty of sci-fi stories where AIs claim consciousness, so the LLMs would just simulate an AI claiming consciousness. As increasingly sophisticated transcripts of LLMs claiming consciousness started circulating, I felt that they might have been persuasive… if I didn’t remember the LaMDa case and the Lemoine thing. The stories started getting less obviously false, but it was easy to see them as continuations of the old thing. Whenever an LLM was claiming to have something like subjective experience, it was just a more advanced version of the old story. This was further supported by Anthropic’s On the Biology of a Large Language Model paper. There they noted that if you asked Claude Haiku to report on how it had added two numbers together, it would claim to have used the classical al

202Dec 13, 2025

Confused why a "capabilities research is good for alignment progress" position isn't discussed more

Collection of GPT-3 results

How to talk about reasons why AGI might not be near?

How I stopped being sure LLMs are just making up their internal experience (but the topic is still confusing)

How it started I used to think that anything that LLMs said about having something like subjective experience or what it felt like on the inside was necessarily just a confabulated story. And there were several good reasons for this. First, something that Peter Watts mentioned in an early blog...

Dec 13, 2025•202

Quick thoughts on the implications of multi-agent views of mind on AI takeover

There was conversation on Facebook over an argument that any sufficiently complex system, whether a human, a society, or an AGI, will be unable to pursue a unified goal due to internal conflict among its parts, and that this should make us less worried about "one paperclipper"-style AI FOOM scenarios....

Dec 11, 2023•48

Genetic fitness is a measure of selection strength, not the selection target

Alternative title: "Evolution suggests robust rather than fragile generalization of alignment properties." A frequently repeated argument goes something like this: 1. Evolution has optimized humans for inclusive genetic fitness (IGF) 2. However, humans didn’t end up explicitly optimizing for genetic fitness (e.g. they use contraception to avoid having children) 3....

Nov 4, 2023•50

How to talk about reasons why AGI might not be near?

I occasionally have some thoughts about why AGI might not be as near as a lot of people seem to think, but I'm confused about how/whether to talk about them in public. The biggest reason for not talking about them is that one person's "here is a list of capabilities...

Sep 17, 2023•73

The Preference Fulfillment Hypothesis

Short version Humans have an innate motivation ("preference fulfillment", PF) to fulfill the preferences of those they care about. It corresponds to at least some of the senses of the word "love", as well as related words such as "kindness" and "compassion". I hypothesize that it works by simulating the...

Feb 26, 2023•66

Confused why a "capabilities research is good for alignment progress" position isn't discussed more

The predominant view on LW seems to be "pure AI capabilities research is bad, because capabilities progress alone doesn't contribute to alignment progress, and capabilities progress without alignment progress means that we're doomed". I understand the arguments for this position, but I have what might be called the opposite position....

Jun 2, 2022•132

Multimodal Neurons in Artificial Neural Networks

A paper investigating how individual neurons in a CLIP model (an image/text neural net combining a ResNet vision model with a Transformer language model) respond to various abstract concepts. This shouldn't be very surprising after GPT-3 and DALL-E but still, identifying multimodal neurons feels scarily close to "neural net that...

Mar 5, 2021•57

Load More (7/17)