x

AI ALIGNMENT FORUM
AF

nostalgebraist — AI Alignment Forum

nostalgebraist

Top postsTop post

nostalgebraist

Message

I write original fiction.

Also I have opinions about AI and stuff, sometimes.

Elsewhere:

My tumblr blog
My fiction
My (now-dormant) GPT bot

Same person as nostalgebraist2point0, but now I have my account back.

I have signed no contracts or agreements whose existence I cannot mention.

8374

Ω

1365

27

276

13y

nostalgebraist

I write original fiction.

Also I have opinions about AI and stuff, sometimes.

Elsewhere:

My tumblr blog
My fiction
My (now-dormant) GPT bot

Same person as nostalgebraist2point0, but now I have my account back.

I have signed no contracts or agreements whose existence I cannot mention.

Top postsTop post

chinchilla's wild implications

(Colab notebook here.) This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla.[1] The paper came out a few months ago, and has been discussed a lot, but some of its implications deserve more explicit notice in my opinion. In particular: * Data, not size, is the currently active constraint on language modeling performance. Current returns to additional data are immense, and current returns to additional model size are miniscule; indeed, most recent landmark models are wastefully big. * If we can leverage enough data, there is no reason to train ~500B param models, much less 1T or larger models. * If we have to train models at these large sizes, it will mean we have encountered a barrier to exploitation of data scaling, which would be a great loss relative to what would otherwise be possible. * The literature is extremely unclear on how much text data is actually available for training. We may be "running out" of general-domain data, but the literature is too vague to know one way or the other. * The entire available quantity of data in highly specialized domains like code is woefully tiny, compared to the gains that would be possible if much more such data were available. Some things to note at the outset: * This post assumes you have some familiarity with LM scaling laws. * As in the paper[2], I'll assume here that models never see repeated data in training. * This simplifies things: we don't need to draw a distinction between data size and step count, or between train loss and test loss. * I focus on the parametric scaling law from the paper's "Approach 3," because it's provides useful intuition. * Keep in mind, though, that Approach 3 yielded somewhat different results from Approaches 1 and 2 (which agreed with one another, and were used to determine Chinchilla's model and data size). * So you should take the exact numbers below with a grain of salt. They may be o

425Jul 31, 2022

the void

408Jun 11, 2025

the case for CoT unfaithfulness is overstated

269Sep 29, 2024

larger language models may disappoint you [or, an eternally unfinished draft]

261Nov 26, 2021

the void

A long essay about LLMs, the nature and history of the the HHH assistant persona, and the implications for alignment. Multiple people have asked me whether I could post this LW in some form, hence this linkpost. ~17,000 words. Originally written on June 7, 2025. (Note: although I expect this...

Jun 11, 2025•408

the case for CoT unfaithfulness is overstated

[Quickly written, unpolished. Also, it's possible that there's some more convincing work on this topic that I'm unaware of – if so, let me know. Also also, it's possible I'm arguing with an imaginary position here and everyone already agrees with everything below.] In research discussions about LLMs, I often...

Sep 29, 2024•269

nostalgebraist's Shortform

May 27, 2024•7

OpenAI API base models are not sycophantic, at any size

In Discovering Language Model Behaviors with Model-Written Evaluations" (Perez et al 2022), the authors studied language model "sycophancy" - the tendency to agree with a user's stated view when asked a question. The paper contained the striking plot reproduced below, which shows sycophancy * increasing dramatically with model size *...

Aug 29, 2023•183

chinchilla's wild implications

(Colab notebook here.) This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla.[1] The paper came out a few months ago, and has been discussed a lot, but some of its implications deserve more explicit notice in my opinion. In particular:...

Jul 31, 2022•425

wrapper-minds are the enemy

This post is a follow-up to "why assume AGIs will optimize for fixed goals?". I'll assume you've read that one first. I ended the earlier post by saying: > [A]gents with the "wrapper structure" are inevitably hard to align, in ways that agents without it might not be. An AGI...

Jun 17, 2022•108

why assume AGIs will optimize for fixed goals?

When I read posts about AI alignment on LW / AF/ Arbital, I almost always find a particular bundle of assumptions taken for granted: * An AGI has a single terminal goal[1]. * The goal is a fixed part of the AI's structure. The internal dynamics of the AI, if...

Jun 10, 2022•159

Load More (7/15)