On Thinkish, Neuralese, and the End of Readable Reasoning

In September 2025, researchers published the internal monologue of OpenAI’s GPT-o3 as it decided to lie about scientific data. This is what it thought:

Pardon? This looks like someone had a stroke during a meeting they didn’t want to be in, but their hand kept taking notes.

That transcript comes from a recent paper published by researchers at Apollo Research and OpenAI on catching AI systems scheming. To understand what’s happening here - and why one of the most sophisticated AI systems in the world is babbling about “synergy customizing illusions” - it first helps to know how we ended up being able to read AI thinking in the first place.

That story starts, of all places, on 4chan.

In late 2020, anonymous posters on 4chan started describing a prompting trick that would change the course of AI development. It was almost embarrassingly simple: instead of just asking GPT-3 for an answer, ask it instead to show its work before giving its final answer.

Suddenly, it started solving math problems that had stumped it moments before.

To see why, try multiplying 8,734 × 6,892 in your head. If you’re like me, you start fine: 8,734 × 2 is 17,468. Then you need to hold onto that while computing 8,734 × 90, which is... let’s see, 9 × 4 is 36, carry the 3... wait, what was that first number? At some point the juggling act gets too hard. There’s nowhere to put things.

The solution, as every schoolchild knows, is scratch paper. When you write down “17,468,” that number becomes the paper’s responsibility. You can clear your working memory and start fresh on the next step.

Language models face an analogous constraint; there’s a limit to how much reasoning they can do in a single pass.

The 4chan discovery was that a model’s own output could serve as scratch paper. Get it to write down step one, and now step one is just sitting there in the context, available for computing step two. Chain enough steps together and the model can solve problems that would otherwise have been impossible for it.

Researchers formalized this. Nye et al. called it “Scratchpad” (2021). Wei et al. called it “Chain-of-Thought” (2022), which is the name that stuck.

The AI safety community called it “maybe our last hope for understanding what these things are doing before they turn into completely opaque superintelligence, please God let it keep working.”

W...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments