Some basic skepticism regarding Nick Cammarata's recent writeup:

• The arguably-main problem of interpretability is that we currently fundamentally don't understand how to write programs that have the capabilities of trained neural networks. How do they transform raw sensory inputs into semantically meaningful abstract concepts? What are the type signatures of those concepts? What operations do they run on them? Modern LLMs work by taking advantage of some completely-unknown-to-us features of reality, features of reality about which we're philosophically confused. This is a large part of why they're opaque; their opaqueness is not just due to their being written in an alien language or mixing up different concepts in same neurons.
• Given that, it seems doubtful that you can produce a comprehensive and legible explanation of a frontier LLM by scaling up normal science powered by LLM agents, as Nick suggests. On the contrary: the theories whose principles SGD implicitly uses to write LLMs' internals sit outside the established paradigms of human science, so producing any such explanation would require inventing such paradigms first. And, well, whether LLM agents are fundamentally capable of that or not, (a) this is not something that they've demonstrated the capability for so far, (b) needing to do this would still invalidate the specific cause for optimism this write-up outlines.
• Sanity-check: Nick states that he now "mostly understands" InceptionV1, and that it's legible all the way through. Does that mean he can now re-implement it in an ordinary programming language, or at least sketch out how such a program would work at a high level? Pretty sure that's not the case. And if so, well, that makes me incredibly skeptical of whatever he has in mind as "mostly understanding" an LLM as well.

You can probably get something useful out of unleashing vast swarms of LLM agents on interpretability tasks. But whatever you'll get, it won't be the full and comprehensive understanding of how they work, and it may not actually bring us much closer to such an understanding. Like, I recall OpenAI doing something similar a few years back, where they had GPT-4 label all GPT-2 neurons. AFAICT, nothing much ever came out of that way of doing things. I think we should expect the same here by default.

Further: the new object-level claims that superposition may have been mistaken and that recursive generalization is the way are plausibly important new advancements. But: (a) per above, I just don't think superposition was ever the main blocker to interpretability, and nothing in the write-up makes me more hopeful about the real blockers, (b) see this thread (including Nick's response) regarding whether the news of the superposition hypothesis' death have been exaggerated or not.