Introduction

I know very little about a staggering number of topics that would be incredibly useful for my research and/or navigating the field. Part of the problem is the sheer number of choices -- I can't make myself study one thing very long because I always feel like I need to learn 20 other things.

To solve this problem, I started to implement monthly deep dives into a topic. A month is short enough that even I can stay relatively on track for that long, while still being enough time to actually learn something. The goal is not to master the topic completely (which would be impossible); it's to get a finer map of the territory, and to be able to discuss relevant ideas on this topic.

This first month was dedicated to transformers and the GPT-3 model, a topic I felt like I had to do, but which actually kind of grew on me.

Note that this post is a very quickly written summary of what I did and how it went (a bit like TurnTrout's sequence, with probably less insights). This is not a distillation post, and if you read it, you will not learn that much about the subject. That being said, it might prove useful if you want to go learn it by yourself.

Thanks to Jérémy, Flora, Connor, Kyle and Laria for great discussions that helped me understand this topic further.

The Plan

I based myself quite loosely on the structure advocated in Scott Young's Ultralearning. Which only means that I made a planning week by week, checked what resources were recommended beforehand, and tried to focus on the direct applications I wanted to make of my learning, which is explaining it to people and having interesting discussions on the subject.

My idea was that in order to understand GPT-3, I needed first to understand the Transformer architecture, then the GPT family, then play with GPT-3 directly. Since I started this deep dive on the 8th of April, the planning was broadly:

Week of the 8th: Study transformers. Here are the resources I had in mind
- Original paper (Attention is all you need)
- Annotated version of original paper
- Stack Overflow explanation of an apparently difficult point (Queries, keys and values)
- Explanatory blog post 1
- Explanatory blog post 2
- Blog post on the historical significance of transformers (nostalgebraist)
- Survey of attention mechanisms
- Survey of transformer family
Week of the 15th: Study the GPT family of models. Resources once again
- Original papers
- History
  - https://

...

Comprehensive Information Gatherings