Advice String Distillation
This post will advocate for a fine-tuning methodology that I think is currently extremely under-rated for alignment and interpretability. It uses Context Distillation, but I don't think it has a proper name of its own yet, so I will refer to it as "Advice String Distillation" here.
The goal is to accomplish the kind of fine-tuning that is done during RLVR, where models are trained to reliably carry out long chains of reasoning in order to accomplish tasks, without actually doing RLVR. A key safety selling-point of Advice String Distillation is the following: The updates to the model's weights have text associated with them. By reading the text, you can get a pretty good idea of how the updates will influence the model's behaviour.
Description
There are two ingredients:
* On-policy self distillation (OPSD). This method trains models to act similarly to a version of themselves that has a given context, without needing to be provided with that context during inference. For example, to teach a model a niche subfield of mathematics, we might do OPSD of the key papers and review articles from that field. This trains the model to act the same way as a version of itself that has read those papers.
* Advice strings. Near-synonyms of "advice string" are "system prompt" or "cheatsheet". These are strings included in the model's context that have been somehow optimized for helping the model complete tasks more successfully.
Advice String Distillation is a two-step process:
1. Search for advice strings that result in high rates of task completion for the tasks we care about.
2. Distill these advice strings into the model's natural behaviour with OPSD.
So when someone gives us a task and a scoring function for that task, we optimize the advice string to produce the largest expected score. We can use the same type of input that RLVR needs.
Other ways of obtaining advice strings are also legitimate. Some people who have spent a lot of time tal