Theo Pavlakou

Post-training: why SFT comes before RL

In this post I'm going to explain why it's sometimes essential to post-train an LLM using Supervised Fine-Tuning, before doing RL and when it might not be necessary. The specific case that I'll be considering is when we have a large, capable model, which we'd like to distil into a smaller model.

Distillation: the teacher and the student

In many cases, we have a large, very capable model which exhibits some behaviour that we'd like it to, but it's too expensive to use for serving for most queries. In these situations, we might try to distil the model into a smaller model (or at least the desired capability/behaviour), which is vastly cheaper. One way to do this is using Supervised Fine-Tuning (SFT). This idea was popularized in Hinton et al. (2015), and in its most pure form, this consists of taking a set of input prompts, putting them through the larger, more capable model (the Teacher) and then training the smaller model (the Student) on the logits of the teacher. However, in practice, what's usually done is to just sample outputs from the teacher for input prompts and then to save those outputs as a dataset. This is a lot cheaper, because it means that we don't need to load both the teacher and the student model in memory when training the student model. In other words, we try to minimize the following Kullback-Leibler (KL) Divergence:

D(pT(y∣x)∣∣pθ(y∣x))=EpT[log⁡(pT(y∣x)pθ(y∣x))]=EpT[log⁡(pT(y∣x))−log⁡(pθ(y∣x))]=H(pT)−EpT[log⁡(pθ(y∣x))]=−EpT[log⁡(pθ(y∣x))]+C≈−1N∑n=1Nlog⁡(pθ(y(n)∣x(n)))+C\begin{align} \mathbb{D}(p_T(y|x) || p_{\theta}(y|x)) &= E_{p_T}\left[\log\left(\frac{p_T(y|x)}{p_\theta(y|x)}\right)\right]\\ &= E_{p_T}\left[\log\left(p_T(y|x)\right) - \log\left({p_\theta(y|x)}\right)\right]\\ &= H(p_T) - E_{p_T}\left[\log\left({p_\theta(y|x)}\right)\right]\\ &= - E_{p_T}\left[\log\left({p_\theta(y|x)}\right)\right] + C\\ &\approx -\frac{1}{N}\sum_{n=1}^N \log\left({p_\theta(y^{(n)}|x^{(n)})}\right) + C \end{align}

where:

  • pθp_\theta is the student model probability distribution, which has parameters θ\theta
  • pTp_T is the teacher model probability distribution
  • x(n)x^{(n)} is the nnth input to the Teacher model
  • y(n)y^{(n)} is the nnth output sequence from the Teacher model for the nnth input
  • H(pT)H(p_T) is the entropy of the Teacher model distribution, which is a constant with respect to the parameters θ\theta (which is why it is replaced by CC)

This is known as the Forward KL divergence and the important thing to note here is that the expectation is taken with respect to the Teacher distribution i.e. we sample from the teacher and then try to force the student to behave similarly.

Reinforcement Learning

The purpose of reinforcement learning is to reinforce existing behaviours that the student may exhibit. For the most part, RL algorithms consist of the student model sampling a few trajectories (i.e. token sequences for a given input prompt) and for some verifier model to say which ones are correct and for the weights of the model to be altered to maximise the probability of those trajectories being repeated. This is what we mean by RL algorithms being on-policy. The policy is just the output distribution of the student model, and when we say that the algorithm is on-policy, we mean that the samples come from the student model, so are by definition on-policy. This is in contrast to SFT, which samples from the teacher model, which would be off-policy (as the teacher distribution is, by definition, different to the student model).

Note: the important part here is that the student model needs to already be able to perform the correct behaviour (or the one that is rewarded positively), otherwise it will never get a positive reward (and thus, not learn).

When optimizing using an RL objective, we typically end up with some learning objective that involves (either implicitly or explicitly) the Reverse KL Divergence:

D(pθ(y∣x)∣∣pT(y∣x))=Epθ[log⁡(pθ(y∣x)pT(y∣x))]≈1N∑n=1Nlog⁡(pθ(y(n)∣x(n))pT(y(n)∣x(n)))\begin{align} \mathbb{D}(p_{\theta}(y|x) || p_T(y|x)) &= E_{p_\theta}\left[\log\left(\frac{p_\theta(y|x)}{p_T(y|x)}\right)\right]\\ &\approx \frac{1}{N}\sum_{n=1}^N \log\left(\frac{p_\theta(y^{(n)}|x^{(n)})}{p_T(y^{(n)}|x^{(n)})}\right) \end{align}

In this case, the expectation is taken with respect to the student distribution (since we're sampling from the student distribution).

The two types of KL divergence

So now that we have mapped SFT to the forward KL divergence and RL to the reverse KL divergence, what are the implications of this? This is where a common property of the two comes into play.

Forward KL Divergence: mode covering

The Forward KL Divergence is known to result in mode covering behaviour. To understand what this means, we should look at the formula above for it again:

D(pT(y∣x)∣∣pθ(y∣x))=EpT[log⁡(pT(y∣x)pθ(y∣x))]\begin{align} \mathbb{D}(p_T(y|x) || p_{\theta}(y|x)) &= E_{p_T}\left[\log\left(\frac{p_T(y|x)}{p_\theta(y|x)}\right)\right] \end{align}

To simplify the analysis, let's just pretend we have a single example from the Teacher distribution instead of using the whole expectation i.e. we are trying to minimize

−log⁡(pθ(y(n)∣x(n)))+C\begin{align} -\log\left({p_\theta(y^{(n)}|x^{(n)})}\right) + C \end{align}

for the y(n)y^{(n)} sampled from the teacher. So what happens if the student gives a very low probability to this sample i.e. pθ(y(n)∣x(n))p_\theta(y^{(n)}|x^{(n)}) is low? Well, the logarithm would become very negative, so the negative log-likelihood would become very positive and the loss would be maximised (the opposite of what we want). In other words, if the teacher samples a particular response, to minimise the Forward KL Divergence, the student is forced to be able to also sample that response, which is what we mean by mode covering. In other words, every mode of the teacher distribution (areas of local maxima) will likely be covered by the student model (as long as we sample from them when creating the distillation data).

Reverse KL Divergence: mode seeking

Now, let's look at the Reverse KL Divergence this time:

D(pθ(y∣x)∣∣pT(y∣x))=Epθ[log⁡(pθ(y∣x)pT(y∣x))]\mathbb{D}(p_{\theta}(y|x) || p_T(y|x)) = E_{p_\theta}\left[\log\left(\frac{p_\theta(y|x)}{p_T(y|x)}\right)\right]

In this case, it's easier to think of the equation as a whole, as opposed to a single sample. If we look at the equation above, we notice that the expectation is over pθp_\theta i.e. the student distribution, which is what we're optimizing for. So what happens if pθ(y∣x)p_\theta(y|x) is high, while pT(y∣x)p_T(y|x) is low? Well, in this case, the logarithm would become very large, which would increase the loss. But since the expectation is taken with respect to the student distribution, the student has a way of cheating in this case: it can just decide to put zero probability mass in the regions where the teacher's distribution has low probability (or in regions it hasn't explored yet). Of course, this means it needs to add more mass elsewhere, which results in it adding mass to other modes, which isn't ideal. However, it's still far better than it adding mass to areas where the teacher distribution has a low probability.

To illustrate this, assume that we have a set of three values x∈{1,2,3}x \in \{1, 2, 3\} and pT(1)=1/2p_T(1) = 1/2 and pT(3)=1/2p_T(3) = 1/2. The Reverse KL Divergence is given by:

∑x=13pθ(x)log⁡(pθ(x)pT(x))\sum_{x=1}^3 p_\theta(x) \log\left(\frac{p_\theta(x)}{p_T(x)}\right)

If we now assume that pθ(x)=1/3p_\theta(x)=1/3 for all values, then the KL divergence is essentially infinite, since we have a term 1/3log⁡((1/3)/0)1/3 \log((1/3)/0) for x=2x=2 (I'm an engineer and use infinity liberally, as my fellow mathematicians cringe). To avoid this cost, we can move all the probability mass that was at x=2x=2 to x=1x=1, which would make the loss:

2/3log⁡((2/3)/(1/2))+1/3log⁡((1/3)/(1/2))=0.0562/3 \log((2/3)/(1/2)) + 1/3 \log((1/3)/(1/2)) = 0.056

Of course, the optimal strategy would be to shift half of that mass to x=1x=1 and half to x=3x=3, like the teacher has, which would result in a loss of 0. But, when optimizing, it's very possible that we hit a local optimum, which is then hard to escape.

Capacity is only half the problem

In many other blogs I've read on this topic, they seem to illustrate this mode-seeking and mode-covering behaviour by focusing on a teacher that has a bimodal distribution and a student that has a unimodal distribution, showing that the student needs to make a trade-off. However, this isn't a necessary condition. This behaviour can arise purely because the student model is stuck in a local optimum i.e. it has collapsed both of its modes onto a single teacher mode and it's unable to escape it. Even if the student is the same size and architecture as the teacher, random weight initialisation can result in the model entering local optima, which results in it not easily being able to escape. This is illustrated in the small widget below.

In this case, both the teacher and the student have the exact same (toy) capacity. They are both a mixture of two Gaussians, with equal weight for each Gaussian and a standard deviation of 1. The only free parameters for each of them are the means of the Gaussians. In the demo below, we can see that if the student means are both initialised close to a single mode of the teacher (~-5.0), and the teacher modes are far away from each other (red distribution), then it's very difficult for the student to move out of the local optimum if optimising for the Reverse KL Divergence, since if it changes the means by any small value, it will end up with a higher KL divergence. Try this in the demo below by moving the sliders by a small amount e.g. 0.2 in either direction and seeing how it affects the Reverse KL Divergence. On the other hand, we can see that when we move either of the means closer to the other teacher mode (5.0), the Forward KL Divergence continues to decrease, until it reaches 5.0 and the two distributions become identical. The Reverse KL Divergence continues to increase until the mean reaches 0 and then starts to decrease again until the distributions become the same again. At that point, both the Forward and Reverse KL Divergences are equal to 0.

Live KL Divergence Simulator (Target Modes at -5 and 5)

Forward KL: D_KL(P || Q)
24.0914
Backward KL: D_KL(Q || P)
0.6932
Parameterised Mean 1 (μ₁)-5.10
Parameterised Mean 2 (μ₂)-4.90
Target P(x)
Parameterised Q(x)

So, why SFT and then RL?

Now we can start to understand why SFT followed by RL is the right way forward. If you go back to the widget above and move one of the means of the student so that it lies completely between the two modes of the teacher, you will see that the Reverse KL Divergence is huge, but the Forward KL Divergence improves. The closer the mean gets to the other mode that isn't covered, the lower the Forward KL Divergence becomes, which is how we might imagine gradient descent on the loss might progress until:

  • By chance, we end up with both modes being covered optimally by the student distribution or
  • All modes of the teacher are covered by the student distribution, but suboptimally

In the latter case, the important part is that the student distribution now allows the student to sample from all modes of the teacher, even if not at the right frequencies, and even if it occasionally samples from a part of the space the teacher would never sample from. With this property, RL can be used to refine the distribution and hopefully guide the student distribution to lock onto all the modes of the teacher distribution, whilst rejecting any part of the distribution that the teacher wouldn't sample from.

If we had started with RL directly, we may never have been able to teach the student how to learn new capabilities (represented by modes in the teacher distribution). By doing SFT first, we enable the student to learn new behaviours, and then by doing RL, we un-teach it the bad behaviours it might have learned along the way.

Summary

In short, SFT and RL play complementary roles in post-training:

  • SFT (Forward KL) is about exploration and coverage. By sampling from the teacher, it pulls the student's probability mass across the empty valleys so that every valid behaviour is at least reachable.
  • RL (Reverse KL) is about sharpening and precision. Once the student can sample the right behaviours, on-policy RL prunes away the mistakes and locks onto the modes the teacher actually favours.

Skipping SFT works fine if your base model already exhibits the desired behaviour occasionally. RL simply amplifies what is already there. However, when you are distilling a genuinely new capability into a smaller model, SFT is what gets you out of the local optimum in the first place.