Helpful context:

RLHF works. It produced ChatGPT, Claude, and every other major assistant-style language model. But it requires training four separate models simultaneously - policy, reference, reward, and value function - then running PPO, which is notoriously unstable and sensitive to hyperparameters. Practitioners often spend more time debugging training dynamics than improving the underlying model.

DPO (Rafailov et al., 2023) showed that you can achieve the same result as RLHF without a reward model and without any RL at all. Just a single supervised loss on paired preferences. It’s simpler, cheaper, and in many benchmarks, better.

The mathematical insight is beautiful: the optimal policy under the RLHF objective is already implicitly defined by the training data. You don’t need to find it iteratively through RL - you can write down a loss function whose solution is exactly that optimal policy, and optimize it directly.

Recalling the RLHF Objective

In RLHF, we train the policy $\pi_\theta$ to maximize:

$$\mathcal{J}(\theta) = \mathbb{E}{x \sim \mathcal{D}, y \sim \pi\theta(\cdot|x)}\left[R(x,y) - \beta\text{KL}\left(\pi_\theta(\cdot|x)|\pi_{\text{ref}}(\cdot|x)\right)\right].$$

The reward $R(x,y)$ measures response quality. The KL penalty keeps the trained policy from drifting too far from the reference policy $\pi_{\text{ref}}$ (typically the SFT model). The parameter $\beta$ controls the tradeoff.

This objective has been studied in the context of entropy-regularized RL. And crucially, it has a closed-form optimal solution (Ziebart et al., 2008):

$$\pi^*(y \mid x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y \mid x)\cdot \exp\left(\frac{R(x,y)}{\beta}\right),$$

where $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x)\exp(R(x,y)/\beta)$ is a normalizing partition function. This is the Boltzmann distribution of responses, weighted by the reference policy and exponentiated reward.

You can verify this: the optimal policy places higher probability on responses with high reward, but its shape stays close to $\pi_{\text{ref}}$ (controlled by $\beta$). Large $\beta$: stays near reference. Small $\beta$: concentrates mass on high-reward responses.

RLHF searches for this optimal policy iteratively, using the reward model and PPO. DPO derives a direct loss function that achieves it in one shot.

The Key Rearrangement

Starting from the closed-form optimal policy, we can solve for the reward:

$$\pi^*(y \mid x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y \mid x)\cdot \exp\left(\frac{R(x,y)}{\beta}\right).$$

Taking logs:

$$\log \pi^*(y \mid x) = \log \pi_{\text{ref}}(y \mid x) + \frac{R(x,y)}{\beta} - \log Z(x).$$

Rearranging for $R(x, y)$:

$$R(x,y) = \beta\log\frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta\log Z(x).$$

This is remarkable. The reward is completely determined by the ratio of the optimal policy to the reference policy. If you know $\pi^*$, you know $R$ - up to the partition function $Z(x)$, which depends only on $x$ and not on the specific response $y$.

The DPO Loss: Deriving It

Now plug this implicit reward into the Bradley-Terry preference model from RLHF Stage 2:

$$P(y_w \succ y_l \mid x) = \sigma\left(R(x, y_w) - R(x, y_l)\right).$$

Substituting:

$$R(x, y_w) - R(x, y_l) = \beta\log\frac{\pi^(y_w|x)}{\pi_{\text{ref}}(y_w|x)} + \beta\log Z(x) - \beta\log\frac{\pi^(y_l|x)}{\pi_{\text{ref}}(y_l|x)} - \beta\log Z(x).$$

The $\beta\log Z(x)$ terms cancel (this is the key trick - the partition function drops out):

$$R(x, y_w) - R(x, y_l) = \beta\log\frac{\pi^(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi^(y_l|x)}{\pi_{\text{ref}}(y_l|x)}.$$

So:

$$P(y_w \succ y_l \mid x) = \sigma\left(\beta\log\frac{\pi^(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi^(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right).$$

Now replace $\pi^*$ with the parameterized policy $\pi_\theta$ we’re training. The DPO loss is simply the negative log-likelihood of the correct preference:

$$\mathcal{L}{\text{DPO}}(\theta) = -\mathbb{E}{(x, y_w, y_l) \sim \mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right].$$

That’s it. Binary cross-entropy over preference pairs, with the policy’s log-probability ratios as the scores. No reward model. No PPO. One loss function.

Discomfort check. The derivation substituted $\pi^$ (the optimal policy, which we’re trying to find) with $\pi_\theta$ (the policy we’re training). This looks circular - we’re using $\pi^$ in the derivation but then replacing it with $\pi_\theta$. The resolution: the DPO loss is constructed so that its minimizer is exactly $\pi^$. We’re not assuming we already have $\pi^$; we’re building a loss whose minimum achieves $\pi^$. Gradient descent on this loss moves $\pi_\theta$ toward $\pi^$ without needing to know $\pi^*$ in advance.

What This Means in Practice

The DPO training procedure is almost identical to ordinary supervised fine-tuning, plus one extra model for reference probabilities.

For each preference pair $(x, y_w, y_l)$ in your dataset:

  1. Run the prompt $x$ and both responses through the policy model $\pi_\theta$. Record $\log\pi_\theta(y_w \mid x)$ and $\log\pi_\theta(y_l \mid x)$.
  2. Run them through the reference model $\pi_{\text{ref}}$ (frozen). Record $\log\pi_{\text{ref}}(y_w \mid x)$ and $\log\pi_{\text{ref}}(y_l \mid x)$.
  3. Compute the DPO loss. Backpropagate through the policy model only ($\pi_{\text{ref}}$ is frozen).

Memory footprint: two models (policy + reference). Compare to RLHF: four models (policy + reference + reward + value function). Training stability: no RL loop, no reward hacking, no clip ratios to tune.

For a 7B model in FP16, DPO needs roughly 28 GB for two model copies, plus gradients and optimizer states for the policy. With LoRA on the policy, this fits comfortably on a single A100-80GB. RLHF on the same model, running PPO with four model copies, requires careful memory engineering even on multiple GPUs.

Understanding the Gradient

What does optimizing the DPO loss actually do to the policy?

Let $h_\theta = \beta\log(\pi_\theta(y_w|x)/\pi_{\text{ref}}(y_w|x)) - \beta\log(\pi_\theta(y_l|x)/\pi_{\text{ref}}(y_l|x))$ be the “margin” between preferred and rejected responses. The loss is $-\log\sigma(h_\theta)$.

The gradient with respect to $\theta$ has an interpretable form (dropping constants):

$$\nabla_\theta \mathcal{L}{\text{DPO}} \propto -\sigma(-h\theta)\left[\nabla_\theta\log\pi_\theta(y_w|x) - \nabla_\theta\log\pi_\theta(y_l|x)\right].$$

The factor $\sigma(-h_\theta)$ is large when the model currently rates $y_l$ above $y_w$ (the preference is “wrong”), and small when the model already correctly prefers $y_w$. This is an implicit weighting: the loss focuses updates on the pairs the model gets wrong, and de-emphasizes pairs it already gets right.

The term in brackets increases the log-probability of the preferred response and decreases the log-probability of the rejected response, simultaneously. This is the key behavior: DPO is pushing up on $y_w$ and pulling down on $y_l$, relative to the reference policy - not in absolute terms but as a ratio.

The Role of $\beta$

The hyperparameter $\beta$ appears in the DPO loss in the same role as in the RLHF objective: it controls how aggressively the policy diverges from the reference.

  • Small $\beta$ (e.g., 0.01): strong optimization pressure, large updates, policy drifts far from reference. Risk of overfitting to the preference data and losing general capabilities.
  • Large $\beta$ (e.g., 1.0): conservative updates, policy stays close to reference. May not fully optimize preferences.

Typical values: $\beta = 0.1$ to $\beta = 0.5$. The choice depends on the quality of the preference data and how different the aligned behavior needs to be from the reference.

Data Requirements

DPO requires a dataset of preference triplets: $(x, y_w, y_l)$ - a prompt and two responses, one preferred. Several sources work:

  • Human-labeled preference data (same as RLHF Stage 2 data). DPO can directly reuse it.
  • AI-labeled preferences (a stronger model like GPT-4 ranks outputs of a weaker model).
  • Self-generated pairs (the model generates two responses; filter by known criteria like length, safety classifiers, or factual verification).

The quality of the preference data dominates DPO performance. Noisy labels (inconsistent human ratings, ambiguous preferences) degrade the result more than they would degrade a reward model (which can average over noise). This is one area where RLHF with a trained reward model has an advantage: the RM learns a smooth function over the preference space, implicitly averaging out label noise.

DPO Variants

The simplicity of DPO has spawned a family of variants addressing specific limitations:

IPO (Azar et al., 2023): Identity Preference Optimization. Modifies the DPO loss to prevent overfitting to the preference data. DPO’s loss approaches zero as the margin $h_\theta \to \infty$, meaning the gradient vanishes and training effectively stops before the policy is fully optimized. IPO adds a regularization term to prevent this:

$$\mathcal{L}{\text{IPO}} = \mathbb{E}\left[\left(h\theta - \frac{1}{2\beta}\right)^2\right].$$

SimPO (Meng et al., 2024): Simple Preference Optimization. Removes the reference model entirely. Instead of using $\pi_{\text{ref}}$ for regularization, SimPO normalizes log-probabilities by sequence length and adds a margin:

$$\mathcal{L}{\text{SimPO}} = -\log\sigma\left(\frac{\beta}{|y_w|}\log\pi\theta(y_w|x) - \frac{\beta}{|y_l|}\log\pi_\theta(y_l|x) - \gamma\right).$$

This eliminates the reference model from memory, halving the inference cost during training.

ORPO (Hong et al., 2024): Odds Ratio Preference Optimization. Combines the SFT loss and preference optimization into a single objective. The model learns to follow instructions and to prefer good responses simultaneously, without requiring a separate SFT stage:

$$\mathcal{L}{\text{ORPO}} = \mathcal{L}{\text{SFT}} - \lambda\mathbb{E}\left[\log\sigma\left(\log\frac{\pi_\theta(y_w|x)}{1-\pi_\theta(y_w|x)} - \log\frac{\pi_\theta(y_l|x)}{1-\pi_\theta(y_l|x)}\right)\right].$$

Practical Impact

DPO is currently the dominant method for open-source LLM alignment. Its simplicity means anyone with a GPU, a dataset, and a few hundred lines of code can align a language model.

Notable aligned models trained with DPO or close variants:

  • Zephyr-7B (HuggingFace): DPO on top of Mistral-7B, using AI-labeled preference data from GPT-4. Competitive with models 10× larger.
  • LLaMA-2-Chat: Meta used a combination of RLHF and DPO-style techniques.
  • Mistral-7B-Instruct: DPO fine-tuned for instruction following.

The pattern is consistent: a relatively small model (7B - 13B parameters) + DPO on high-quality preference data + open-source release. This democratized aligned LLM development in a way that full RLHF never did.

Discomfort check. DPO isn’t always better than RLHF in practice. The approximation that the reference policy is fixed throughout training can hurt if the policy moves far from the reference early in training (making the log-ratio terms unreliable). For very large-scale training, where the extra compute for PPO is affordable and reward hacking can be carefully managed, RLHF with PPO may still be preferred. Research by Anthropic and others suggests that online methods - where the model generates fresh responses during training rather than learning from a fixed dataset - often outperform offline DPO. The field is still actively evolving.

Summary

Concept Formula / Explanation
RLHF objective $\mathbb{E}[R(x,y)] - \beta\text{KL}(\pi | \pi_{\text{ref}})$
Optimal RLHF policy $\pi^*(y
Implicit reward $R(x,y) = \beta\log(\pi^*/\pi_{\text{ref}}) + \beta\log Z(x)$
Key insight $Z(x)$ cancels in the preference probability
DPO loss $-\log\sigma(\beta\log\frac{\pi_\theta(y_w
vs. RLHF No reward model, no PPO, no value function; 2 models instead of 4
$\beta$ Controls divergence from reference; typical values 0.1 - 0.5
Gradient Upweights $y_w$ and downweights $y_l$ relative to reference; weighted by how wrong the model currently is
IPO Prevents vanishing gradient in DPO by adding regularization
SimPO Removes reference model entirely
ORPO Combines SFT and preference optimization

DPO turns a reinforcement learning problem into a supervised learning problem by exploiting the closed-form solution to the RLHF objective. The partition function $Z(x)$ cancels, eliminating the hardest computational piece. What remains is a clean, stable, two-model training procedure.

Read next: