The famous InstructGPT result is still the cleanest argument for post-training: a 1.3B aligned model was preferred over the 175B GPT-3 base ~85% of the time on instruction-following. Alignment beat a 100x scale gap.
That number got a lot of people to implement RLHF. Most of them later ripped it out and switched to DPO. A smaller group skipped both and went to verifier-based RL.
This post is the decision tree I wish I'd had when I started: what each pipeline actually looks like in TRL, where it breaks, and which one you should reach for first in 2026. The code blocks are runnable end-to-end against open weights — pick one and you have a working stack by tomorrow.
The three-way choice
Before any code, the picture:
PPO RLHF — sample, score with a reward model, update with PPO under a KL leash. The original InstructGPT recipe. Powerful, fiddly, expensive.
DPO — collapse the reward model and the RL loop into a single supervised loss on preference pairs. Trains like SFT, no sampling loop.
RLVR — verifier-based RL. The reward is ground truth (unit tests pass, math answer is correct, JSON parses). No human preferences at all.
A rough rule that holds in most post-training shops I've talked to:
- Style, tone, instruction-following → DPO by default, PPO only if you can afford on-policy sampling.
- Math, code, structured output, tool-use → RLVR. Don't waste a reward model on something a checker can score.
- Mixed product behavior → SFT first, then DPO, then a verifier-RL pass on the verifiable slices.
The rest of this post is the why behind that table, and the actual training code.
SFT first, always
Every pipeline below assumes you've done SFT. The SFT model is both the starting policy for the RL/DPO step and the frozen reference the KL term anchors against.
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
from transformers import AutoTokenizer
MODEL = "Qwen/Qwen2.5-0.5B"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft").select(range(5000))
trainer = SFTTrainer(
model=MODEL,
train_dataset=ds,
args=SFTConfig(
output_dir="qwen-sft",
per_device_train_batch_size=4,
learning_rate=2e-5,
num_train_epochs=1,
bf16=True,
),
tokenizer=tokenizer,
)
trainer.train()
SFT teaches the model to imitate a fixed target. It runs out of road the moment "good" isn't a single sentence away — helpfulness, tone, "did you actually answer the question" are comparative judgments, not next-token predictions. That's the whole reason the other stages exist.
Path A: classical PPO RLHF
Step 1 — train the reward model
The reward model (RM) is a scalar head on top of a transformer: (prompt, response) → r. You train it on pairwise comparisons with the Bradley-Terry loss:
L = -log σ(r(x, y_chosen) - r(x, y_rejected))
Translation: push the score of the chosen response above the rejected one, by enough margin that softmax probabilities match human preferences.
from trl import RewardTrainer, RewardConfig
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from datasets import load_dataset
RM_BASE = "Qwen/Qwen2.5-0.5B"
tokenizer = AutoTokenizer.from_pretrained(RM_BASE)
model = AutoModelForSequenceClassification.from_pretrained(RM_BASE, num_labels=1)
ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train").select(range(10000))
trainer = RewardTrainer(
model=model,
args=RewardConfig(
output_dir="qwen-rm",
per_device_train_batch_size=8,
learning_rate=1e-5,
num_train_epochs=1,
bf16=True,
max_length=1024,
),
train_dataset=ds,
tokenizer=tokenizer,
)
trainer.train()
Warning: the RM overfits fast. Track validation pairwise accuracy, not training loss. If train accuracy keeps climbing while eval plateaus around 0.65–0.70, stop training. A slightly underfit RM is far better than a sharp one — sharp RMs are the easiest to exploit.
OpenAI used a 6B RM against a 175B policy. The RM doesn't need to be as big as the policy; it just needs to be a stable judge.
Step 2 — PPO with a KL penalty
PPO samples completions from the current policy, scores them with the RM, and updates the policy with clipped policy-gradient. The KL penalty is what keeps the run from imploding:
r_total = r_RM(x, y) - β · KL(π_θ(·|x) || π_ref(·|x))
Drop the KL term and the policy walks off the manifold the RM was trained on, finds a strange region of token space that scores high, and produces nonsense. With KL, every step is leashed to the SFT reference.
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from transformers import AutoModelForSequenceClassification, AutoTokenizer
policy = AutoModelForCausalLMWithValueHead.from_pretrained("qwen-sft")
ref = AutoModelForCausalLMWithValueHead.from_pretrained("qwen-sft") # frozen
rm = AutoModelForSequenceClassification.from_pretrained("qwen-rm")
tokenizer = AutoTokenizer.from_pretrained("qwen-sft")
config = PPOConfig(
output_dir="qwen-ppo",
learning_rate=1e-6,
per_device_train_batch_size=4,
mini_batch_size=2,
num_ppo_epochs=4,
kl_coef=0.05, # β — start here
cliprange=0.2,
cliprange_value=0.2,
bf16=True,
)
trainer = PPOTrainer(
args=config,
model=policy,
ref_model=ref,
reward_model=rm,
train_dataset=prompt_dataset,
tokenizer=tokenizer,
)
trainer.train()
Three dashboards to keep open:
Mean reward — should rise, then plateau. If it keeps climbing past your RM's eval accuracy ceiling, the policy is hacking the RM.
KL to reference — should stay bounded. A spike means the policy is sprinting away from SFT. Raisekl_coef.
A separate judge on held-out prompts — never trust the RM as ground truth. Read samples, or score with a different model entirely.
kl_coef between 0.02 and 0.2 covers most cases. I start at 0.05 and only move it when the KL graph misbehaves.
Why PPO breaks
After a few runs the failure modes get predictable:
Reward hacking — the policy finds outputs the RM loves and humans don't. Karpathy's line that RLHF is . Canonical reference for the full pipeline.- DPO paper — arxiv.org/abs/2305.18290. Short, worth reading in full.
SOCIAL SHARE CARD GENERATOR