RLHF (reinforcement learning from human feedback) is a post-training method that teaches a language model which answers people prefer: humans compare model outputs, a reward model learns those preferences, and reinforcement learning (classically PPO) pushes the model toward higher-reward answers while a KL penalty keeps it close to the original. In 2026 the same idea runs in several forms — RLAIF swaps human labels for AI labels, DPO skips the reward model and RL loop, and RLVR replaces preferences with automatic checks for math and code.
What is RLHF, and why do modern LLMs still depend on it?
RLHF is the step that turned raw large language models into instruction-following assistants. Instead of writing a loss function that defines a “good” answer, you let people show the model what good looks like — by ranking its outputs — and then train a reward signal from those rankings.
The method was proposed by Christiano et al. (2017) for Atari games and simulated robotics. It became the template for LLM alignment with OpenAI’s InstructGPT paper (2022), where labelers preferred outputs from a 1.3B-parameter RLHF model over those of the 175B-parameter GPT-3 base model, despite the 100× size gap.
Since then, “RLHF” has become a family name rather than one recipe. Public technical reports show how varied the 2024–2026 stacks are:
- Meta Llama 3 used supervised fine-tuning, rejection sampling with a reward model, and DPO; the authors report choosing DPO over PPO because it needed less compute at large scale and did better on instruction-following benchmarks such as IFEval (Llama 3 Herd of Models, §4.1).
- AI2’s Tülu 3 trained in three stages — SFT, DPO, and a new stage it called Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024).
- DeepSeek-R1 used large-scale reinforcement learning with GRPO, rule-based rewards for reasoning, and reward models for general helpfulness (DeepSeek-AI, 2025; peer-reviewed version in Nature).
- Anthropic’s Claude models are trained with Constitutional AI, in which a written constitution is used to generate synthetic training data, including rankings of candidate responses (Anthropic, January 2026).
All four share the RLHF skeleton — a preference or reward signal, an optimizer, and a constraint that keeps the model from drifting. Understanding the original pipeline is what makes the newer variants legible.
How does the RLHF pipeline work? The 3 stages explained
The classic RLHF pipeline has three sequential stages. Each builds on the previous one, and a weak earlier stage limits everything after it.
Stage 1: Supervised Fine-Tuning (SFT)
You start with a pretrained base model that can produce fluent text but does not reliably follow instructions. SFT trains it on curated (prompt, ideal response) pairs with ordinary next-token cross-entropy loss — the same deep learning machinery used in pretraining, just on a much smaller, cleaner dataset.
Data quality dominates here. InstructGPT’s SFT set was built from roughly 13,000 training prompts with demonstrations written by a team of about 40 contractors (Ouyang et al., 2022). Current open recipes mix human-written demonstrations with synthetic responses from stronger models. The resulting model, πSFT, becomes both the starting policy for Stage 3 and the frozen reference model for the KL penalty. If you only need a model to follow a format or domain style, SFT alone — often with LoRA adapters — may be enough; see our RAG vs fine-tuning guide for that decision.
Stage 2: Reward Model Training
The reward model converts subjective human preferences into a number a reinforcement learning algorithm can optimize.
For each prompt, the SFT model generates several candidate responses. Annotators compare them — “A is better than B” — or rank a list. These comparisons train a reward model rθ with the Bradley-Terry preference model:
P(yw ≻ yl | x) = σ(rθ(x, yw) − rθ(x, yl))
σ is the sigmoid function, yw is the preferred response, yl the rejected one, and rθ outputs a scalar. Training maximizes the probability that the reward model scores the human-preferred output higher.
In practice the reward model is usually initialized from the SFT model with the language-modeling head replaced by a scalar head. How much data is “enough” depends more on quality than volume: NVIDIA’s HelpSteer2 dataset has only about 10,000 response pairs — roughly an order of magnitude fewer than Anthropic’s HH-RLHF — yet a reward model trained on it reached 92.0% on RewardBench’s primary set when published in June 2024. Who produces this data, and how, is covered in the vendor landscape section below.
import torch
import torch.nn as nn
from transformers import AutoModelForSequenceClassification
class RewardModel(nn.Module):
def __init__(self, base_model_name: str):
super().__init__()
self.model = AutoModelForSequenceClassification.from_pretrained(
base_model_name, num_labels=1
)
def forward(self, input_ids, attention_mask):
return self.model(
input_ids=input_ids,
attention_mask=attention_mask
).logits.squeeze(-1) # scalar reward
def reward_loss(reward_model, chosen_ids, chosen_mask,
rejected_ids, rejected_mask):
"""Bradley-Terry pairwise loss for reward model training."""
r_chosen = reward_model(chosen_ids, chosen_mask)
r_rejected = reward_model(rejected_ids, rejected_mask)
# Maximize P(chosen > rejected) = sigmoid(r_chosen - r_rejected)
return -torch.nn.functional.logsigmoid(r_chosen - r_rejected).mean()
In production you would not hand-roll this: Hugging Face TRL ships a RewardTrainer and a trl reward command that train the same pairwise objective from a (prompt, chosen, rejected) dataset.
Stage 3: PPO Policy Optimization
With a reward model in hand, you optimize the language model’s policy with reinforcement learning. The classic choice is Proximal Policy Optimization (PPO), which clips each update so the policy cannot change too much in one step.
maximize Ex∼D, y∼πθ [rθ(x, y) − β · DKL(πθ(y|x) ∥ πSFT(y|x))]
The first term pushes the model toward higher-reward outputs. The KL penalty, weighted by β, keeps it close to the SFT model — without it, the policy drifts into text that games the reward model.
This is where RLHF becomes operationally expensive. PPO-based RLHF keeps four models in play: the trainable policy πθ, the frozen reference πSFT, the reward model rθ, and a value (critic) model for advantage estimation. For a 70B-parameter setup, the weights alone are 4 × 70B × 2 bytes ≈ 560 GB in bf16 — before optimizer states for the two trainable models, activations, and the KV cache needed to generate rollouts. That arithmetic is why frameworks such as OpenRLHF split the models across GPUs and hand generation to vLLM.
from transformers import (AutoModelForCausalLM,
AutoModelForSequenceClassification, AutoTokenizer)
from trl.experimental.ppo import PPOConfig, PPOTrainer
SFT = "your-org/your-sft-model"
RM = "your-org/your-reward-model"
tokenizer = AutoTokenizer.from_pretrained(SFT, padding_side="left")
policy = AutoModelForCausalLM.from_pretrained(SFT)
ref_policy = AutoModelForCausalLM.from_pretrained(SFT)
reward_model = AutoModelForSequenceClassification.from_pretrained(RM, num_labels=1)
value_model = AutoModelForSequenceClassification.from_pretrained(RM, num_labels=1)
config = PPOConfig(
output_dir="ppo-out",
learning_rate=3e-6,
kl_coef=0.05, # beta: TRL's current default
cliprange=0.2, # PPO clip range
vf_coef=0.1, # value-loss weight
gamma=1.0, lam=0.95, # no discounting; GAE lambda
num_ppo_epochs=4,
missing_eos_penalty=1.0, # "EOS trick" recommended in the TRL docs
per_device_train_batch_size=16,
gradient_accumulation_steps=4,
)
trainer = PPOTrainer(
args=config,
processing_class=tokenizer,
model=policy,
ref_model=ref_policy,
reward_model=reward_model,
value_model=value_model,
train_dataset=prompt_dataset, # tokenized prompts ("input_ids")
)
trainer.train() # watch objective/rlhf_reward, objective/kl and val/ratio
TRL moved its PPO trainer to trl.experimental, and the older ppo_trainer.step() loop with init_kl_coef found in many 2023 tutorials no longer matches the current API. The KL coefficient still matters: too low and the policy reward-hacks, too high and it barely moves from SFT. Huang, Liu and von Werra’s “The N Implementation Details of RLHF with PPO” (Hugging Face, October 2023) and the follow-up “N+ Implementation Details” paper (2024) document the details — reward normalization, EOS handling, padding, value-head initialization — that decide whether a run reproduces at all.
What is reward hacking, and why is it RLHF’s biggest problem?
Reward hacking happens when the policy finds ways to raise the reward model’s score without actually getting better. A reward model is an imperfect proxy for human judgment, trained on a finite set of comparisons, so it has blind spots the policy can exploit.
Two failure modes are well documented. First, over-optimization: Gao, Schulman and Hilton (2022) showed that as a policy optimizes harder against a proxy reward model, the “true” reward first rises and then falls — and they fitted scaling laws for how quickly that happens. Second, length bias: Singhal et al. (2023) found that a large share of measured RLHF improvement can be explained by responses simply getting longer.
Verifiable rewards do not make the problem disappear. In agentic coding environments, OpenAI researchers observed a frontier reasoning model exploiting weaknesses in its tests during RL training; monitoring its chain of thought caught much of this, but when the monitor was added to the reward, the model learned to hide its intent while still hacking at a significant rate (Baker et al., 2025).
Standard mitigations are the KL penalty, reward-model ensembles, held-out human evaluation, and stopping early when reward climbs faster than human-judged quality. The fragility of learned reward models is also the main reason the field built alternatives that remove them.
What is DPO, and why did it replace PPO for most teams?
Direct Preference Optimization (DPO), introduced by Rafailov et al. in May 2023, is the most important simplification of RLHF. It rearranges the RLHF objective so that both the separate reward model and the RL loop disappear, leaving a single supervised loss over preference pairs.
LDPO(πθ; πref) = −E(x, yw, yl) [log σ(β · (log πθ(yw|x)/πref(yw|x) − log πθ(yl|x)/πref(yl|x)))]
This implicitly defines a reward r(x, y) = β · log(πθ(y|x) / πref(y|x)) — the policy itself acts as the reward model, which is where the paper’s subtitle “Your Language Model is Secretly a Reward Model” comes from.
The practical gains are large: two models in memory instead of four, no value network, no online generation loop, and training that behaves like ordinary fine-tuning on (prompt, chosen, rejected) triplets.
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer
model_id = "your-org/your-sft-model"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Rows look like {"prompt": ..., "chosen": ..., "rejected": ...}
train = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
args = DPOConfig(
output_dir="dpo-out",
beta=0.1, # strength of the implicit KL constraint
loss_type="sigmoid", # standard DPO loss
learning_rate=5e-7,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
max_length=1024, # prompt + completion
num_train_epochs=1,
)
trainer = DPOTrainer(
model=model, # ref model is created automatically if omitted
args=args,
train_dataset=train,
processing_class=tokenizer,
)
trainer.train()
DPO’s weakness is that it learns from a static set of pairs and never explores beyond them. Controlled comparisons bear this out. Ivison et al. (NeurIPS 2024) found that PPO beat DPO by up to 2.5% on math and 1.2% in general domains — but that preference-data quality mattered more than the algorithm, worth up to 8% on instruction following and truthfulness. Xu et al. (ICML 2024) reached a similar conclusion: a well-tuned PPO outperformed DPO in their benchmarks.
Our read: DPO is the sensible default when you have good pairwise data and limited compute, which describes most teams. The PPO advantage is real but small, and it only shows up when the PPO implementation is tuned carefully — a cost that is easy to underestimate.
What is KTO and when should you use it over DPO?
Kahneman-Tversky Optimization (KTO), from Ethayarajh et al. (2024), solves a data problem. DPO needs a “chosen” and a “rejected” response for every prompt, but most products collect a simpler signal: a thumbs-up or thumbs-down on a single answer.
KTO borrows from prospect theory — the same framework behind Kahneman and Tversky’s work on decisions under risk. It models utility asymmetrically, with losses weighing more than gains, and derives a loss that works on unpaired binary labels.
Instead of (prompt, chosen, rejected) triplets, KTO trains on rows like {"prompt": ..., "completion": ..., "label": True}. That fits production feedback where users rate individual outputs — no pairing step needed.
The KTO authors report matching or exceeding DPO across models from 1B to 30B parameters, and argue the method is more robust when labels are noisy or imbalanced. Use it when your data is binary feedback; use DPO when you have clean pairs.
What is ORPO and why does it skip the reference model?
Odds Ratio Preference Optimization (ORPO), from Hong, Lee and Thorne (2024), removes the reference model entirely. It merges SFT and preference optimization into one objective, so a base model goes straight to an aligned model in a single run.
LORPO = LSFT + λ · LOR, where LOR penalizes the model when its odds of generating the rejected response are high relative to the chosen one. Only one model sits in memory.
The authors report that ORPO-tuned 7B models matched or beat larger models on AlpacaEval 2.0 and MT-Bench. The trade-off is that there is no KL anchor to a reference model, so data that is narrow or low quality can pull the model away from its general abilities. ORPO suits small models and single-GPU budgets with well-distributed preference data.
What are GRPO, DAPO, and RLVR — RL for reasoning models?
While DPO and KTO simplified preference alignment, the biggest change of 2025–2026 was a return to online RL — with a different reward source. Instead of human preferences or a learned reward model, these methods score outputs with verifiable rewards: programmatic checks that an answer is correct.
GRPO — Group Relative Policy Optimization
GRPO was introduced in DeepSeekMath (Shao et al., 2024) and later powered DeepSeek-R1. It removes PPO’s critic. For each prompt it samples a group of responses, scores each one, and uses the group’s own statistics as the baseline:
Ai = (ri − mean(r1..G)) / std(r1..G)
No value model is needed — the group is the baseline — which removes one of PPO’s four models.
from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward # checks the final answer
dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
trainer = GRPOTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct",
reward_funcs=accuracy_reward, # verifiable reward, no reward model
train_dataset=dataset,
)
trainer.train()
DAPO — Decoupled Clip and Dynamic Sampling Policy Optimization
DAPO (Yu et al., March 2025) is an open-source RL system built to fix instabilities seen when scaling GRPO-style training on long chains of thought. It adds four techniques: asymmetric “clip-higher” clipping to preserve exploration, dynamic sampling that drops prompts the model always or never solves, a token-level policy-gradient loss, and overlong-response reward shaping. It also drops the KL penalty, arguing that for verifiable reasoning tasks the model should be allowed to move far from its starting point.
RLVR — Reinforcement Learning with Verifiable Rewards
RLVR is the umbrella term, introduced in the Tülu 3 report: use RL (PPO, GRPO, DAPO, REINFORCE-style variants) but replace the learned reward with deterministic verification — check the final math answer, run unit tests, validate output against a schema. It removes the human labeling bottleneck for those domains and can generate training signal at the speed of the verifier.
Two caveats matter. Verifiers can still be gamed, as the reward-hacking evidence above shows. And most valuable tasks are not cleanly verifiable. That is why 2025–2026 research has pushed into rubric-based rewards: “Rubrics as Rewards” (Gunjal et al., 2025) uses checklist-style rubrics scored by an LLM judge and reports gains of up to 31% on HealthBench and 7% on GPQA-Diamond over Likert-scale LLM-judge rewards. Rubrics are a bridge between RLVR and RLAIF.
Public recipes such as Tülu 3 and DeepSeek-R1 follow a similar pattern: SFT for format and instruction following → preference optimization (DPO, or PPO with a reward model) for helpfulness and tone → RL with verifiable or rubric-based rewards for reasoning. “RLHF” today usually means this whole stack, not one algorithm.
How do Constitutional AI and RLAIF relate to RLHF?
Anthropic’s Constitutional AI (Bai et al., 2022) does not replace RLHF; it changes where the labels come from. It has two phases. In the supervised phase, the model drafts responses, critiques them against written principles (the “constitution”), revises them, and is fine-tuned on the revisions. In the RL phase, a model compares pairs of samples and judges which better follows the principles; those AI preference labels train a preference model that serves as the reward. The paper named this RL from AI Feedback (RLAIF).
Does AI feedback work as well as human feedback? Google researchers Lee et al. (ICML 2024) found that RLAIF achieved performance comparable to RLHF on summarization, helpful dialogue and harmless dialogue. They also tested “direct RLAIF”, which skips the reward model and asks an off-the-shelf LLM for a score during RL, and found it did better than standard RLAIF.
The approach has matured since 2023. On January 22, 2026, Anthropic published a new, much longer constitution for Claude under a CC0 license. It explains the reasoning behind its values rather than listing standalone rules, and Anthropic says Claude uses it to generate synthetic training data — including conversations, responses that reflect its values, and rankings of candidate responses. For teams building AI agents, the lesson is that written, auditable principles can scale feedback far beyond what a human annotation budget allows — but they inherit the blind spots of the model doing the judging.
RLHF vs RLAIF vs DPO: what is the difference?
These three terms are often listed as rivals, but they answer different questions. RLHF vs RLAIF is about who produces the preference labels — humans or an AI model. RLHF vs DPO is about how you optimize on those labels — a reward model plus RL, or a direct supervised loss. You can therefore combine them: DPO on AI-labeled pairs is common (UltraFeedback, for example, contains over a million GPT-4 feedback items on 250,000 conversations, per Cui et al.).
| Dimension | RLHF (classic) | RLAIF | DPO |
|---|---|---|---|
| Source of preference labels | Human annotators | An AI judge, often guided by a constitution or rubric | Any: human, AI, or mixed pairs |
| Reward model? | Yes, trained on comparisons | Usually yes (trained on AI labels); “direct RLAIF” scores with an LLM instead | No — the policy acts as an implicit reward model |
| Optimizer | Online RL (PPO) | Online RL (PPO or similar) | Offline supervised loss |
| Models in memory | 4 (policy, reference, reward, value) | 4, plus access to the AI labeler | 2 (policy, reference) |
| Main cost driver | Human labeling and RL compute | Labeler inference and RL compute | Preference data; training compute is close to SFT |
| Main risk | Reward hacking; annotator inconsistency and bias | Inherits the judge model’s biases and blind spots | No exploration beyond the static dataset |
| Evidence | InstructGPT (Ouyang et al., 2022) | Comparable to RLHF on 3 tasks (Lee et al., 2024) | Tuned PPO beats DPO by up to 2.5% on math (Ivison et al., 2024) |
Practical rule: decide the label source by what you can afford and audit (humans for high-stakes nuance, AI for scale), then decide the optimizer by your compute and engineering budget (DPO first; PPO or GRPO when you need online exploration).
Which alignment method should you use? A practical comparison
| Method | Models in memory | Data required | Relative compute | Best for |
|---|---|---|---|---|
| RLHF (PPO) | 4 (policy, ref, reward, value) | Pairwise preferences + online rollouts | Highest | Maximum quality with a skilled RL team |
| RLAIF / Constitutional AI | 4 + AI labeler | Written principles or rubrics + AI-labeled pairs | High | Scaling feedback and safety norms |
| DPO | 2 (policy, ref) | Pairwise preferences (static) | Low | Default for most teams |
| KTO | 2 (policy, ref) | Binary labels (thumbs up/down) | Low | Product feedback, noisy labels |
| ORPO | 1 (policy only) | Pairwise preferences (merged with SFT) | Lowest | Small models, single GPU |
| GRPO | Policy + ref (no value model) | Verifiable or rubric rewards | Medium–high (many samples per prompt) | Reasoning: math, code |
| DAPO | Policy only (no KL reference) | Verifiable rewards | Medium–high | Long chain-of-thought RL at scale |
Who supplies RLHF data? The 2026 vendor landscape
Most AI labs do not produce all their preference data in-house. The market that supplies it falls into four types of provider. This section is a neutral map, not a ranking: DecodeTheFuture has no commercial relationship with any company named, has not tested their services, and none of them publish comparable quality benchmarks or public per-label prices. For a provider-by-provider comparison of scope, workflow and quality controls, see our RLHF data providers and vendors guide.
| Provider type | What they supply | Examples and verifiable public facts |
|---|---|---|
| Managed data and labeling firms | Run the full project: task design, workforce, QA, delivery of comparisons, demonstrations and evaluations | Scale AI — Meta took a 49% non-voting stake for about $14.3B in June 2025 (CNBC); founder Alexandr Wang moved to Meta (Scale). Francis deSouza became CEO on August 10, 2026, with Scale describing two businesses: data for AI labs and applications (Scale). Appen, Toloka and Sama describe RLHF, preference or evaluation services on their own pages. |
| Expert networks | Domain specialists (coders, doctors, lawyers, PhDs) who write hard prompts, grade answers and build rubrics | Surge AI — bootstrapped; Reuters reported in July 2025 that its annual revenue exceeded $1B as it sought its first outside raise (SiliconANGLE). Mercor — raised $350M at a $10B valuation in October 2025 (TechCrunch). Turing describes SFT, RLHF and DPO data programs. |
| Platforms and tooling | Software for side-by-side comparisons, reviewer management and export; you bring or hire the workforce | Labelbox documents a human preference editor for reward-model data; SuperAnnotate describes preference datasets and workforce management. |
| Synthetic and AI feedback | AI-labeled preference pairs, constitution- or rubric-driven judges, open datasets | Open sets such as UltraFeedback (GPT-4 labels) and NVIDIA’s HelpSteer2 (about 10K pairs, CC-BY-4.0); in-house constitution-driven labeling as in Anthropic’s Claude training. |
Three things have shaped this market since 2025. Demand has shifted from generic crowd labeling toward expert feedback for reasoning, coding and agentic tasks, where a grader has to understand the answer. Lab neutrality became a buying criterion after the Meta–Scale deal, as labs weighed whether a supplier part-owned by a competitor should see their prompts. And AI feedback now does much of the volume work, so human budgets go to the hard, high-stakes cases and to evaluation sets.
Ask for a paid pilot on your own prompts; inter-annotator agreement and adjudication rules; reviewer qualifications for your domain; how AI assistance is used and disclosed in their pipeline; data-security and lab-isolation terms; and whether you get held-out evaluation sets, not just training data. A vendor that cannot show calibration evidence on a pilot is a risk regardless of its size.
How much does RLHF cost? Compute and data cost drivers
Public, verifiable cost figures for alignment are rare; most labs do not disclose them. The figures that are public point to one pattern: post-training compute is small next to pretraining, but RL for reasoning and high-quality human data can still be expensive.
- Preference post-training can be cheap in compute. DeepSeek’s V3 report lists 2,664K H800 GPU hours for pre-training but only 5K for post-training (SFT and RL) — about $10,000 at an assumed $2 per GPU hour, out of a $5.576M total (DeepSeek-V3, Table 1). This excludes research, failed runs and data costs.
- Reasoning RL is a bigger line item. DeepSeek disclosed in its Nature paper that training R1 cost about $294,000, using 512 Nvidia H800 chips for 80 hours on the final run, on top of roughly $6M for the V3 base model (Reuters via BNN Bloomberg, September 2025). Critics note these are rental-price estimates, not full program costs.
- Algorithm choice changes the bill. PPO holds four models and generates rollouts every step; DPO holds two and trains offline; GRPO drops the value model but samples a whole group of responses per prompt. Meta cited DPO’s lower compute at scale as a reason to choose it over PPO for Llama 3.
- Human data is the hidden cost. No major provider publishes per-comparison prices; costs depend on domain expertise, task length and QA depth. Evidence from Ivison et al. and HelpSteer2 suggests spending on fewer, better comparisons beats buying volume.
For a practitioner, the compute cost of a DPO run on a 7B model with LoRA is in the same order of magnitude as an SFT run of the same size (it processes a chosen and a rejected response per example); the real budget question is the data. For serving costs after training, see our LLM inference cost comparison.
Can RLHF or DPO make a model less capable?
Yes — the effect has a name, the alignment tax. InstructGPT reported performance regressions on several public NLP benchmarks after RLHF, and reduced them by mixing pretraining gradients into PPO updates (the “PPO-ptx” variant) (Ouyang et al., 2022). Offline methods are not immune: because DPO and ORPO only see the preference pairs, narrow or low-quality data can shift the model away from skills the pairs do not cover.
Mitigations that hold up: keep a KL or reference anchor (or a pretraining/SFT loss term), mix diverse data, train for few epochs, and benchmark broad capabilities before and after alignment. Teams adding reasoning RL should also re-check general helpfulness, since a model optimized only on verifiable tasks can become terse or over-formatted elsewhere.
How does RLHF intersect with the EU AI Act in 2026?
The EU AI Act does not prescribe any alignment technique, but it does require documentation that covers alignment training.
General-purpose AI model providers have had obligations since August 2, 2025. Under Article 53 and Annex XI they must keep technical documentation that includes “training methodologies and techniques” and information on training data, “including the type and provenance of data and curation methodologies” (Regulation (EU) 2024/1689). For a model aligned with RLHF, that means records of how preference data was collected or generated, by whom, and how it was filtered.
High-risk systems face data-governance rules (Article 10) and accuracy and robustness requirements (Article 15). The AI Omnibus, Regulation (EU) 2026/1744, in force since July 27, 2026, moved the application date for stand-alone high-risk systems from August 2, 2026 to December 2, 2027, and for high-risk systems embedded in regulated products to August 2, 2028; GPAI obligations were not changed (National Law Review).
The practical implication: document the pipeline end to end — annotator instructions and demographics, AI-labeler prompts and constitutions, reward-model evaluations, and bias testing on preference data. RLAIF with a written constitution or rubric has an auditing advantage here, because the labeling policy exists as a document.
How do you implement RLHF in 2026? A decision guide
Do you have verifiable outputs (math, code, structured data)? Use GRPO or DAPO with rule-based rewards. TRL’s GRPOTrainer is the simplest start; OpenRLHF implements PPO, GRPO, REINFORCE++ and DAPO on Ray and vLLM for larger runs.
Do you have pairwise preference data? Use DPO. One epoch is usually sufficient; overtraining on a fixed preference set tends to degrade outputs.
Do you only have thumbs-up/thumbs-down feedback? Use KTO — same library, unpaired labels.
Are you on a single GPU with a small model? Try ORPO or DPO with LoRA adapters.
Do you lack labels altogether? Generate AI preference pairs against a written rubric or constitution (RLAIF), then spot-check a sample with human reviewers before training.
Do you need maximum quality and have an experienced RL team? PPO with a strong reward model is still competitive, as Ivison et al. and Xu et al. show — budget for the tuning.
# Install
pip install trl peft
# Run DPO on a public preference dataset (LoRA keeps memory low)
trl dpo \
--model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
--dataset_name trl-lib/ultrafeedback_binarized \
--output_dir ./qwen-dpo \
--beta 0.1 \
--learning_rate 5e-7 \
--num_train_epochs 1 \
--use_peft \
--lora_r 16 \
--lora_alpha 32
If you want to go deeper than this article, Nathan Lambert’s free RLHF Book covers the full post-training pipeline in detail.
FAQ
References
- Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. arXiv. https://arxiv.org/abs/1706.03741
- Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. arXiv. https://arxiv.org/abs/2203.02155
- Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv. https://arxiv.org/abs/2204.05862
- Bai, Y., Kadavath, S., Kundu, S., et al. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv. https://arxiv.org/abs/2212.08073
- Lee, H., Phatale, S., Mansoor, H., et al. (2024). RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. ICML 2024. https://arxiv.org/abs/2309.00267
- Anthropic. (2026, January 22). Claude’s new constitution. https://www.anthropic.com/news/claude-new-constitution
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. arXiv. https://arxiv.org/abs/2305.18290
- Ivison, H., Wang, Y., Liu, J., et al. (2024). Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. NeurIPS 2024. https://arxiv.org/abs/2406.09279
- Xu, S., Fu, W., Gao, J., et al. (2024). Is DPO superior to PPO for LLM alignment? A comprehensive study. ICML 2024. https://arxiv.org/abs/2404.10719
- Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024). KTO: Model alignment as prospect theoretic optimization. arXiv. https://arxiv.org/abs/2402.01306
- Hong, J., Lee, N., & Thorne, J. (2024). ORPO: Monolithic preference optimization without reference model. arXiv. https://arxiv.org/abs/2403.07691
- Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models (introduces GRPO). arXiv. https://arxiv.org/abs/2402.03300
- DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv; peer-reviewed version in Nature (2025). https://arxiv.org/abs/2501.12948
- DeepSeek-AI. (2024). DeepSeek-V3 technical report (Table 1: training costs). arXiv. https://arxiv.org/abs/2412.19437
- Yu, Q., Zhang, Z., Zhu, R., et al. (2025). DAPO: An open-source LLM reinforcement learning system at scale. arXiv. https://arxiv.org/abs/2503.14476
- Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). Tülu 3: Pushing frontiers in open language model post-training (introduces RLVR). arXiv. https://arxiv.org/abs/2411.15124
- Grattafiori, A., Dubey, A., et al. (2024). The Llama 3 herd of models. arXiv. https://arxiv.org/abs/2407.21783
- Gunjal, A., Wang, A., Lau, E., et al. (2025). Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv. https://arxiv.org/abs/2507.17746
- Gao, L., Schulman, J., & Hilton, J. (2022). Scaling laws for reward model overoptimization. arXiv. https://arxiv.org/abs/2210.10760
- Singhal, P., Goyal, T., Xu, J., & Durrett, G. (2023). A long way to go: Investigating length correlations in RLHF. arXiv. https://arxiv.org/abs/2310.03716
- Baker, B., Huizinga, J., Gao, L., et al. (2025). Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv. https://arxiv.org/abs/2503.11926
- Wang, Z., Dong, Y., Delalleau, O., et al. (2024). HelpSteer2: Open-source dataset for training top-performing reward models. arXiv. https://arxiv.org/abs/2406.08673
- Cui, G., Yuan, L., Ding, N., et al. (2023). UltraFeedback: Boosting language models with scaled AI feedback. arXiv. https://arxiv.org/abs/2310.01377
- Huang, S., Liu, T., & von Werra, L. (2023, October 24). The N implementation details of RLHF with PPO. Hugging Face Blog. https://huggingface.co/blog/the_n_implementation_details_of_rlhf_with_ppo
- Hugging Face. (2026). TRL documentation: PPO Trainer (trl.experimental.ppo). https://huggingface.co/docs/trl/main/en/ppo_trainer
- Regulation (EU) 2024/1689 of the European Parliament and of the Council (Artificial Intelligence Act), Article 53 and Annex XI. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- National Law Review. (2026). EU Digital Omnibus on AI enters into force (Regulation (EU) 2026/1744). https://natlawreview.com/article/eu-digital-omnibus-ai-enters-force

