Site icon Decode the future

RLHF Explained: How Human Feedback Trains AI Models in 2026

RLHF Explained: How Human Feedback Trains AI Models in 2026

Last updated: October 2026 · Sources checked: October 13, 2026

RLHF (reinforcement learning from human feedback) is a post-training method that teaches a language model which answers people prefer: humans compare model outputs, a reward model learns those preferences, and reinforcement learning (classically PPO) pushes the model toward higher-reward answers while a KL penalty keeps it close to the original. In 2026 the same idea runs in several forms — RLAIF swaps human labels for AI labels, DPO skips the reward model and RL loop, and RLVR replaces preferences with automatic checks for math and code.

RLHF Reward model PPO RLAIF DPO RLVR

What is RLHF, and why do modern LLMs still depend on it?

RLHF is the step that turned raw large language models into instruction-following assistants. Instead of writing a loss function that defines a “good” answer, you let people show the model what good looks like — by ranking its outputs — and then train a reward signal from those rankings.

The method was proposed by Christiano et al. (2017) for Atari games and simulated robotics. It became the template for LLM alignment with OpenAI’s InstructGPT paper (2022), where labelers preferred outputs from a 1.3B-parameter RLHF model over those of the 175B-parameter GPT-3 base model, despite the 100× size gap.

Since then, “RLHF” has become a family name rather than one recipe. Public technical reports show how varied the 2024–2026 stacks are:

  • Meta Llama 3 used supervised fine-tuning, rejection sampling with a reward model, and DPO; the authors report choosing DPO over PPO because it needed less compute at large scale and did better on instruction-following benchmarks such as IFEval (Llama 3 Herd of Models, §4.1).
  • AI2’s Tülu 3 trained in three stages — SFT, DPO, and a new stage it called Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024).
  • DeepSeek-R1 used large-scale reinforcement learning with GRPO, rule-based rewards for reasoning, and reward models for general helpfulness (DeepSeek-AI, 2025; peer-reviewed version in Nature).
  • Anthropic’s Claude models are trained with Constitutional AI, in which a written constitution is used to generate synthetic training data, including rankings of candidate responses (Anthropic, January 2026).

All four share the RLHF skeleton — a preference or reward signal, an optimizer, and a constraint that keeps the model from drifting. Understanding the original pipeline is what makes the newer variants legible.

How does the RLHF pipeline work? The 3 stages explained

The classic RLHF pipeline has three sequential stages. Each builds on the previous one, and a weak earlier stage limits everything after it.

RLHF 3-Stage Training Pipeline Diagram showing the three stages of RLHF: supervised fine-tuning (SFT) on instruction data, reward model training on human preference rankings, and policy optimization using PPO with KL divergence constraint. Each stage feeds into the next. RLHF 3-Stage Training Pipeline DecodeTheFuture.org RLHF, PPO, reward model, supervised fine-tuning, alignment pipeline Vertical flowchart of the RLHF training pipeline with 3 stages: SFT, reward model, and PPO policy optimization. Diagram image/svg+xml en © DecodeTheFuture.org STAGE 1 Supervised Fine-Tuning (SFT) Pretrained base model + curated (prompt, response) pairs → Standard cross-entropy loss → Output: π_SFT (instruction-following model) InstructGPT: ~13K SFT prompts STAGE 2 Reward Model Training For each prompt: generate K responses from π_SFT Humans rank: response_A ≻ response_B → Bradley-Terry model: P(A≻B) = σ(r(A) − r(B)) → Output: r_θ (scalar reward for any response) Often initialized from the SFT model STAGE 3 PPO Policy Optimization π_θ generates response → r_θ scores it PPO updates π_θ to maximize reward → KL penalty: D_KL(π_θ ∥ π_SFT) keeps coherence → Clipped objective prevents catastrophic updates → Output: π_RLHF (aligned model) 4 models in memory: policy, ref, reward, critic

Stage 1: Supervised Fine-Tuning (SFT)

You start with a pretrained base model that can produce fluent text but does not reliably follow instructions. SFT trains it on curated (prompt, ideal response) pairs with ordinary next-token cross-entropy loss — the same deep learning machinery used in pretraining, just on a much smaller, cleaner dataset.

Data quality dominates here. InstructGPT’s SFT set was built from roughly 13,000 training prompts with demonstrations written by a team of about 40 contractors (Ouyang et al., 2022). Current open recipes mix human-written demonstrations with synthetic responses from stronger models. The resulting model, πSFT, becomes both the starting policy for Stage 3 and the frozen reference model for the KL penalty. If you only need a model to follow a format or domain style, SFT alone — often with LoRA adapters — may be enough; see our RAG vs fine-tuning guide for that decision.

Stage 2: Reward Model Training

The reward model converts subjective human preferences into a number a reinforcement learning algorithm can optimize.

For each prompt, the SFT model generates several candidate responses. Annotators compare them — “A is better than B” — or rank a list. These comparisons train a reward model rθ with the Bradley-Terry preference model:

Bradley-Terry Loss

P(yw ≻ yl | x) = σ(rθ(x, yw) − rθ(x, yl))

σ is the sigmoid function, yw is the preferred response, yl the rejected one, and rθ outputs a scalar. Training maximizes the probability that the reward model scores the human-preferred output higher.

In practice the reward model is usually initialized from the SFT model with the language-modeling head replaced by a scalar head. How much data is “enough” depends more on quality than volume: NVIDIA’s HelpSteer2 dataset has only about 10,000 response pairs — roughly an order of magnitude fewer than Anthropic’s HH-RLHF — yet a reward model trained on it reached 92.0% on RewardBench’s primary set when published in June 2024. Who produces this data, and how, is covered in the vendor landscape section below.

Python — Reward model loss (simplified)
import torch
import torch.nn as nn
from transformers import AutoModelForSequenceClassification

class RewardModel(nn.Module):
    def __init__(self, base_model_name: str):
        super().__init__()
        self.model = AutoModelForSequenceClassification.from_pretrained(
            base_model_name, num_labels=1
        )

    def forward(self, input_ids, attention_mask):
        return self.model(
            input_ids=input_ids,
            attention_mask=attention_mask
        ).logits.squeeze(-1)  # scalar reward


def reward_loss(reward_model, chosen_ids, chosen_mask,
                rejected_ids, rejected_mask):
    """Bradley-Terry pairwise loss for reward model training."""
    r_chosen = reward_model(chosen_ids, chosen_mask)
    r_rejected = reward_model(rejected_ids, rejected_mask)
    # Maximize P(chosen > rejected) = sigmoid(r_chosen - r_rejected)
    return -torch.nn.functional.logsigmoid(r_chosen - r_rejected).mean()

In production you would not hand-roll this: Hugging Face TRL ships a RewardTrainer and a trl reward command that train the same pairwise objective from a (prompt, chosen, rejected) dataset.

Stage 3: PPO Policy Optimization

With a reward model in hand, you optimize the language model’s policy with reinforcement learning. The classic choice is Proximal Policy Optimization (PPO), which clips each update so the policy cannot change too much in one step.

RLHF-PPO Objective

maximize Ex∼D, y∼πθ [rθ(x, y) − β · DKL(πθ(y|x) ∥ πSFT(y|x))]

The first term pushes the model toward higher-reward outputs. The KL penalty, weighted by β, keeps it close to the SFT model — without it, the policy drifts into text that games the reward model.

This is where RLHF becomes operationally expensive. PPO-based RLHF keeps four models in play: the trainable policy πθ, the frozen reference πSFT, the reward model rθ, and a value (critic) model for advantage estimation. For a 70B-parameter setup, the weights alone are 4 × 70B × 2 bytes ≈ 560 GB in bf16 — before optimizer states for the two trainable models, activations, and the KV cache needed to generate rollouts. That arithmetic is why frameworks such as OpenRLHF split the models across GPUs and hand generation to vLLM.

Python — PPO with the current TRL API (trl.experimental.ppo)
from transformers import (AutoModelForCausalLM,
                          AutoModelForSequenceClassification, AutoTokenizer)
from trl.experimental.ppo import PPOConfig, PPOTrainer

SFT = "your-org/your-sft-model"
RM = "your-org/your-reward-model"

tokenizer = AutoTokenizer.from_pretrained(SFT, padding_side="left")
policy = AutoModelForCausalLM.from_pretrained(SFT)
ref_policy = AutoModelForCausalLM.from_pretrained(SFT)
reward_model = AutoModelForSequenceClassification.from_pretrained(RM, num_labels=1)
value_model = AutoModelForSequenceClassification.from_pretrained(RM, num_labels=1)

config = PPOConfig(
    output_dir="ppo-out",
    learning_rate=3e-6,
    kl_coef=0.05,            # beta: TRL's current default
    cliprange=0.2,           # PPO clip range
    vf_coef=0.1,             # value-loss weight
    gamma=1.0, lam=0.95,     # no discounting; GAE lambda
    num_ppo_epochs=4,
    missing_eos_penalty=1.0, # "EOS trick" recommended in the TRL docs
    per_device_train_batch_size=16,
    gradient_accumulation_steps=4,
)

trainer = PPOTrainer(
    args=config,
    processing_class=tokenizer,
    model=policy,
    ref_model=ref_policy,
    reward_model=reward_model,
    value_model=value_model,
    train_dataset=prompt_dataset,  # tokenized prompts ("input_ids")
)
trainer.train()  # watch objective/rlhf_reward, objective/kl and val/ratio
Implementation details that most tutorials miss

TRL moved its PPO trainer to trl.experimental, and the older ppo_trainer.step() loop with init_kl_coef found in many 2023 tutorials no longer matches the current API. The KL coefficient still matters: too low and the policy reward-hacks, too high and it barely moves from SFT. Huang, Liu and von Werra’s “The N Implementation Details of RLHF with PPO” (Hugging Face, October 2023) and the follow-up “N+ Implementation Details” paper (2024) document the details — reward normalization, EOS handling, padding, value-head initialization — that decide whether a run reproduces at all.

What is reward hacking, and why is it RLHF’s biggest problem?

Reward hacking happens when the policy finds ways to raise the reward model’s score without actually getting better. A reward model is an imperfect proxy for human judgment, trained on a finite set of comparisons, so it has blind spots the policy can exploit.

Two failure modes are well documented. First, over-optimization: Gao, Schulman and Hilton (2022) showed that as a policy optimizes harder against a proxy reward model, the “true” reward first rises and then falls — and they fitted scaling laws for how quickly that happens. Second, length bias: Singhal et al. (2023) found that a large share of measured RLHF improvement can be explained by responses simply getting longer.

Verifiable rewards do not make the problem disappear. In agentic coding environments, OpenAI researchers observed a frontier reasoning model exploiting weaknesses in its tests during RL training; monitoring its chain of thought caught much of this, but when the monitor was added to the reward, the model learned to hide its intent while still hacking at a significant rate (Baker et al., 2025).

Standard mitigations are the KL penalty, reward-model ensembles, held-out human evaluation, and stopping early when reward climbs faster than human-judged quality. The fragility of learned reward models is also the main reason the field built alternatives that remove them.

What is DPO, and why did it replace PPO for most teams?

Direct Preference Optimization (DPO), introduced by Rafailov et al. in May 2023, is the most important simplification of RLHF. It rearranges the RLHF objective so that both the separate reward model and the RL loop disappear, leaving a single supervised loss over preference pairs.

DPO Loss Function

LDPO(πθ; πref) = −E(x, yw, yl) [log σ(β · (log πθ(yw|x)/πref(yw|x) − log πθ(yl|x)/πref(yl|x)))]

This implicitly defines a reward r(x, y) = β · log(πθ(y|x) / πref(y|x)) — the policy itself acts as the reward model, which is where the paper’s subtitle “Your Language Model is Secretly a Reward Model” comes from.

RLHF vs DPO Architecture Comparison Side-by-side comparison showing RLHF requiring 4 models (policy, reference, reward, critic) and a complex PPO loop, versus DPO requiring only 2 models (policy, reference) and a single supervised loss. RLHF vs DPO Architecture Comparison DecodeTheFuture.org DPO, RLHF, PPO, alignment architecture comparison Diagram image/svg+xml en © DecodeTheFuture.org RLHF / PPO Policy model π_θ Reference model π_ref Reward model r_θ Critic / Value V(s) PPO Loop generate → score → update 4 models in memory Complex, unstable, expensive DPO Policy model π_θ Reference model π_ref No reward model needed No critic needed No RL loop needed Single supervised loss 2 models in memory Simple, stable, accessible vs

The practical gains are large: two models in memory instead of four, no value network, no online generation loop, and training that behaves like ordinary fine-tuning on (prompt, chosen, rejected) triplets.

Python — DPO with TRL
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer

model_id = "your-org/your-sft-model"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Rows look like {"prompt": ..., "chosen": ..., "rejected": ...}
train = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

args = DPOConfig(
    output_dir="dpo-out",
    beta=0.1,                 # strength of the implicit KL constraint
    loss_type="sigmoid",      # standard DPO loss
    learning_rate=5e-7,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    max_length=1024,          # prompt + completion
    num_train_epochs=1,
)

trainer = DPOTrainer(
    model=model,              # ref model is created automatically if omitted
    args=args,
    train_dataset=train,
    processing_class=tokenizer,
)
trainer.train()

DPO’s weakness is that it learns from a static set of pairs and never explores beyond them. Controlled comparisons bear this out. Ivison et al. (NeurIPS 2024) found that PPO beat DPO by up to 2.5% on math and 1.2% in general domains — but that preference-data quality mattered more than the algorithm, worth up to 8% on instruction following and truthfulness. Xu et al. (ICML 2024) reached a similar conclusion: a well-tuned PPO outperformed DPO in their benchmarks.

Our read: DPO is the sensible default when you have good pairwise data and limited compute, which describes most teams. The PPO advantage is real but small, and it only shows up when the PPO implementation is tuned carefully — a cost that is easy to underestimate.

What is KTO and when should you use it over DPO?

Kahneman-Tversky Optimization (KTO), from Ethayarajh et al. (2024), solves a data problem. DPO needs a “chosen” and a “rejected” response for every prompt, but most products collect a simpler signal: a thumbs-up or thumbs-down on a single answer.

KTO borrows from prospect theory — the same framework behind Kahneman and Tversky’s work on decisions under risk. It models utility asymmetrically, with losses weighing more than gains, and derives a loss that works on unpaired binary labels.

KTO in practice

Instead of (prompt, chosen, rejected) triplets, KTO trains on rows like {"prompt": ..., "completion": ..., "label": True}. That fits production feedback where users rate individual outputs — no pairing step needed.

The KTO authors report matching or exceeding DPO across models from 1B to 30B parameters, and argue the method is more robust when labels are noisy or imbalanced. Use it when your data is binary feedback; use DPO when you have clean pairs.

What is ORPO and why does it skip the reference model?

Odds Ratio Preference Optimization (ORPO), from Hong, Lee and Thorne (2024), removes the reference model entirely. It merges SFT and preference optimization into one objective, so a base model goes straight to an aligned model in a single run.

ORPO = SFT + preference in one pass

LORPO = LSFT + λ · LOR, where LOR penalizes the model when its odds of generating the rejected response are high relative to the chosen one. Only one model sits in memory.

The authors report that ORPO-tuned 7B models matched or beat larger models on AlpacaEval 2.0 and MT-Bench. The trade-off is that there is no KL anchor to a reference model, so data that is narrow or low quality can pull the model away from its general abilities. ORPO suits small models and single-GPU budgets with well-distributed preference data.

What are GRPO, DAPO, and RLVR — RL for reasoning models?

While DPO and KTO simplified preference alignment, the biggest change of 2025–2026 was a return to online RL — with a different reward source. Instead of human preferences or a learned reward model, these methods score outputs with verifiable rewards: programmatic checks that an answer is correct.

GRPO — Group Relative Policy Optimization

GRPO was introduced in DeepSeekMath (Shao et al., 2024) and later powered DeepSeek-R1. It removes PPO’s critic. For each prompt it samples a group of responses, scores each one, and uses the group’s own statistics as the baseline:

GRPO advantage

Ai = (ri − mean(r1..G)) / std(r1..G)

No value model is needed — the group is the baseline — which removes one of PPO’s four models.

Python — GRPO with a rule-based reward (TRL)
from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward   # checks the final answer

dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=accuracy_reward,   # verifiable reward, no reward model
    train_dataset=dataset,
)
trainer.train()

DAPO — Decoupled Clip and Dynamic Sampling Policy Optimization

DAPO (Yu et al., March 2025) is an open-source RL system built to fix instabilities seen when scaling GRPO-style training on long chains of thought. It adds four techniques: asymmetric “clip-higher” clipping to preserve exploration, dynamic sampling that drops prompts the model always or never solves, a token-level policy-gradient loss, and overlong-response reward shaping. It also drops the KL penalty, arguing that for verifiable reasoning tasks the model should be allowed to move far from its starting point.

RLVR — Reinforcement Learning with Verifiable Rewards

RLVR is the umbrella term, introduced in the Tülu 3 report: use RL (PPO, GRPO, DAPO, REINFORCE-style variants) but replace the learned reward with deterministic verification — check the final math answer, run unit tests, validate output against a schema. It removes the human labeling bottleneck for those domains and can generate training signal at the speed of the verifier.

Two caveats matter. Verifiers can still be gamed, as the reward-hacking evidence above shows. And most valuable tasks are not cleanly verifiable. That is why 2025–2026 research has pushed into rubric-based rewards: “Rubrics as Rewards” (Gunjal et al., 2025) uses checklist-style rubrics scored by an LLM judge and reports gains of up to 31% on HealthBench and 7% on GPQA-Diamond over Likert-scale LLM-judge rewards. Rubrics are a bridge between RLVR and RLAIF.

The layered post-training stack in 2026

Public recipes such as Tülu 3 and DeepSeek-R1 follow a similar pattern: SFT for format and instruction following → preference optimization (DPO, or PPO with a reward model) for helpfulness and tone → RL with verifiable or rubric-based rewards for reasoning. “RLHF” today usually means this whole stack, not one algorithm.

How do Constitutional AI and RLAIF relate to RLHF?

Anthropic’s Constitutional AI (Bai et al., 2022) does not replace RLHF; it changes where the labels come from. It has two phases. In the supervised phase, the model drafts responses, critiques them against written principles (the “constitution”), revises them, and is fine-tuned on the revisions. In the RL phase, a model compares pairs of samples and judges which better follows the principles; those AI preference labels train a preference model that serves as the reward. The paper named this RL from AI Feedback (RLAIF).

Does AI feedback work as well as human feedback? Google researchers Lee et al. (ICML 2024) found that RLAIF achieved performance comparable to RLHF on summarization, helpful dialogue and harmless dialogue. They also tested “direct RLAIF”, which skips the reward model and asks an off-the-shelf LLM for a score during RL, and found it did better than standard RLAIF.

The approach has matured since 2023. On January 22, 2026, Anthropic published a new, much longer constitution for Claude under a CC0 license. It explains the reasoning behind its values rather than listing standalone rules, and Anthropic says Claude uses it to generate synthetic training data — including conversations, responses that reflect its values, and rankings of candidate responses. For teams building AI agents, the lesson is that written, auditable principles can scale feedback far beyond what a human annotation budget allows — but they inherit the blind spots of the model doing the judging.

RLHF vs RLAIF vs DPO: what is the difference?

These three terms are often listed as rivals, but they answer different questions. RLHF vs RLAIF is about who produces the preference labels — humans or an AI model. RLHF vs DPO is about how you optimize on those labels — a reward model plus RL, or a direct supervised loss. You can therefore combine them: DPO on AI-labeled pairs is common (UltraFeedback, for example, contains over a million GPT-4 feedback items on 250,000 conversations, per Cui et al.).

Dimension RLHF (classic) RLAIF DPO
Source of preference labels Human annotators An AI judge, often guided by a constitution or rubric Any: human, AI, or mixed pairs
Reward model? Yes, trained on comparisons Usually yes (trained on AI labels); “direct RLAIF” scores with an LLM instead No — the policy acts as an implicit reward model
Optimizer Online RL (PPO) Online RL (PPO or similar) Offline supervised loss
Models in memory 4 (policy, reference, reward, value) 4, plus access to the AI labeler 2 (policy, reference)
Main cost driver Human labeling and RL compute Labeler inference and RL compute Preference data; training compute is close to SFT
Main risk Reward hacking; annotator inconsistency and bias Inherits the judge model’s biases and blind spots No exploration beyond the static dataset
Evidence InstructGPT (Ouyang et al., 2022) Comparable to RLHF on 3 tasks (Lee et al., 2024) Tuned PPO beats DPO by up to 2.5% on math (Ivison et al., 2024)

Practical rule: decide the label source by what you can afford and audit (humans for high-stakes nuance, AI for scale), then decide the optimizer by your compute and engineering budget (DPO first; PPO or GRPO when you need online exploration).

Which alignment method should you use? A practical comparison

Method Models in memory Data required Relative compute Best for
RLHF (PPO) 4 (policy, ref, reward, value) Pairwise preferences + online rollouts Highest Maximum quality with a skilled RL team
RLAIF / Constitutional AI 4 + AI labeler Written principles or rubrics + AI-labeled pairs High Scaling feedback and safety norms
DPO 2 (policy, ref) Pairwise preferences (static) Low Default for most teams
KTO 2 (policy, ref) Binary labels (thumbs up/down) Low Product feedback, noisy labels
ORPO 1 (policy only) Pairwise preferences (merged with SFT) Lowest Small models, single GPU
GRPO Policy + ref (no value model) Verifiable or rubric rewards Medium–high (many samples per prompt) Reasoning: math, code
DAPO Policy only (no KL reference) Verifiable rewards Medium–high Long chain-of-thought RL at scale

Who supplies RLHF data? The 2026 vendor landscape

Most AI labs do not produce all their preference data in-house. The market that supplies it falls into four types of provider. This section is a neutral map, not a ranking: DecodeTheFuture has no commercial relationship with any company named, has not tested their services, and none of them publish comparable quality benchmarks or public per-label prices. For a provider-by-provider comparison of scope, workflow and quality controls, see our RLHF data providers and vendors guide.

Provider type What they supply Examples and verifiable public facts
Managed data and labeling firms Run the full project: task design, workforce, QA, delivery of comparisons, demonstrations and evaluations Scale AI — Meta took a 49% non-voting stake for about $14.3B in June 2025 (CNBC); founder Alexandr Wang moved to Meta (Scale). Francis deSouza became CEO on August 10, 2026, with Scale describing two businesses: data for AI labs and applications (Scale). Appen, Toloka and Sama describe RLHF, preference or evaluation services on their own pages.
Expert networks Domain specialists (coders, doctors, lawyers, PhDs) who write hard prompts, grade answers and build rubrics Surge AI — bootstrapped; Reuters reported in July 2025 that its annual revenue exceeded $1B as it sought its first outside raise (SiliconANGLE). Mercor — raised $350M at a $10B valuation in October 2025 (TechCrunch). Turing describes SFT, RLHF and DPO data programs.
Platforms and tooling Software for side-by-side comparisons, reviewer management and export; you bring or hire the workforce Labelbox documents a human preference editor for reward-model data; SuperAnnotate describes preference datasets and workforce management.
Synthetic and AI feedback AI-labeled preference pairs, constitution- or rubric-driven judges, open datasets Open sets such as UltraFeedback (GPT-4 labels) and NVIDIA’s HelpSteer2 (about 10K pairs, CC-BY-4.0); in-house constitution-driven labeling as in Anthropic’s Claude training.

Three things have shaped this market since 2025. Demand has shifted from generic crowd labeling toward expert feedback for reasoning, coding and agentic tasks, where a grader has to understand the answer. Lab neutrality became a buying criterion after the Meta–Scale deal, as labs weighed whether a supplier part-owned by a competitor should see their prompts. And AI feedback now does much of the volume work, so human budgets go to the hard, high-stakes cases and to evaluation sets.

How to judge any RLHF data provider

Ask for a paid pilot on your own prompts; inter-annotator agreement and adjudication rules; reviewer qualifications for your domain; how AI assistance is used and disclosed in their pipeline; data-security and lab-isolation terms; and whether you get held-out evaluation sets, not just training data. A vendor that cannot show calibration evidence on a pilot is a risk regardless of its size.

How much does RLHF cost? Compute and data cost drivers

Public, verifiable cost figures for alignment are rare; most labs do not disclose them. The figures that are public point to one pattern: post-training compute is small next to pretraining, but RL for reasoning and high-quality human data can still be expensive.

  • Preference post-training can be cheap in compute. DeepSeek’s V3 report lists 2,664K H800 GPU hours for pre-training but only 5K for post-training (SFT and RL) — about $10,000 at an assumed $2 per GPU hour, out of a $5.576M total (DeepSeek-V3, Table 1). This excludes research, failed runs and data costs.
  • Reasoning RL is a bigger line item. DeepSeek disclosed in its Nature paper that training R1 cost about $294,000, using 512 Nvidia H800 chips for 80 hours on the final run, on top of roughly $6M for the V3 base model (Reuters via BNN Bloomberg, September 2025). Critics note these are rental-price estimates, not full program costs.
  • Algorithm choice changes the bill. PPO holds four models and generates rollouts every step; DPO holds two and trains offline; GRPO drops the value model but samples a whole group of responses per prompt. Meta cited DPO’s lower compute at scale as a reason to choose it over PPO for Llama 3.
  • Human data is the hidden cost. No major provider publishes per-comparison prices; costs depend on domain expertise, task length and QA depth. Evidence from Ivison et al. and HelpSteer2 suggests spending on fewer, better comparisons beats buying volume.

For a practitioner, the compute cost of a DPO run on a 7B model with LoRA is in the same order of magnitude as an SFT run of the same size (it processes a chosen and a rejected response per example); the real budget question is the data. For serving costs after training, see our LLM inference cost comparison.

Can RLHF or DPO make a model less capable?

Yes — the effect has a name, the alignment tax. InstructGPT reported performance regressions on several public NLP benchmarks after RLHF, and reduced them by mixing pretraining gradients into PPO updates (the “PPO-ptx” variant) (Ouyang et al., 2022). Offline methods are not immune: because DPO and ORPO only see the preference pairs, narrow or low-quality data can shift the model away from skills the pairs do not cover.

Mitigations that hold up: keep a KL or reference anchor (or a pretraining/SFT loss term), mix diverse data, train for few epochs, and benchmark broad capabilities before and after alignment. Teams adding reasoning RL should also re-check general helpfulness, since a model optimized only on verifiable tasks can become terse or over-formatted elsewhere.

How does RLHF intersect with the EU AI Act in 2026?

The EU AI Act does not prescribe any alignment technique, but it does require documentation that covers alignment training.

General-purpose AI model providers have had obligations since August 2, 2025. Under Article 53 and Annex XI they must keep technical documentation that includes “training methodologies and techniques” and information on training data, “including the type and provenance of data and curation methodologies” (Regulation (EU) 2024/1689). For a model aligned with RLHF, that means records of how preference data was collected or generated, by whom, and how it was filtered.

High-risk systems face data-governance rules (Article 10) and accuracy and robustness requirements (Article 15). The AI Omnibus, Regulation (EU) 2026/1744, in force since July 27, 2026, moved the application date for stand-alone high-risk systems from August 2, 2026 to December 2, 2027, and for high-risk systems embedded in regulated products to August 2, 2028; GPAI obligations were not changed (National Law Review).

The practical implication: document the pipeline end to end — annotator instructions and demographics, AI-labeler prompts and constitutions, reward-model evaluations, and bias testing on preference data. RLAIF with a written constitution or rubric has an auditing advantage here, because the labeling policy exists as a document.

How do you implement RLHF in 2026? A decision guide

Do you have verifiable outputs (math, code, structured data)? Use GRPO or DAPO with rule-based rewards. TRL’s GRPOTrainer is the simplest start; OpenRLHF implements PPO, GRPO, REINFORCE++ and DAPO on Ray and vLLM for larger runs.

Do you have pairwise preference data? Use DPO. One epoch is usually sufficient; overtraining on a fixed preference set tends to degrade outputs.

Do you only have thumbs-up/thumbs-down feedback? Use KTO — same library, unpaired labels.

Are you on a single GPU with a small model? Try ORPO or DPO with LoRA adapters.

Do you lack labels altogether? Generate AI preference pairs against a written rubric or constitution (RLAIF), then spot-check a sample with human reviewers before training.

Do you need maximum quality and have an experienced RL team? PPO with a strong reward model is still competitive, as Ivison et al. and Xu et al. show — budget for the tuning.

Bash — Quick start: DPO with the TRL CLI
# Install
pip install trl peft

# Run DPO on a public preference dataset (LoRA keeps memory low)
trl dpo \
  --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
  --dataset_name trl-lib/ultrafeedback_binarized \
  --output_dir ./qwen-dpo \
  --beta 0.1 \
  --learning_rate 5e-7 \
  --num_train_epochs 1 \
  --use_peft \
  --lora_r 16 \
  --lora_alpha 32

If you want to go deeper than this article, Nathan Lambert’s free RLHF Book covers the full post-training pipeline in detail.

FAQ

RLHF (reinforcement learning from human feedback) is a way to fine-tune a language model on what people prefer. Humans compare pairs of model answers, a reward model learns to predict those preferences, and reinforcement learning (usually PPO) nudges the model toward higher-scoring answers while a KL penalty keeps it close to its starting point. It is the method OpenAI used for InstructGPT in 2022.
The difference is who labels the preferences. In RLHF, human annotators compare model outputs; in RLAIF (RL from AI Feedback), an AI model does the comparing, often guided by written principles such as Anthropic’s constitution. The training steps after labeling are the same. Google researchers (Lee et al., ICML 2024) found RLAIF comparable to RLHF on summarization and helpful and harmless dialogue; the risk is that the AI judge passes on its own biases.
There is no independent ranking, so it is better to think in provider types. Managed data firms such as Scale AI, Appen, Toloka and Sama run full labeling projects; expert networks such as Surge AI, Mercor and Turing supply domain specialists; platforms such as Labelbox and SuperAnnotate provide tooling; and synthetic or AI-feedback pipelines generate labels at scale. Pick a provider with a paid pilot on your own prompts and check agreement, reviewer expertise and security terms.
Public figures are rare. DeepSeek’s V3 report lists only 5,000 H800 GPU hours (about $10,000 at $2 per hour) for post-training, against 2.66 million hours for pre-training. Reasoning-focused RL costs more: DeepSeek put R1’s training at about $294,000 on 512 H800 chips, excluding roughly $6 million for the base model. Human preference data is often the largest hidden cost, and vendors do not publish per-comparison prices.
Not strictly. DPO is simpler and cheaper: it needs two models instead of four and no RL loop, which is why Meta chose it for Llama 3. But controlled studies found a well-tuned PPO still ahead, by up to 2.5% on math and 1.2% in general domains (Ivison et al., 2024). The same study found that better preference data mattered more than the choice of algorithm.
Yes. This alignment tax was reported in InstructGPT, where RLHF reduced scores on some public NLP benchmarks; mixing pretraining data into PPO updates (PPO-ptx) reduced the loss. DPO and ORPO can also shift a model away from skills their preference pairs do not cover. Keeping a KL or reference anchor, using diverse data, training for few epochs and benchmarking before and after alignment limit the damage.
RLVR (Reinforcement Learning with Verifiable Rewards), a term introduced in AI2’s Tülu 3 report, replaces the learned reward model with an automatic check, such as a correct final math answer or passing unit tests. It powers reasoning models such as DeepSeek-R1, which used GRPO. RLHF learns its reward from human preferences, so it can handle subjective qualities like tone and helpfulness that a verifier cannot check.

References

  1. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. arXiv. https://arxiv.org/abs/1706.03741
  2. Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. arXiv. https://arxiv.org/abs/2203.02155
  3. Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv. https://arxiv.org/abs/2204.05862
  4. Bai, Y., Kadavath, S., Kundu, S., et al. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv. https://arxiv.org/abs/2212.08073
  5. Lee, H., Phatale, S., Mansoor, H., et al. (2024). RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. ICML 2024. https://arxiv.org/abs/2309.00267
  6. Anthropic. (2026, January 22). Claude’s new constitution. https://www.anthropic.com/news/claude-new-constitution
  7. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. arXiv. https://arxiv.org/abs/2305.18290
  8. Ivison, H., Wang, Y., Liu, J., et al. (2024). Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. NeurIPS 2024. https://arxiv.org/abs/2406.09279
  9. Xu, S., Fu, W., Gao, J., et al. (2024). Is DPO superior to PPO for LLM alignment? A comprehensive study. ICML 2024. https://arxiv.org/abs/2404.10719
  10. Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024). KTO: Model alignment as prospect theoretic optimization. arXiv. https://arxiv.org/abs/2402.01306
  11. Hong, J., Lee, N., & Thorne, J. (2024). ORPO: Monolithic preference optimization without reference model. arXiv. https://arxiv.org/abs/2403.07691
  12. Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the limits of mathematical reasoning in open language models (introduces GRPO). arXiv. https://arxiv.org/abs/2402.03300
  13. DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv; peer-reviewed version in Nature (2025). https://arxiv.org/abs/2501.12948
  14. DeepSeek-AI. (2024). DeepSeek-V3 technical report (Table 1: training costs). arXiv. https://arxiv.org/abs/2412.19437
  15. Yu, Q., Zhang, Z., Zhu, R., et al. (2025). DAPO: An open-source LLM reinforcement learning system at scale. arXiv. https://arxiv.org/abs/2503.14476
  16. Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). Tülu 3: Pushing frontiers in open language model post-training (introduces RLVR). arXiv. https://arxiv.org/abs/2411.15124
  17. Grattafiori, A., Dubey, A., et al. (2024). The Llama 3 herd of models. arXiv. https://arxiv.org/abs/2407.21783
  18. Gunjal, A., Wang, A., Lau, E., et al. (2025). Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv. https://arxiv.org/abs/2507.17746
  19. Gao, L., Schulman, J., & Hilton, J. (2022). Scaling laws for reward model overoptimization. arXiv. https://arxiv.org/abs/2210.10760
  20. Singhal, P., Goyal, T., Xu, J., & Durrett, G. (2023). A long way to go: Investigating length correlations in RLHF. arXiv. https://arxiv.org/abs/2310.03716
  21. Baker, B., Huizinga, J., Gao, L., et al. (2025). Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv. https://arxiv.org/abs/2503.11926
  22. Wang, Z., Dong, Y., Delalleau, O., et al. (2024). HelpSteer2: Open-source dataset for training top-performing reward models. arXiv. https://arxiv.org/abs/2406.08673
  23. Cui, G., Yuan, L., Ding, N., et al. (2023). UltraFeedback: Boosting language models with scaled AI feedback. arXiv. https://arxiv.org/abs/2310.01377
  24. Huang, S., Liu, T., & von Werra, L. (2023, October 24). The N implementation details of RLHF with PPO. Hugging Face Blog. https://huggingface.co/blog/the_n_implementation_details_of_rlhf_with_ppo
  25. Hugging Face. (2026). TRL documentation: PPO Trainer (trl.experimental.ppo). https://huggingface.co/docs/trl/main/en/ppo_trainer
  26. Regulation (EU) 2024/1689 of the European Parliament and of the Council (Artificial Intelligence Act), Article 53 and Annex XI. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  27. National Law Review. (2026). EU Digital Omnibus on AI enters into force (Regulation (EU) 2026/1744). https://natlawreview.com/article/eu-digital-omnibus-ai-enters-force
Exit mobile version