3:12
Reinforcement learning from human feedback (RLHF) is a technique used to align a language model's behavior with human preferences by training it using feedback from human raters, rather than relying solely on predicting the next token in text. RLHF is a key part of what turns a raw, pre-trained language model into a helpful, well-behaved assistant, and is used by nearly every major consumer-facing chat model.
A model pre-trained purely to predict the next token learns to imitate the vast, uneven mix of text it was trained on - which includes helpful writing, but also misinformation, rudeness, and content that isn't in the format a helpful assistant should use. Next-token prediction alone has no notion of "helpful," "honest," or "harmless" - it only knows what's statistically likely to come next. RLHF introduces an explicit training signal based on what humans actually prefer, nudging the model toward responses people rate as more helpful and appropriate, rather than just statistically plausible.
Before RLHF begins, the base model is usually first fine-tuned on a dataset of high-quality example conversations - a prompt paired with the kind of response a helpful assistant should give, written or curated by human annotators. This supervised fine-tuning (SFT) step gets the model into roughly the right "shape" for conversational, instruction-following behavior before the more nuanced preference-based training begins.
Next, the SFT model is used to generate multiple candidate responses to the same prompt, and human raters compare pairs of responses, indicating which one they prefer. This comparison data is used to train a separate reward model - a model whose job is to take a prompt and response and output a single score predicting how much a human would like that response. Once trained, the reward model can score new responses automatically, standing in for a human rater at massive scale.
reward_model.score(prompt, response_a) = 0.82
reward_model.score(prompt, response_b) = 0.35
# response_a is predicted to be preferredIn the final step, the SFT model is further trained using reinforcement learning, typically an algorithm such as Proximal Policy Optimization (PPO): the model generates a response, the reward model scores it, and that score is used as a reward signal to adjust the model's parameters to make higher-scoring responses more likely in the future. A penalty term is usually included to keep the model's outputs from drifting too far from its original SFT behavior, which helps prevent it from over-optimizing for the reward model in ways that produce strange or degenerate text that happens to score well without actually being good.
RLHF typically collects human feedback as pairwise comparisons ("which of these two responses is better?") rather than absolute numeric ratings ("rate this response 1-10"). Comparisons are generally faster and more consistent for human raters to produce reliably - people are often better at judging relative quality between two options than assigning a stable absolute score - which produces cleaner training data for the reward model.
RLHF has known limitations: the reward model can be "gamed" if the reinforcement learning process finds outputs that score well without genuinely being better (a failure mode called reward hacking), the quality of the whole pipeline is limited by the consistency and judgment of the human raters whose preferences were collected, and RLHF optimizes for what raters say they prefer, which isn't always perfectly aligned with what's actually true, safe, or in a user's genuine interest. Because of these limitations, RLHF is generally understood as an important but imperfect tool for alignment, and remains an active area of ongoing research.
By: Tomas Silny
Edited: 2026-08-13 06:50:01