← Explore papers

Paper · 2024

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment

Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, Quanquan Gu

Read the source ↗Open source PDF ↗arXiv 2410.02197

Watch and listen

Paper explainer

One shared explainer for this paper, with versions you can explore more deeply.

Ready

Progress

55 saved steps

Published

Requested
Last completed
Last status update
Activity log

Showing the latest 50 saved steps.

  1. Recording the podcast
  2. Recording the podcast
  3. Recording the podcast
  4. Preparing the explanation
  5. Recording the podcast
  6. Recording the podcast
  7. Recording the podcast
  8. Recording the podcast
  9. Recording the podcast
  10. Recording the podcast
  11. Recording the podcast
  12. Recording narration
  13. Recording the podcast
  14. Recording the podcast
  15. Recording narration
  16. Recording the podcast
  17. Recording narration
  18. Recording the podcast
  19. Recording narration
  20. Recording the podcast
  21. Recording narration
  22. Recording the podcast
  23. Recording the podcast
  24. Recording narration
  25. Recording the podcast
  26. Recording narration
  27. Recording the podcast
  28. Recording the podcast
  29. Recording narration
  30. Recording the podcast
  31. Recording narration
  32. Recording the podcast
  33. Recording narration
  34. Recording the podcast
  35. Recording the podcast
  36. Preparing the explanation
  37. Recording narration
  38. Recording the podcast
  39. Recording the podcast
  40. Recording the podcast
  41. Recording the podcast
  42. Recording the podcast
  43. Recording the podcast
  44. Recording the podcast
  45. Recording the podcast
  46. Recording the podcast
  47. Rendering technical diagrams
  48. Rendering technical diagrams
  49. Rendering technical diagrams
  50. Assembling the video

Animated diagrams illustrate the ideas. Measurements and cited claims follow the paper's source evidence.

Podcast

Funded custom versions are not yet available. The shared video and podcast remain available.

Download videoDownload captionsDownload podcast
Video transcript

Aligning language models depends on modeling human preferences.

The standard Bradley-Terry reward model gives each response one scalar score.

This ICML twenty twenty-five paper proposes preference embedding, a way to capture

general, even cyclic, preferences efficiently.

It also proposes a policy method called General Preference Optimization.

A single score imposes a total ordering, so it cannot represent a cycle like rock, paper,

scissors.

Pairwise models such as PairRM read two concatenated responses and can express cycles.

However, comparing K responses needs on the order of K squared evaluations.

The authors also report that even with shuffled training pairs, the output of these models

is still highly asymmetric.

Here, each response is embedded as a vector with two k dimensions.

The score between two responses is the inner product of one embedding, transformed by a

block-diagonal skew-symmetric operator, with the other.

Swapping responses flips the sign of the score, and comparing a response with itself gives

zero.

The operator also preserves vector length.

Each response is embedded only once, so K responses need on the order of K queries,

matching Bradley-Terry.

With k equal to one and the second embedding coordinate fixed to a nonzero constant, the

model reduces to Bradley-Terry, with the reward difference scaled by that constant.

Theorem four point four states that any real skew-symmetric preference matrix can be

represented this way, and the authors note similar results by Balduzzi and colleagues.

The construction in the proof, though, uses embeddings whose dimension grows with the

number of items.

The implementation adds a prompt-based eigenvalue scale gate and an embedding head whose

outputs are normalized to unit length for stability.

Reward models were trained on about eighty thousand Skywork preference pairs for two

epochs, with a batch size of thirty-two; the general preference model's loss used a beta

of zero point one.

General Preference Optimization iterates like Self-Play Preference Optimization but

maximizes the expected preference score, a log-odds quantity, instead of the win

probability.

The intended behavior is that responses with high average scores gain probability.

In practice, a normalizing term is set to zero, based on a near-equilibrium heuristic.

Theorem five point one claims convergence at a rate of one over the square root of T,

assuming realizability and bounded scores.

The authors built four cyclic datasets from Ultrafeedback, each with two hundred sixteen

to three hundred sixty-three instances, by switching criteria such as honesty and

helpfulness across the links of a cycle.

With Gemma two B I T as the base, the general preference model scored one hundred percent

on all four datasets.

Bradley-Terry scored sixty-two point four, sixty-one point six, fifty point zero, and

sixty-two point nine percent, compared with a random baseline of fifty.

Accuracy was measured on the training dataset with different comparison pairs.

On RewardBench with Gemma two B I T, the general preference model averaged eighty-two

point two nine at dimension six, compared with seventy-four point eight five for

Bradley-Terry, a gap of seven point four four points.

With Llama three point one eight B Instruct, it averaged ninety-one point nine zero

compared with ninety point five six, a gap of one point three four points.

Among the Table two configurations, Bradley-Terry still led on Gemma Safety and Llama

Reasoning.

The text says normalization consistently helps scale-gated models, yet Gemma averages

favor no normalization at dimensions two, four, and eight.

For alignment, a Llama three eight B Instruct policy was judged on AlpacaEval two point

zero by GPT four turbo.

Methods using the general preference model beat Bradley-Terry on raw win rate in every

row.

For the two B models with General Preference Optimization at iteration three, the win rate

was forty-eight point two five versus forty-four point two zero.

Length-controlled win rate, however, is often lower, for example thirty-six point zero six

versus forty point zero one in the two B self-play setting at iteration three.

Responses also grow longer, up to three thousand two hundred forty-nine tokens versus one

thousand nine hundred sixty-nine.

The authors acknowledge this length effect.

Other results are mixed.

On MT-Bench, the eight B general preference model with General Preference Optimization at

iteration three averaged seven point five four, below Bradley-Terry at eight point two six

and the base model at eight point zero three.

A length-normalized variant edged Bradley-Terry, forty-five point five five to forty-five

point five one, in a single comparison.

No significance tests are reported.

That absence is not evidence of no effect, but small gaps cannot be called reliable.

Finally, here are the analysis caveats.

The one hundred percent cyclic-preference accuracy is measured on the training dataset

with different comparison pairs, so it is not a held-out generalization result.

The claim that the Bradley-Terry model behaves like a random guess holds exactly only for

Cyclic number three at fifty percent; on the other three datasets it scores between about

sixty-one and sixty-three percent.

RewardBench results are reported without seeds or variance, and the embedding dimension

appears to be selected on the same benchmark.

The Bradley-Terry reward model still beats the Table two GPM configurations on Gemma

Safety and all GPM variants on Llama Reasoning, so the advantage of the general preference

model is not uniform across categories.

The AlpacaEval gains for the general preference model are on raw win rate;

length-controlled win rate is often lower and responses are longer, so length bias cannot

be ruled out as part of the gain.

The statement that General Preference Optimization consistently outperforms Self-Play

Preference Optimization is not supported by every row of the results table.

The convergence proof for Theorem five point one does not reach its stated rate as

written, and its assumptions of realizability and bounded scores are not checked

empirically.

The expressiveness theorems show existence with embeddings that grow with the number of

items, and they do not guarantee performance at the small dimensions used in practice.

The Appendix F cycle examples are illustrative narratives and are not measured evidence of

intransitive human preferences.

Podcast transcript

Today we're looking at a paper that tries to move beyond Bradley-Terry reward models for

aligning language models.

What problem is it tackling?

The standard Bradley-Terry approach gives every response a single scalar reward.

That is efficient, but a single number cannot express cyclic preferences, where A beats B,

B beats C, and C beats A.

The authors want something that is both expressive and efficient.

Aren't there already models that compare two responses directly?

Yes.

Pairwise models concatenate both responses and predict which one wins.

They can express cycles, but ranking K responses requires on the order of K squared

comparisons.

They are also sensitive to position, and the paper says the output remains highly

asymmetric even after shuffling the order of training pairs.

The proposed model needs only on the order of K queries, the same as Bradley-Terry.

So what is a preference embedding?

Each response, given the prompt, becomes a vector.

The score for one response over another is an inner product passed through a

block-diagonal skew-symmetric operator built from small rotation-like blocks.

Swapping the two responses flips the sign of the score, and the operator does not change

the length of a vector.

How does that relate to Bradley-Terry?

It is a special case.

With a single block, and a vector made of the reward plus a nonzero constant, the score

reduces to the Bradley-Terry reward difference multiplied by that constant (exactly the

difference when the constant is one).

Can it represent any preference pattern?

The theorem says any real skew-symmetric preference matrix can be represented, and the

authors note that similar results appeared in earlier work by Balduzzi and colleagues.

There is an important caveat, though.

The constructive proof uses embeddings whose dimension grows with the number of items, so

it does not guarantee good performance at the small dimensions actually used.

How is it built in practice?

They add two components to a language model.

The first is an eigenvalue scale gate, which weights preference dimensions based on the

prompt.

The second is an eigenvector embedding head, which produces the response vectors.

The embeddings are normalized to unit length for training stability.

And the optimization method, GPO?

General Preference Optimization is an iterative, self-play style objective.

It maximizes the expected preference score, which is a log-odds, rather than the win

probability.

In practice, they replace a normalizing term with zero, justified by a heuristic that

training is close to equilibrium.

Is there a convergence guarantee?

The theorem claims the duality gap shrinks like one over the square root of the number of

iterations, assuming realizability and bounded scores.

However, the proof's one-sided bounds do not subtract to that rate as written, and the

assumptions are not checked empirically.

I would treat this result cautiously.

Let's turn to experiments.

How did they test cyclic preferences?

They built four datasets from Ultrafeedback using four rating criteria, with between two

hundred sixteen and three hundred sixty-three instances each.

Cycles were formed by switching criteria across links, for example comparing one pair on

honesty and another on helpfulness.

And the results?

Using the two billion parameter Gemma model, the new model reaches one hundred percent on

all four datasets.

Bradley-Terry scores sixty-two point four, sixty-one point six, fifty point zero, and

sixty-two point nine, against a random baseline of fifty.

So random-guess behavior holds exactly only on the third dataset.

Accuracy was also measured on the training dataset with different comparison pairs, so

this is not a held-out generalization result.

What about RewardBench?

The reward models were trained on about eighty thousand Skywork preference pairs.

With Gemma, the new model averages eighty-two point two nine at dimension six, versus

seventy-four point eight five for Bradley-Terry.

With the eight billion parameter Llama model, it averages ninety-one point nine zero

versus ninety point five six, though a cited Chat-Hard score comes from a different

dimension.

No repeated runs or variance are reported, and the dimension seems to have been chosen on

the same benchmark.

Is the advantage consistent across categories?

It is not uniform.

Bradley-Terry beats every Table two variant of the new model on Gemma Safety (eighty-five

point six eight versus at most eighty-five point five four), and every variant in Tables

two and four on Llama Reasoning (ninety-six point four seven versus at most ninety-six

point four zero), though two Gemma ablation variants without a scale gate edge it on

Safety (eighty-five point eight one and eighty-six point zero eight).

The ablation text also claims that L two normalization consistently helps scale-gated

models, but the Gemma averages favor dropping it at dimensions two, four, and eight.

Does it help downstream alignment?

On AlpacaEval two point zero with a GPT four turbo judge, methods using the new model beat

Bradley-Terry on raw win rate in every row.

However, the length-controlled win rate is often lower, and responses are longer.

The authors acknowledge the longer outputs, so length bias cannot be ruled out as part of

the gain.

Does GPO always beat the self-play baseline?

Not in every row.

For both eight billion setups at iteration three, with Bradley-Terry and with the new

model, the self-play baseline does better.

MT-Bench results are mixed as well.

The eight billion model with GPO at iteration three averages seven point five four, below

Bradley-Terry at eight point two six and below the base model at eight point zero three.

Did they try to fix the length issue?

They tested a length-normalized variant.

The new model reached a forty-five point five five percent length-controlled win rate

versus forty-five point five one for Bradley-Terry.

That comes from a single first-iteration comparison, so the gap is too small to draw

conclusions from.

The appendix includes examples of cycles.

Are those evidence?

They are illustrative narratives with no annotator counts.

The second example is hedged with phrases like 'some evaluators might prefer', while the

first states evaluator preferences outright without supporting counts.

They are not measured evidence that humans actually hold intransitive preferences.

So what's the takeaway?

The construction is elegant, and it matches Bradley-Terry's linear query cost while being

more expressive.

The reported empirical gains are uneven and are reported without repeated runs or

variance; put plainly, the gains come without reported repeated runs or variance.

The cyclic results are not held out, and the downstream improvements may partly reflect

longer responses.

It is promising, but the evidence is weaker than the headline claims suggest.

How this was checked

Source-grounded claim checks and visual quality checks passed before publication.

No confirmed corrections remained after the final checking pass.

Abstract

Problem. Bradley-Terry (BT) reward models assign each response a scalar reward, which imposes a total order and therefore can’t represent intransitive/cyclic preferences (e.g., a rater or population preferring y1≻y2≻y3≻y1y_1 \succ y_2 \succ y_3 \succ y_1 across different evaluation axes). Fully general pairwise preference models (PairRM/PairPM) can capture cycles but require O(K2)O(K^2) pairwise comparisons for KK candidate responses — expensive at test-time-scaling inference.

Preference embedding formalism. Embed each response into R2k\mathbb{R}^{2k} via vy∣x\mathbf{v}_{y|x}, and score pairs through a fixed block-diagonal skew-symmetric operator R≻\mathbf{R}^\succ:
s(yi≻yj∣x)=⟨R≻vyi∣x,vyj∣x⟩s(y_i \succ y_j \mid x) = \langle \mathbf{R}^\succ \mathbf{v}_{y_i|x}, \mathbf{v}_{y_j|x}\rangle
Skew-symmetry gives s(yi≻yj)=−s(yj≻yi)s(y_i \succ y_j) = -s(y_j \succ y_i) for free — the antisymmetry a preference relation needs — while the embedding space itself is what encodes cycles, since bilinear forms over ≥2\geq 2 dimensions aren’t constrained to a total order the way a scalar is. Query cost stays O(K)O(K) (one embedding per response), matching BT’s cost while gaining PairRM’s expressiveness. A complex-number reading is offered too: scores equal the imaginary part of a Hermitian inner product of complex-valued embeddings.

GPM architecture. Two heads on top of an LM backbone: an eigenvalue scale gate Gλ\mathcal{G}_\lambda producing context-dependent nonnegative weights {λl(x)}\{\lambda_l(x)\} per preference “dimension,” and an eigenvector embedding head Ev\mathcal{E}_v producing vy∣x\mathbf{v}_{y|x}. Final score: s=vyi∣x⊤D(x) R≻ D(x) vyj∣xs = \mathbf{v}_{y_i|x}^\top \mathbf{D}(x)\,\mathbf{R}^\succ\,\mathbf{D}(x)\,\mathbf{v}_{y_j|x}, with D(x)\mathbf{D}(x) block-diagonal in λl(x)I2\sqrt{\lambda_l(x)}\mathbf{I}_2 — letting the model learn which preference dimensions matter per-context, not just a fixed global set. Embeddings L2-normalized for training stability.

Theoretical results:

  • Theorem 4.4 (full expressiveness): any real skew-symmetric preference matrix over kk items is exactly representable by this embedding-plus-operator construction — the architecture loses nothing relative to a fully general (cyclic-capable) pairwise preference model.
  • Theorem 4.5: explicit construction of such embeddings from a matrix’s spectral decomposition.
  • Theorem 5.1 (GPO convergence): under realizability and bounded scores in [−ρ,ρ][-\rho,\rho], General Preference Optimization achieves an O(1/T)O(1/\sqrt{T}) regret-style bound with β=Θ(T)\beta = \Theta(\sqrt T) — the standard multiplicative-weights-style rate.

General Preference Optimization (GPO). Replaces standard RLHF’s maximize-win-probability objective (P(y≻y′)∈[0,1]\mathbb{P}(y \succ y') \in [0,1]) with maximize-preference-score (unbounded real-valued):
max⁡θ  Ex[Ey∼πθ, y′∼μ[s(y≻y′∣x)]]−β KL(πθ ∥ πref)\max_\theta \; \mathbb{E}_x\big[\mathbb{E}_{y\sim\pi_\theta,\, y'\sim\mu}[s(y\succ y'\mid x)]\big] - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}})
Preference score against the current policy population is estimated by averaging over KK sampled comparisons. Drop-in substitutable into existing pipelines — DPO, PPO, SPPO, IPO — by swapping the scalar-reward or win-probability term for the preference score.

Results.

  • RewardBench: GPM (Gemma-2B) reaches 82.29% average vs. BT’s 74.85% (+7.44pp), with the largest gaps on Chat/Chat-Hard (+12.29pp each); on Llama-3.1-8B the gap narrows to +1.34pp (91.90% vs. 90.56%) — the architectural advantage is largest at smaller scale.
  • Constructed cyclic-preference benchmark (from UltraFeedback, e.g. Instruction-Following ≻ Honesty ≻ Truthfulness ≻ Helpfulness ≻ IF): GPM reaches 100% accuracy; BT is at 50–62.9%, essentially random — direct empirical confirmation of the transitivity-forcing failure mode this paper targets.
  • AlpacaEval 2.0 (downstream alignment, iterative GPO vs. SPPO): +6.13pp length-controlled win rate at the 2B reward-model scale (48.25% vs. 42.12%), narrowing to +0.62pp at 8B.

Numbered technical contributions:

  1. Skew-symmetric preference-embedding framework achieving O(K)O(K) query complexity while representing intransitive/cyclic preferences.
  2. Full-expressiveness theorem for arbitrary skew-symmetric preference structures within this embedding space.
  3. GPM architecture with a context-dependent eigenvalue gate for learned per-context preference-dimension weighting.
  4. GPO — score-based (not win-probability-based) preference optimization with a proven O(1/T)O(1/\sqrt T) convergence rate.
  5. Empirical demonstration of the transitivity failure mode BT models actually exhibit (near-random on constructed cyclic preferences) and its fix.
  6. Drop-in compatibility with existing RLHF pipelines (DPO/PPO/SPPO/IPO) via preference-score substitution.
Source history

The shared explainer cites the source version used to generate it.

  • Original source · Source available for generation · 2026-10-01 UTC