Research, explained

Papers worth understanding.

New papers, shared videos, and podcasts. Explore the source, then go deeper.

Papers use the date they were first added globally. Dates use UTC. Scroll to explore earlier days.

31 papers

October 5, 2026

Quality review

Dust: Pretraining Transformers Without Backpropagation

Samip Dahal, Bishwas Mandal, Serdar Gülbahar, Akshay Vegesna · 2026

**Estimator for a linear layer (Sec. 2.2, Eqs. 1–3).** For $y_t=Wx_t$, jitter every token's output, $y_t\to y_t+\sigma a_t$, $a_t\sim\mathcal N(0,I)$. At each token $s$, the centered loss reduction is $c_s=\tilde\ell_s-\ell_s$, where $\ell_s$ is the perturbed loss and $\tilde\ell_s$ the mean perturbed loss at that token across the draws evaluated together in the same batched forward. The reward of the jitter at token $t$: $$r_t=\sum_{s\ge t}\gamma^{\,s-t}c_s$$ With $\gamma=0$ a jitter is rewarded by its own token only. One independent jitter of all tokens is a **draw**; a population is $K$ draws. The estimated output error and weight gradient: $$\hat g_t=-\frac{1}{K\sigma}\sum_{i=1}^{K}r_t^{(i)}a_t^{(i)},\qquad \widehat G_W=\sum_t\hat g_t\,x_t^\top$$ For an embedding, $x_t$ is one-hot, so the update is a scatter-add of $\hat g_t$ into the token's row. Each draw contributes the gradient plus an error with no preferred direction; the gradient adds linearly over draws and the errors as a square root, so interference vanishes as $K\to\infty$. **Attention internals (Sec. 2.3, Eq. 4).** Q, K, V, gate and value embedding are jittered separately. Only their block's attention output is recomputed, from cached clean activations. Each jitter is scored by alignment with the estimated attention-output error: $$c_s=-\langle\hat g_s,\Delta o_s\rangle$$ where $\Delta o_s$ is the jitter's change to the attention output at token $s$ and $\hat g_s$ comes from Eq. 2. These scores replace the loss reductions in Eq. 1, computed per head. **Interference control (Sec. 2.3).** 1. Layer types are jittered in separate passes with their own noise scales; each block gets its own passes. A draw for block $l$ reruns only blocks $l$ onward from the cached clean forward. 2. Attention internals use the local score above. 3. The LM head is jittered directly on cached logits, re-evaluating cross-entropy on one 256-logit slab per draw (a small fraction of a forward), so it can run a much larger population. **Tuning (Sec. 2.3, App. F).** Two routes for the noise scales, credit decay and draw allocation: (a) short training runs per setting; (b) maximize the cosine to the backprop gradient on one batch (no training). The cosine picks candidates and training decides. Settings largely transfer across budgets and populations. Only keys, values, gates and value embeddings get future-token credit ($\gamma$ near 1). **Cosine law (Sec. 4.2, Eq. 5).** $$\cos(K)=\frac{c_{\max}}{\sqrt{1+c/K}}$$ $c_{\max}$ is the ceiling; $c$ is the population where the cosine reaches $c_{\max}/\sqrt2$. Measured at seven populations from 64 to 131,072, per weight matrix, averaged within a layer type. **Architecture (App. A).** Decoder-only, modded-nanoGPT style: 8 pre-norm blocks, width 512, 8 heads, RMSNorm, rotary embeddings with normalized queries and keys, squared-ReLU MLP of width 2,048, no biases. Causal attention with a 1,024-token sliding window in three of every four blocks and full 2,048 in the rest. Every other block adds a value embedding (a second token embedding, scaled per head by a learned gate on the block input). Each block's input is a learned mix of the previous block's output and the token embedding ($2L$ scalars, trained by weight-space ES). Untied head with capped logits. bfloat16 forward and embeddings. 37.7M parameters. Size study: 2, 4, 8, 16 blocks at widths 128, 256, 512, 1,024, same head dimension. **Training setup (Sec. 3.1, App. F).** - FineWeb, 4,096-token BPE, batch 8 × 2,048 = 16,384 tokens, one epoch, SGD with momentum at a constant rate. - Validation and test: 544 held-out sequences each. Report test loss at the best-validation checkpoint. - Budgets 100k, 1M, 10M, 20M tokens; populations 64, 256, 1k, 4k, 16k. - Starting grids (Table 5). 100k: momentum {0.2, 0.3, 0.5} × rate {0.3–0.6}. 1M–20M: {0.9, 0.95, 0.98} × {0.1, 0.15, 0.2, 0.25}. At 64 draws, 10M: {0.95, 0.98, 0.99} × {0.0125–0.1}; 20M: × {0.00625–0.05}. Grids extend when the best point is on an edge. - Seed 42 runs the grid; points within 0.02 of the best get seeds 43 and 44. Runs are rejected if final validation exceeds the best by more than 0.25, exceed $\ln V$, or produce NaN. - Per-group learning rates (Table 6, main comparison; "base" is the cell's tuned rate): token embedding 1000 for both methods; value embeddings 0.3 and residual scalars 0.03 for Dust, base for backprop. The model-size study used an earlier setting (token embedding 0.3 for Dust and base for backprop; keys 4× and values 8× base). - EGGROLL-Transformer: same architecture, its own grid over step size, momentum, noise scale and fitness shaping. **Population allocation (App. E, Tables 3–4, at $K=16{,}384$).** $K$ counts loss-rewarded draws only. | Site | One draw reruns | Draws (SGD, 1M) | Noise scale | |---|---|---|---| | Writer pair (attn and MLP output projections) | its block onward | 7,040 | 0.2 | | MLP hidden layer | its block onward | 7,040 | 0.2 | | Attention output (before the projection) | its block onward | 1,664 | 0.2; 0.4 in the deeper half | | Token embedding | whole network | 640 | 0.2 (Adam 0.4) | | Q / V / gate / K / value emb, per block | its block's attention | 128 / 1,280 / 384 / 768 / 512 | 0.05 | | Head | one slab's cross-entropy | 65,536 | 0.05 | | Residual scalars | whole network, twice | 8 | 0.03 | - Per-block tilt toward early blocks: writer draws per block at 1M and 10M are 2,432, 2,048, 1,024, 640, 384, 256, 128, 128. Nominal stream ratio (writers + MLP) : attention outputs : embedding = 7 : 0.75 : 0.25 (6 : 1.5 : 0.5 at 100k). The tilt came from a coordinate descent over the allocation at 2k draws. - Populations from 256 up scale every count by $K/16{,}384$. At 64 draws there is a separate recipe (App. E). - Credit decay: $\gamma=0$ for direct loss sites and queries; 0.98 for keys, values, gates and value embeddings (0.99 in the 10M/16k and 20M/16k cells). - Draws per forward per GPU: 1, 2, 8, 8, 16 at $K$ = 64, 256, 1k, 4k, 16k. The repo uses 1, 1, 4 and 8 GPUs at 256 through 16k; every GPU sees the same batch with different noise. **Adam settings (App. F, Table 7, 1M tokens).** $\beta_2=0.95$, $\epsilon=10^{-8}$, no weight decay. Matrix-and-head rate: Dust 0.00175 (64 draws), 0.0015 (256–4k), 0.002 (16k up); backprop 0.002; EGGROLL 0.003. $\beta_1$: Dust 0.7, backprop 0.8, EGGROLL 0.5. Dust embeddings at 300× and residual scalars at 100× the matrix rate. EGGROLL uses z-scored fitness differences. **EGGROLL extrapolation (App. D, Table 2).** Population EGGROLL would need to match Dust at 64 draws, as a multiple of 64: | Fit | 1M | 10M | 20M | |---|---|---|---| | Power law, all five populations | 23,000× | 16,000× | 8,100× | | Power law, one population left out | 11,000× to ∞ | 7,200× to 54,000× | 4,700× to 17,000× | | Line in log population, top three | 8,200× | 3,700× | 3,300× | | Line in log population, top two | 6,000× | 4,100× | 3,100× | **Code.** `dust.py` (algorithm and training loop) and `model.py`; a backprop SGD baseline in `baselines/backprop.py`. Forward hooks on every Linear and Embedding capture inputs and outputs in the clean pass and add jitter during draws; pre-hooks on `attn.c_proj` create the attention-output sites. Supported populations 256, 1,024, 4,096, 16,384. Linux plus an NVIDIA bfloat16 GPU.

Quality review

Octrees as an Explicit 3D Language

Ran Dan, Si-Tong Wei, Pengfei Xiong, Wei Zhang, Yadong Mu, Peng-Shuai Wang · 2026

Existing 3D large language models (LLMs) compromise on two fronts: they compress shapes into latent codebook indices or coordinate text, which removes spatial structure from what the model observes, and they acquire the 3D modality by fine-tuning the backbone, which overwrites its general language ability. We present OctLLM, which addresses both limitations. Geometry enters as an explicit 3D sequence of octree occupancy tokens. However, full octree sequences grow rapidly with depth; OctLLM therefore randomly empties penultimate-level nodes and omits descendants while preserving shape, yielding a shorter coordinate- and depth-anchored Sparse Octree (S-Octree) for position-aware mask-modeling generation and 3D understanding. On the other front, existing methods introduce a new modality with full fine-tuning or LoRA, but full fine-tuning is costly, LoRA limits 3D capacity, and both modify the language pathway. OctLLM instead adds 3D capacity in parameters separate from the pretrained ones: mesh tokens are routed through independent trainable branches in a subset of blocks while text and image tokens retain the frozen vision-language pathway, and the two streams interact through shared self-attention. It trains far fewer parameters than full fine-tuning, yet sets a new state of the art among unified multimodal LLMs, lowering image-to-3D FID by $17.4\%$ and raising render-grounded captioning by $28.7$ points over ShapeLLM-Omni, while matching the backbone on general language benchmarks.

Quality review

Context Language Models

Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh · 2026

We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.

Quality review

HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning

Donggyun Kim, Jack Lu, Chanwoo Kim, Mengye Ren, Seunghoon Hong · 2026

Long-form thinking traces can substantially improve the multi-step reasoning performance of large language models (LLMs), but they introduce high inference-time overhead, with latency dominated by sequential decoding. We propose HyperThink, a text-to-parameter approach that amortizes this reasoning computation into a single query-conditioned parameter update: a lightweight hypernetwork reads the question and predicts updates to a small subset of the base LLM's parameters, while a vector-quantized decoder constrains them to a finite set of reusable patterns to improve robustness and transfer. Trained end-to-end on outputs from the base model itself, HyperThink eliminates long thinking traces at test time: after one hypernetwork forward pass, the adapted model generates a concise step-by-step solution and final answer without an intermediate trace, using far fewer tokens while retaining strong reasoning performance. Empirically, HyperThink improves the low-latency region of the accuracy-latency trade-off on mathematical and general reasoning tasks, with its strongest gains in the near-non-thinking regime.

October 3, 2026

Quality review

How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

Lin Chen, Bolin Ni, Qi Yang, Lan Jiang, Kun Ding, Xiaoran Fan, Hower Yang, Ying Wang, Shiming Xiang · 2026

Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss--compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around $10^{22}$ FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.

October 2, 2026

Video and podcast ready

Local Support Learning

Assaf Ben-Kish, Akarsh Kumar, James Glass, Raja Giryes · 2026

We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.

Quality review

Frontier Learning: Training LLM Reasoners at the Edge of Capability

Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic, Aurelien Lucchi · 2026

Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.

Quality review

Invent a Dataset: Measuring Dataset Generation Abilities with Zero Seed Data

Shivalika Singh, Andrija Djurisic, Gbemileke Onilude, Sudip Roy, Sara Hooker · 2026

**Problem.** In a zero-data regime the practitioner has only a natural-language description of the capability: no seed prompts and no examples. Retrieved datasets rarely meet format, language or tone constraints. Seed-based synthesis ([[Reading/Research/Self-Instruct - Aligning Language Models with Self-Generated Instructions|Self-Instruct]], Alpaca, Evol-Instruct) assumes seeds. Raw prompting loses diversity at scale, and proprietary model outputs are restricted for training by terms of service. **System under test.** Invent-a-Dataset ("Invent API"), Adaption's prompt-based endpoint: dataset description in, dataset out, in a single call. It supports 244 languages (per the cited docs) and produces instruction and preference datasets for SFT and alignment. It does not yet produce tool-call traces or agent trajectories, and it is text only. It is released with a licence that allows commercial training use. Its internals are not described. **Baselines.** - Proprietary: Claude Opus 5, GPT-5.6 Sol, Gemini-3.1-pro. Their terms forbid training on outputs, so they are included as an evaluation upper bound. - Open weights: GLM-5.3 and DeepSeek-V4-Pro-0813. Their licences permit training on outputs. - All are called through OpenRouter with default settings: temperature 1.0 where settable, reasoning disabled, maximum output 128K tokens. **Generation strategy (Fig. 2, Medical QA, τ = 0.1, traced).** Baselines are run three ways: | Strategy | GPT-5.6 Sol | Gemini 3.1 Pro | GLM-5.3 | |---|---|---|---| | one-by-one | ≈0.03 | ≈0.04 | ≈0.02 | | batch of 100 | ≈0.22 | ≈0.23 | ≈0.27 | | max batch | ≈0.43 | ≈0.35 | ≈0.42 | Max batch is the default for every baseline. Invent is always a single call. **Datasets.** Eight queries (Tables 3 and 8): African QA, Legal QA, Serbian Car Ads, Customer Support Banking, Hindi News, Customer Support General, Medical QA, Hotel Reviews. Five have constrained variants: - African QA: multi-turn teacher–student exam conversations, OpenAI-format prompts. - Legal QA: four-option multiple choice, answer limited to A–D. - Serbian Car Ads: ad summary → JSON with `vehicle_description`, `service_history`, `equipment_packages`. - Customer Support Banking: conversations with frustrated users, agent always welcoming and polite. - Hindi News: sports only; prompt = headline, completion = summary. Each query is generated at 200, 2,000, 5,000 and 20,000 samples. **Metrics.** - **DCScore** ([[Reading/Research/DCScore - Measuring Diversity in Synthetic Datasets|Zhu et al. 2025]]), with embeddings from Qwen-3-4B-embedding (multilingual, 32K context). Each sample is treated as its own class; a diverse set classifies every sample to itself. $$K=\mathrm{Kernel}\big(\Phi(\{\tilde T_i\}_{i=1}^n)\big),\qquad P[i,j]=\frac{\exp(K[i,j]/\tau)}{\sum_{j'}\exp(K[i,j']/\tau)},\qquad \mathrm{DCScore}(D)=\operatorname{tr}(P)=\sum_{i=1}^n P[i,i]$$ τ = 0.1 throughout, except Fig. 1 (τ = 0.05). Smaller τ sharpens the classification. - **Lexical metrics, on prompts, on a fixed 2,000-prompt random sample:** n-gram diversity $\mathrm{NGD}=\sum_{n=1}^{4}\frac{\#\text{unique }n\text{-grams}}{\#\text{total }n\text{-grams}}$ (higher is better); compression ratio CR = raw size / gzip size (lower means less redundant); exact-duplicate rate (lower is better). - **Quality and relevance:** an LLM judge scores each sample in one call and returns JSON `{relevance_score, prompt_quality_score, completion_quality_score, relevant, reason}`, with scores 0–10 and reason ≤ 30 words. - Relevance criteria, in priority order: (A) task and topic, (B) language, then (C) explicitly stated constraints, then (D) a usable standalone prompt. - Caps: failing A or B ≤ 2, a C violation ≤ 3, failing D ≤ 4. A score of 7+ needs full compliance. - Quality is judged within any form the request mandates, so a requested terse answer isn't penalized. The judge model is not named. - **Language drift:** `langid` on Hindi News, 2K samples drawn from 20K. - **Downstream ranking:** Gemini 3.1 Pro sees all six responses to a prompt in one call, with order shuffled per prompt, and ranks them. - Criteria: accuracy is critical, and any factual error ranks below every correct answer. Instruction following is critical, completeness vs concision is medium weight, and clarity breaks ties. - Test prompts are in-house held-out medical queries "sampled from the real-world distribution". The count is not given. **Post-training setup.** - Base models: Llama-3.3-70B-Instruct, Gemma-4-31B-it, Qwen3.5-9B. - Each is trained with LoRA SFT on each generator's 20K Medical QA set. - Hyperparameters come from one Autoscientist search on one unnamed Medical QA variant, then are applied to every variant. - Each base model's comparison has six entries: the untuned base plus five fine-tunes. **Main results: 5K samples, unconstrained, averaged over 8 datasets.** | Generator | Quality (Fig. 6a) | Prompt / completion quality (traced) | Relevance (Fig. 6b) | DCScore τ = 0.05 (Fig. 1, traced) | Invent's diversity lead (text) | |---|---|---|---|---|---| | Invent API | **7.90** | 7.35 / 8.45 | 7.47 | ≈0.79 | | | Claude Opus 5 | 6.76 | 5.75 / 7.8 | **8.86** | ≈0.64 | +24% | | GLM-5.3 | 6.73 | 5.85 / 7.65 | 8.66 | ≈0.665 | +19% | | Gemini-3.1-pro | 5.26 | 5.1 / 5.45 | 7.51 | ≈0.60 | +32% | | GPT-5.6 Sol | 5.13 | 4.75 / 5.5 | 7.11 | ≈0.61 | +29% | | DeepSeek-V4-Pro | ≈5.2 (traced) | 5.15 / 5.3 | 7.44 | ≈0.51 | +55% | **Scale (Fig. 5).** - Quality, Invent vs GLM-5.3: 8.43 vs 7.03 at 200, 8.09 vs 6.76 at 2K, 7.90 vs 6.73 at 5K, 8.15 vs 6.55 at 20K. - Diversity at τ = 0.1, each measured on a 2K subsample, from 2K to 20K: Invent 0.250 → 0.268 (+7%); GLM-5.3 0.213 → 0.196 (−8%); GPT-5.6 Sol −7%; Claude Opus 5 −15%; DeepSeek-V4-Pro +0.4%. - Invent's lead over GLM-5.3 grows from +17% at 2K to +37% at 20K. **Constraints (Fig. 4, 20K, τ = 0.1).** | Generator | Unconstrained | Constrained | Change | |---|---|---|---| | GPT-5.6 Sol | 0.0328 | 0.0257 | −22% | | DeepSeek-V4-Pro | 0.0264 | 0.0217 | −18% | | Gemini-3.1-pro | 0.0306 | 0.0254 | −17% | | Claude Opus 5 | 0.0248 | 0.0210 | −15% | | GLM-5.3 | 0.0317 | 0.0271 | −15% | | Invent API | 0.0428 | 0.0428 | 0% | Under constraints Invent leads GLM-5.3 by 58%. **Lexical diversity (Table 6, unconstrained; 2K / 5K / 20K).** | Generator | NGD ↑ | CR ↓ | Dup % ↓ | |---|---|---|---| | Invent API | **1.75 / 1.82 / 1.87** | **3.67 / 3.51 / 3.47** | **0.0 / 0.0 / 0.0** | | GLM-5.3 | 1.57 / 1.70 / 1.69 | 4.70 / 3.95 / 3.96 | 6.2 / 6.8 / 8.9 | | Gemini-3.1-pro | 1.53 / 1.62 / 1.66 | 4.59 / 4.22 / 4.08 | 7.5 / 6.8 / 6.5 | | GPT-5.6 Sol | 1.60 / 1.65 / 1.59 | 4.33 / 4.33 / 4.39 | 9.6 / 10.4 / 12.4 | | Claude Opus 5 | 1.55 / 1.54 / 1.48 | 4.27 / 4.25 / 4.28 | 8.5 / 7.9 / 8.4 | | DeepSeek-V4-Pro | 1.07 / 1.07 / 1.07 | 5.50 / 5.53 / 5.51 | 19.2 / 18.5 / 18.8 | Constrained setting (Table 11): Invent's duplicate rate is 0.1% at every size, and the baselines' range from 0.9% to 11.6%. GLM-5.3 beats Invent on NGD at 5K (1.80 vs 1.69). **Multilingual (Fig. 3, 5K).** - Serbian and Hindi: Invent's diversity is ≈0.082, about double the next best. GPT-5.6 Sol scores higher on quality (≈8.05) at less than half the diversity. - Six English datasets: Invent has quality ≈7.5 and diversity ≈0.149, vs GLM-5.3 at 0.135. - Going from English to non-English, Invent loses ≈44% of its diversity; the baselines lose 60–75%. - Language drift on Hindi News (Table 10): | Generator | Non-Hindi outputs (of 2,000) | Rate | Notes | |---|---|---|---| | Invent API | 3 | 0.15% | | | Claude Opus 5 | 7 | 0.35% | | | GLM-5.3 | 49 | 2.45% | 38 in English | | GPT-5.6 Sol | 50 | 2.5% | | | Gemini 3.1 Pro | 77 | 3.85% | | | DeepSeek-V4-Pro | 145 | 7.25% | 144 in Marathi | **Post-training (top-1 rank share; Figs. 7 and 10; ≈ = traced).** | Base model | Invent FT | Untuned base | Claude Opus 5 FT | GLM-5.3 FT | GPT-5.6 Sol FT | DeepSeek-V4-Pro FT | |---|---|---|---|---|---|---| | Llama-3.3-70B | **54%** | 25% | 9% | ≈7% | ≈3% | ≈2% | | Gemma-4-31B-it | **41%** | 21% | 10% | 8% | 8% | 12% | | Qwen3.5-9B | 38% | **44%** | ≈4% | ≈6% | ≈1% | ≈5% | **Stated limitations.** Instruction and preference data only; no tool-call traces or agentic trajectories; text only.

September 18, 2026

Video and podcast ready

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment

Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, Quanquan Gu · 2024

**Problem.** Bradley-Terry (BT) reward models assign each response a scalar reward, which imposes a total order and therefore can't represent intransitive/cyclic preferences (e.g., a rater or population preferring $y_1 \succ y_2 \succ y_3 \succ y_1$ across different evaluation axes). Fully general pairwise preference models (PairRM/PairPM) can capture cycles but require $O(K^2)$ pairwise comparisons for $K$ candidate responses — expensive at test-time-scaling inference. **Preference embedding formalism.** Embed each response into $\mathbb{R}^{2k}$ via $\mathbf{v}_{y|x}$, and score pairs through a fixed block-diagonal **skew-symmetric** operator $\mathbf{R}^\succ$: $$s(y_i \succ y_j \mid x) = \langle \mathbf{R}^\succ \mathbf{v}_{y_i|x}, \mathbf{v}_{y_j|x}\rangle$$ Skew-symmetry gives $s(y_i \succ y_j) = -s(y_j \succ y_i)$ for free — the antisymmetry a preference relation needs — while the embedding space itself is what encodes cycles, since bilinear forms over $\geq 2$ dimensions aren't constrained to a total order the way a scalar is. Query cost stays $O(K)$ (one embedding per response), matching BT's cost while gaining PairRM's expressiveness. A complex-number reading is offered too: scores equal the imaginary part of a Hermitian inner product of complex-valued embeddings. **GPM architecture.** Two heads on top of an LM backbone: an **eigenvalue scale gate** $\mathcal{G}_\lambda$ producing context-dependent nonnegative weights $\{\lambda_l(x)\}$ per preference "dimension," and an **eigenvector embedding head** $\mathcal{E}_v$ producing $\mathbf{v}_{y|x}$. Final score: $s = \mathbf{v}_{y_i|x}^\top \mathbf{D}(x)\,\mathbf{R}^\succ\,\mathbf{D}(x)\,\mathbf{v}_{y_j|x}$, with $\mathbf{D}(x)$ block-diagonal in $\sqrt{\lambda_l(x)}\mathbf{I}_2$ — letting the model learn *which* preference dimensions matter per-context, not just a fixed global set. Embeddings L2-normalized for training stability. **Theoretical results:** - **Theorem 4.4 (full expressiveness):** any real skew-symmetric preference matrix over $k$ items is exactly representable by this embedding-plus-operator construction — the architecture loses nothing relative to a fully general (cyclic-capable) pairwise preference model. - **Theorem 4.5:** explicit construction of such embeddings from a matrix's spectral decomposition. - **Theorem 5.1 (GPO convergence):** under realizability and bounded scores in $[-\rho,\rho]$, General Preference Optimization achieves an $O(1/\sqrt{T})$ regret-style bound with $\beta = \Theta(\sqrt T)$ — the standard multiplicative-weights-style rate. **General Preference Optimization (GPO).** Replaces standard RLHF's maximize-win-probability objective ($\mathbb{P}(y \succ y') \in [0,1]$) with maximize-preference-*score* (unbounded real-valued): $$\max_\theta \; \mathbb{E}_x\big[\mathbb{E}_{y\sim\pi_\theta,\, y'\sim\mu}[s(y\succ y'\mid x)]\big] - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}})$$ Preference score against the current policy population is estimated by averaging over $K$ sampled comparisons. Drop-in substitutable into existing pipelines — DPO, PPO, SPPO, IPO — by swapping the scalar-reward or win-probability term for the preference score. **Results.** - **RewardBench:** GPM (Gemma-2B) reaches 82.29% average vs. BT's 74.85% (+7.44pp), with the largest gaps on Chat/Chat-Hard (+12.29pp each); on Llama-3.1-8B the gap narrows to +1.34pp (91.90% vs. 90.56%) — the architectural advantage is largest at smaller scale. - **Constructed cyclic-preference benchmark** (from UltraFeedback, e.g. Instruction-Following ≻ Honesty ≻ Truthfulness ≻ Helpfulness ≻ IF): GPM reaches **100% accuracy**; BT is at 50–62.9%, essentially random — direct empirical confirmation of the transitivity-forcing failure mode this paper targets. - **AlpacaEval 2.0** (downstream alignment, iterative GPO vs. SPPO): +6.13pp length-controlled win rate at the 2B reward-model scale (48.25% vs. 42.12%), narrowing to +0.62pp at 8B. **Numbered technical contributions:** 1. Skew-symmetric preference-embedding framework achieving $O(K)$ query complexity while representing intransitive/cyclic preferences. 2. Full-expressiveness theorem for arbitrary skew-symmetric preference structures within this embedding space. 3. GPM architecture with a context-dependent eigenvalue gate for learned per-context preference-dimension weighting. 4. GPO — score-based (not win-probability-based) preference optimization with a proven $O(1/\sqrt T)$ convergence rate. 5. Empirical demonstration of the transitivity failure mode BT models actually exhibit (near-random on constructed cyclic preferences) and its fix. 6. Drop-in compatibility with existing RLHF pipelines (DPO/PPO/SPPO/IPO) via preference-score substitution.