Dust: Pretraining Transformers Without Backpropagation

Estimator for a linear layer (Sec. 2.2, Eqs. 1–3). For $y_t=Wx_t$, jitter every token’s output, $y_t\to y_t+\sigma a_t$, $a_t\sim\mathcal N(0,I)$. At each token $s$, the centered loss reduction is $c_s=\tilde\ell_s-\ell_s$, where $\ell_s$ is the perturbed loss and $\tilde\ell_s$ the mean perturbed loss at that token across the draws evaluated together in the same batched forward. The reward of the jitter at token $t$:
$r_t=\sum_{s\ge t}\gamma^{\,s-t}c_s$
With $\gamma=0$ a jitter is rewarded by its own token only. One independent jitter of all tokens is a draw; a population is $K$ draws. The estimated output error and weight gradient:
$\hat g_t=-\frac{1}{K\sigma}\sum_{i=1}^{K}r_t^{(i)}a_t^{(i)},\qquad \widehat G_W=\sum_t\hat g_t\,x_t^\top$
For an embedding, $x_t$ is one-hot, so the update is a scatter-add of $\hat g_t$ into the token’s row. Each draw contributes the gradient plus an error with no preferred direction; the gradient adds linearly over draws and the errors as a square root, so interference vanishes as $K\to\infty$.

Attention internals (Sec. 2.3, Eq. 4). Q, K, V, gate and value embedding are jittered separately. Only their block’s attention output is recomputed, from cached clean activations. Each jitter is scored by alignment with the estimated attention-output error:
$c_s=-\langle\hat g_s,\Delta o_s\rangle$
where $\Delta o_s$ is the jitter’s change to the attention output at token $s$ and $\hat g_s$ comes from Eq. 2. These scores replace the loss reductions in Eq. 1, computed per head.

Interference control (Sec. 2.3).

1. Layer types are jittered in separate passes with their own noise scales; each block gets its own passes. A draw for block $l$ reruns only blocks $l$ onward from the cached clean forward.
2. Attention internals use the local score above.
3. The LM head is jittered directly on cached logits, re-evaluating cross-entropy on one 256-logit slab per draw (a small fraction of a forward), so it can run a much larger population.

Tuning (Sec. 2.3, App. F). Two routes for the noise scales, credit decay and draw allocation: (a) short training runs per setting; (b) maximize the cosine to the backprop gradient on one batch (no training). The cosine picks candidates and training decides. Settings largely transfer across budgets and populations. Only keys, values, gates and value embeddings get future-token credit ($\gamma$ near 1).

Cosine law (Sec. 4.2, Eq. 5).
$\cos(K)=\frac{c_{\max}}{\sqrt{1+c/K}}$
$c_{\max}$ is the ceiling; $c$ is the population where the cosine reaches $c_{\max}/\sqrt2$. Measured at seven populations from 64 to 131,072, per weight matrix, averaged within a layer type.

Architecture (App. A). Decoder-only, modded-nanoGPT style: 8 pre-norm blocks, width 512, 8 heads, RMSNorm, rotary embeddings with normalized queries and keys, squared-ReLU MLP of width 2,048, no biases. Causal attention with a 1,024-token sliding window in three of every four blocks and full 2,048 in the rest. Every other block adds a value embedding (a second token embedding, scaled per head by a learned gate on the block input). Each block’s input is a learned mix of the previous block’s output and the token embedding ($2L$ scalars, trained by weight-space ES). Untied head with capped logits. bfloat16 forward and embeddings. 37.7M parameters. Size study: 2, 4, 8, 16 blocks at widths 128, 256, 512, 1,024, same head dimension.

Training setup (Sec. 3.1, App. F).

• FineWeb, 4,096-token BPE, batch 8 × 2,048 = 16,384 tokens, one epoch, SGD with momentum at a constant rate.
• Validation and test: 544 held-out sequences each. Report test loss at the best-validation checkpoint.
• Budgets 100k, 1M, 10M, 20M tokens; populations 64, 256, 1k, 4k, 16k.
• Starting grids (Table 5). 100k: momentum {0.2, 0.3, 0.5} × rate {0.3–0.6}. 1M–20M: {0.9, 0.95, 0.98} × {0.1, 0.15, 0.2, 0.25}. At 64 draws, 10M: {0.95, 0.98, 0.99} × {0.0125–0.1}; 20M: × {0.00625–0.05}. Grids extend when the best point is on an edge.
• Seed 42 runs the grid; points within 0.02 of the best get seeds 43 and 44. Runs are rejected if final validation exceeds the best by more than 0.25, exceed $\ln V$, or produce NaN.
• Per-group learning rates (Table 6, main comparison; “base” is the cell’s tuned rate): token embedding 1000 for both methods; value embeddings 0.3 and residual scalars 0.03 for Dust, base for backprop. The model-size study used an earlier setting (token embedding 0.3 for Dust and base for backprop; keys 4× and values 8× base).
• EGGROLL-Transformer: same architecture, its own grid over step size, momentum, noise scale and fitness shaping.

Population allocation (App. E, Tables 3–4, at $K=16{,}384$). $K$ counts loss-rewarded draws only.

Site	One draw reruns	Draws (SGD, 1M)	Noise scale
Writer pair (attn and MLP output projections)	its block onward	7,040	0.2
MLP hidden layer	its block onward	7,040	0.2
Attention output (before the projection)	its block onward	1,664	0.2; 0.4 in the deeper half
Token embedding	whole network	640	0.2 (Adam 0.4)
Q / V / gate / K / value emb, per block	its block’s attention	128 / 1,280 / 384 / 768 / 512	0.05
Head	one slab’s cross-entropy	65,536	0.05
Residual scalars	whole network, twice	8	0.03

• Per-block tilt toward early blocks: writer draws per block at 1M and 10M are 2,432, 2,048, 1,024, 640, 384, 256, 128, 128. Nominal stream ratio (writers + MLP) : attention outputs : embedding = 7 : 0.75 : 0.25 (6 : 1.5 : 0.5 at 100k). The tilt came from a coordinate descent over the allocation at 2k draws.
• Populations from 256 up scale every count by $K/16{,}384$. At 64 draws there is a separate recipe (App. E).
• Credit decay: $\gamma=0$ for direct loss sites and queries; 0.98 for keys, values, gates and value embeddings (0.99 in the 10M/16k and 20M/16k cells).
• Draws per forward per GPU: 1, 2, 8, 8, 16 at $K$ = 64, 256, 1k, 4k, 16k. The repo uses 1, 1, 4 and 8 GPUs at 256 through 16k; every GPU sees the same batch with different noise.

Adam settings (App. F, Table 7, 1M tokens). $\beta_2=0.95$, $\epsilon=10^{-8}$, no weight decay. Matrix-and-head rate: Dust 0.00175 (64 draws), 0.0015 (256–4k), 0.002 (16k up); backprop 0.002; EGGROLL 0.003. $\beta_1$: Dust 0.7, backprop 0.8, EGGROLL 0.5. Dust embeddings at 300× and residual scalars at 100× the matrix rate. EGGROLL uses z-scored fitness differences.

EGGROLL extrapolation (App. D, Table 2). Population EGGROLL would need to match Dust at 64 draws, as a multiple of 64:

Fit	1M	10M	20M
Power law, all five populations	23,000×	16,000×	8,100×
Power law, one population left out	11,000× to ∞	7,200× to 54,000×	4,700× to 17,000×
Line in log population, top three	8,200×	3,700×	3,300×
Line in log population, top two	6,000×	4,100×	3,100×

Code. dust.py (algorithm and training loop) and model.py; a backprop SGD baseline in baselines/backprop.py. Forward hooks on every Linear and Embedding capture inputs and outputs in the clean pass and add jitter during draws; pre-hooks on attn.c_proj create the attention-output sites. Supported populations 256, 1,024, 4,096, 16,384. Linux plus an NVIDIA bfloat16 GPU.