Louis-Antoine Mullie, MD
About Blog

Blog

DUST estimates the gradient of a blurred loss

DUST pretrains transformers using only forward passes. In expectation, it follows the gradient that backprop would compute for a blurred version of the loss, with credit to later tokens discounted or cut off.

Pick one layer of a language model, add a little Gaussian noise to its output, and run the forward pass to see whether the loss went up or down. Repeat this a few thousand times with fresh noise, then average the noise draws, weighting each one by how much it raised the loss. The weighted average is an estimate of the gradient, and an optimizer can use it in place of the gradient from backprop.

Methods like this are called zeroth order because they use only values of the loss and run entirely on forward passes.

Zeroth-order methods

The idea goes back to Kiefer and Wolfowitz (1952), who estimated a derivative from noisy finite differences, which in many dimensions means perturbing one coordinate at a time. Forty years later Spall (1992) perturbed all coordinates at once along a random direction, Williams (1992) applied the same idea to reinforcement learning, and Widrow and Lehr (1990) put the noise on the neurons, which became known as node perturbation. Evolution strategies later ran it on over a thousand parallel workers, and MeZO uses it to fine-tune language models.

Because the method needs only forward passes, it can train networks containing operations that are hard to backpropagate through, such as discrete choices, calls to external programs, or recurrences too long to unroll. When it perturbs the weights, as MeZO does, it also uses no more memory than inference.

Each forward pass returns a single number, however, and in \(d\) dimensions a random direction is nearly orthogonal to the gradient, so the variance of the estimate grows with \(d\) (Nesterov and Spokoiny, 2017). A model with a hundred million weights would need a population of roughly a hundred million, and the common conclusion was that these methods suit small networks and fine-tuning, and that pretraining requires backprop (Lillicrap et al., 2020).

Per-token activation noise

DUST (paper, GitHub, tweet), from Samip Dahal and colleagues at Q Labs, applies this idea to pretraining transformers. It puts the noise on the activations, as node perturbation does, and draws independent noise at every token. A sequence of 2,048 tokens gives 2,048 separate perturbations from one forward pass, which the authors call a virtual population.

At 100k and 1M training tokens, DUST reaches a lower loss than backprop trained on the same tokens. It needs a few hundred draws per update at 100k and a thousand at 1M, and each draw costs about one forward pass, so it uses far more compute than backprop. From 1M tokens up, the authors extrapolate that it is a thousand to ten thousand times as efficient as their transformer implementation of EGGROLL, a state-of-the-art evolution strategy that perturbs weights. Larger models also do better at a fixed population size. At most population sizes, a 243M-parameter model beats one 120 times smaller, which is the opposite of what the dimension argument predicts.

The gradient estimator

For a linear layer \(y_t = W x_t\), DUST jitters the output at every token, \(y_t \to y_t + \sigma a_t\) with \(a_t \sim \mathcal N(0, I)\), and runs the forward pass. In a causal model, a jitter at token \(t\) can change the loss at \(t\) and, through attention, the losses at later tokens, but not the earlier ones, so DUST scores each jitter by a discounted sum of the losses from \(t\) on:

\[F_t = \sum_{s \ge t} \gamma^{\,s-t}\, \ell_s .\]

The discount \(\gamma\) controls how many later tokens count. With \(\gamma = 0\), a jitter is scored only by the loss at its own token, and with \(\gamma = 1\), every later loss counts in full, as in backprop.

Counts toward the score Full loss Not counted
The bars show the loss at each token of a sentence. A jitter at the highlighted token is scored by the loss at that token plus a \(\gamma\)-discounted share of each later loss, and earlier tokens do not count. Dragging the slider changes \(\gamma\). The released code sets it to 0 for most layers and to 0.98 for attention keys, values, value embeddings, and their gates, though it scores those differently. Clicking a word moves the jitter.

With \(K\) independent draws, DUST subtracts the population average \(\bar F_t\) from each score and weights the corresponding noise by the difference:

\[\hat g_t = \frac{1}{K\sigma} \sum_{i=1}^{K} \bigl(F_t^{(i)} - \bar F_t\bigr)\, a_{i,t} .\]

Draws that scored above the average pull the estimate toward their noise, and draws that scored below it push the estimate away, so the sum estimates the error signal at the output of the layer. Multiplying it by the layer input and summing over tokens gives a weight gradient, \(\hat G_W = \sum_t \hat g_t\, x_t^\top\). Backprop builds the same outer product from the same input, but it computes the output error with the chain rule, and DUST measures it from forward passes.

Below average Above average DUST estimate Target slope
DUST estimate0.00
Target slope0.00
The curve is a one-dimensional slice of the score as a function of the layer output. Each dot is a jitter, blue if it scored below the average and orange if above, and the purple line is the slope DUST estimates from them. Each press of Resample draws a new set of jitters and adds the resulting estimate to the line below. The dashed target is the slope of the blurred score rather than the tangent of the curve, and the estimates center on \((K-1)/K\) of it. The next section explains both, and the spread of the estimates shrinks as \(K\) grows.

Expectation of the estimator

The estimate has a simple expectation:

\[\begin{aligned} \mathbb E\bigl[\hat g_t\bigr] &= \frac{K-1}{K}\, \nabla_{y_t} F_{t,\sigma}(y), \\[4pt] F_{t,\sigma}(y) &= \mathbb E_A\bigl[F_t(y + \sigma A)\bigr], \end{aligned}\]

and its variance falls as \(1/K\). In expectation, DUST follows the gradient of a blurred loss, scaled by \((K-1)/K\). The blurred loss is the expected loss when noise of size \(\sigma\) is added to the layer output.

The proof has two steps. The noise in each draw is independent of the other draws, so it is correlated only with its own score, and Stein’s lemma turns that correlation into \(\sigma\) times the gradient of the blurred score. Each draw is also one of the \(K\) terms in the average it is compared with, so it cancels \(1/K\) of its own signal.

Sharp loss Blurred loss Backprop slope
Backprop slope0.00
Blurred slope0.00
Jitters of size \(\sigma\) average out bumps narrower than \(\sigma\), so they measure the slope of the blurred loss (teal), which is the average of the loss shifted by every possible jitter (faint copies). Backprop measures the slope of the sharp loss (dashed), and the two agree as \(\sigma\) goes to zero.

As \(\sigma\) shrinks, the blurred gradient approaches the ordinary gradient, with an error of order \(\sigma^2\). With \(\gamma = 1\), the ordinary gradient at \(y_t\) is the output error that backprop computes, so the expected weight update is \(\partial L/\partial W\) up to the \((K-1)/K\) factor. DUST uses only forward passes, but in expectation it follows the backprop gradient of a blurred loss, which is also discounted when \(\gamma < 1\).

The \((K-1)/K\) factor

The factor \((K-1)/K\) comes from the baseline. Each draw is compared with an average that includes the draw itself, so a high score also raises the baseline, and the margin over the baseline is \((K-1)/K\) of the margin over the average of the other draws. With two draws, the expected estimate is half the gradient.

Raw estimates Corrected Average of 4,000 runs Target slope
Raw average0.00
Corrected average0.00
Standard deviation±0.00
The histograms show 4,000 runs of the estimator. The raw estimates average \((K-1)/K\) of the target, multiplying them by \(K/(K-1)\) centers them on the target, and both histograms narrow as \(K\) grows.

With 16,384 draws the factor is \(1 - 1/16{,}384\), which differs from one by less than 0.01%. The released code, however, subtracts the baseline within small chunks of draws that share a forward pass, two to sixteen draws at a time (DRAW_CHUNK_SIZES). At a population of 256 each chunk holds two draws, so the estimates built from these chunks are half the gradient they target. The tuned learning rate absorbs most of the difference, but the token and value embeddings keep fixed learning rates, so for them it does not. Comparing each draw with the average of the other draws would remove the factor, since it multiplies the estimate by \(K/(K-1)\).

Consequences

Cosine similarity and population size

Q Labs measure the cosine similarity between the DUST estimate and the backprop gradient as the population grows, and fit \(\cos(K) = c_{\max}/\sqrt{1 + c/K}\). In high dimensions, any estimator whose mean is fixed and whose variance falls as \(1/K\) approximately follows this curve, and the theorem identifies both constants. The ceiling \(c_{\max}\) is the cosine between the backprop gradient and the blurred, discounted gradient that DUST targets, and \(c\) is the noise-to-signal ratio of a single draw, which is also the population at which the cosine reaches \(c_{\max}/\sqrt{2}\). In a toy simulation, one value of \(c\) fits every population from 16 to 16,384 and matches the ratio measured on single draws. The announcement describes backprop-like gradients as emerging at large populations. According to the theorem, the expected update points in the same direction at every population size, and a larger population only reduces the noise around it, as Section 2.2 of the paper also says.

The blur as a curvature penalty

For small \(\sigma\) the blurred loss is the loss plus a curvature penalty,

\[F_{t,\sigma}(y) \approx F_t(y) + \frac{\sigma^2}{2}\, \operatorname{tr} \nabla_y^2 F_t(y) ,\]

so in expectation DUST follows the gradient of the loss plus a penalty on the trace of its Hessian with respect to the layer activations. The penalty favors flat regions, which is a known effect of noise injection (Bishop, 1995; Orvieto et al., 2022). The first open question in the paper is whether DUST picks up “higher-order curvature that pulls the search toward flat regions,” and this term is one candidate. It can be tested by training with backprop while adding the same noise to the activations. That run follows the blurred gradient in expectation, without the discount and with much lower variance than DUST. If it also reaches a lower loss than plain backprop, the blur is enough to explain the DUST result. If it does not, the discount or the noise in the estimate is the more likely explanation.

Non-differentiable networks

The identity behind the theorem requires differentiating only the Gaussian density, so the blurred loss is smooth even when \(F\) has jumps and kinks, and its gradient is well defined for networks that use argmax, sampling, tool calls, or external programs. Zeroth-order methods have always been able to train such networks, and the theorem identifies the gradient that DUST follows in expectation in that case.

Relation to RetNet and Gated DeltaNet-2

The score can also be written as a recurrence that runs from the last token back to the first,

\[F_t = \ell_t + \gamma\, F_{t+1} ,\]

which has the same form as the recurrence in RetNet (Sun et al., 2023), \(S_t = \gamma\, S_{t-1} + k_t v_t^\top\), run backward in time. In RetNet, each token adds the outer product of its key and value to a state that decays by a factor of \(\gamma\) per token. In the DUST score, the state is a single number and each token adds its loss.

The gradient of the score is a discounted sum of the gradients of the later losses,

\[\nabla_{y_t} F_t = \sum_{s \ge t} \gamma^{\,s-t}\, \frac{\partial \ell_s}{\partial y_t} ,\]

and each \(\partial \ell_s/\partial y_t\) is a sum over paths through the network from token \(t\) to token \(s\). A path moves between tokens only in layers that mix tokens, such as attention, and its steps between tokens add up to \(s - t\) regardless of how many layers it crosses. If the backward pass multiplies each step from token \(u\) to token \(u'\) by \(\gamma^{\,u'-u}\), every path from \(t\) to \(s\) is multiplied by \(\gamma^{\,s-t}\). The gradient of the score is therefore the gradient that backprop would compute for the total loss if each step between tokens were damped this way in the backward pass, with the forward pass unchanged. For softmax attention, where token \(s\) reads token \(t\) with weight \(A_{st}\), the gradient that the backward pass sends to the key and value of token \(t\) uses \(\gamma^{\,s-t} A_{st}\) in place of \(A_{st}\), which is the attention matrix multiplied elementwise by the RetNet decay mask. The query of token \(s\) stays within its own token, so its gradient is unchanged. With \(\gamma = 0\), only the diagonal remains, so each token receives credit for its own loss alone. With \(\gamma = 1\), the attention matrix is unchanged.

Weight in backward pass Weight in forward pass Masked
Column weight kept0%
The grid shows the attention weights of one head over the sentence, with the reading tokens down the side and the tokens they read along the bottom, and the area of each square is proportional to its weight. The dashed outline is the weight \(A_{st}\) in the forward pass, and the filled square is the weight \(\gamma^{\,s-t} A_{st}\) in the backward pass. Dragging the slider changes \(\gamma\), and clicking a word moves the jitter. The numbers on the right are the discount factors in the column of the jittered token, and the readout is the total backward weight in that column as a percentage of the forward weight.

The damping is simpler to state for recurrent layers. Gated DeltaNet-2 (Hatamizadeh et al., 2026), a recent linear-attention layer from NVIDIA, uses a fixed-size state in place of a growing cache of keys and values. At each token it decays the state channel by channel, erases what is stored along the current key, and writes the new value, with a separate gate for each operation:

\[S_t = \bigl(I - k_t (b_t \odot k_t)^\top\bigr) \operatorname{Diag}(\alpha_t)\, S_{t-1} + k_t (w_t \odot v_t)^\top ,\]

where \(\alpha_t\) is the decay gate, \(b_t\) the erase gate, and \(w_t\) the write gate. A path through the state passes from one token to the next through one application of this update, so damping each step by \(\gamma\) replaces \(\operatorname{Diag}(\alpha_t)\) with \(\operatorname{Diag}(\gamma \alpha_t)\). For the recurrence, the DUST discount is equivalent to backprop with every decay multiplied by \(\gamma\) in the backward pass. At \(\gamma = 0.98\), this amounts to forgetting an extra 2% per token, a shift of \(\ln \gamma \approx -0.02\) in the log-decay that the layer already learns. The layer also passes its queries, keys, and values through short causal convolutions, which mix a few neighboring tokens, so a convolution tap that reaches back \(j\) tokens is damped by \(\gamma^{\,j}\). Together with the theorem, this means that in expectation, scoring jitters by \(F_t\) gives the backprop gradient of a blurred loss through a network whose memory decays by an extra factor of \(\gamma\) per token in the backward pass. The released code applies the discount more narrowly, as the next section describes.

Which activations get the discount

The released code uses \(\gamma = 0.98\) (0.99 in its recipe for 10M tokens and 16,384 draws) for the activations that later tokens read, which are the attention keys, values, value embeddings, and the gates on those embeddings. Queries and all other layers use \(\gamma = 0\), so each of their jitters is scored only by the loss at its own token.

The discounted activations are not scored by later losses directly. A jitter on one of them is scored by how much it moves the attention output of its block along that output’s estimated error, and the discount is applied to these scores. The estimated error comes from draws scored only by the loss at each token. In the backward-pass picture, the first step from the jittered token to a later token is damped by \(\gamma^{\,s-t}\), and further steps between tokens, in later layers, get no credit. For a value, the expected estimate is \(\sum_s \gamma^{\,s-t} A_{st}\, \hat g_s\), with \(\hat g_s\) now the estimated error at the attention output, so the code computes the damped backward pass through attention one layer at a time.

In Gated DeltaNet-2, the same criterion would also give the discount to the decay, erase, and write gates. The write gate scales what is stored, as a value does. The decay and erase gates store nothing themselves, but a jitter on either one changes what every later token reads. Because of the short convolution, a jitter on the query projection also reaches the next few tokens, so it would need a short discount as well.

Summary

In expectation, the DUST estimate is \((K-1)/K\) times the backprop error signal for a blurred, discounted loss. Three parameters control how it differs from backprop. The noise scale \(\sigma\) blurs the loss, the discount \(\gamma\) limits how far credit extends to later tokens, and the population size \(K\) sets the variance. In the limit \(\sigma \to 0\) with \(\gamma = 1\), and after correcting for the \((K-1)/K\) factor, the expected estimate equals the backprop gradient.

The blur and the discount may help training. A large population costs compute, and reducing that cost is the main open problem.

The figures use a toy one-dimensional loss and a made-up attention pattern. The estimator, the blur, the \((K-1)/K\) factor, and the discounted attention in the figures are computed from the formulas above.