Decoding
Acceptance length is the speculative-decoding metric that matters
A draft model does not make decoding faster by itself. It makes decoding faster only while the target model keeps the tokens the draft proposed.
Autoregressive decoding is a loop of small matrix multiplies. Each new token waits on the previous one, and the GPU spends much of that step moving weights and the KV cache rather than doing arithmetic. Speculative decoding tries to break the loop: a cheap draft proposes several tokens, and the target model checks that whole span in one forward pass.
The check is strict. The target distribution, not the draft, decides which prefix survives. Everything after the first rejection is thrown away, including the work the draft spent producing it. The useful output of a round is therefore not “tokens proposed.” It is the accepted prefix length.
Where the speedup actually comes from
Call the draft cost per token d, the target cost per token t, and the mean number of accepted draft tokens per verification a. One round proposes k draft tokens and then runs the target once over that span. A reasonable first estimate of tokens per unit time is:
tokens / cost ≈ (a + 1) / (k · d + t)
The + 1 is the token the target samples at the first rejection, or at the end of a fully accepted span. Two consequences follow immediately.
- Raising
khelps only whilearises with it. Past the point where the draft and target disagree, extra draft tokens are pure overhead. - A draft that is half as expensive but accepted half as often can be slower than no speculation at all. The comparison is against
1 / t, the plain target loop.
This is why “our draft is 8× smaller” is not a result. Report a on the traffic you actually serve, at the temperature you actually use, and the wall-clock against a tuned baseline that already uses continuous batching.
What moves acceptance length
Agreement is a property of two distributions on one prompt, not a property of either model alone. Acceptance falls when the draft was trained on a different domain, when the target is sampled hot, or when the prompt is in a region the draft rarely saw. Greedy decoding hides this. At temperature 0 both models often collapse onto the same high-probability token, so a lab demo looks better than production chat, code, or tool-calling traffic.
Verification is also not free memory traffic. The target still reads the cache for every accepted and rejected position it scores. If the workload is already bandwidth-bound on a long context, a large proposal width can increase bytes moved faster than it increases accepted tokens. Measure cache read volume per accepted token, not only kernel time on the draft.
What to log
A serving change is worth keeping when these four numbers move in the same direction on a recorded trace:
- mean accepted prefix length, split by prompt class
- draft tokens generated per accepted token
- inter-token latency at the 50th and 95th percentile
- tokens per second per GPU at the production batch, not at batch 1
If accepted length stays flat while draft cost rises, turn speculation off for that route. The target loop was already the faster system.