Decode Notes

Evaluation

A benchmark score is a claim about a test set

A leaderboard number says the model did well on those items, under that prompt, with that decoder. It does not say the model will do well on the next thing you ask it.

Public suites are reused for years. Their questions, answers, and explanations are copied into blogs, forums, training scrapes, and synthetic data. A model can score higher because it memorized the item, because the prompt template matches the one used in training, or because the metric forgives a format the production parser will reject. Those are different events. A single accuracy hides which one happened.

Contamination is an overlap measurement

The clean definition is overlap between the training corpus and the evaluation items, not a vibe about whether the model “feels familiar” with the subject. Practical checks are narrower than the definition, and they should be reported as such:

A canary that never fires does not prove the public suite was unseen. It proves the canary was unseen. Overlap against a corpus you do not have cannot be measured from the model weights alone. Say that limit in the report instead of converting it into “no contamination detected.”

The decoder is part of the result

The same weights produce different scores under greedy decoding, temperature sampling, a different system prompt, or a tool-calling wrapper. Majority vote over 32 samples measures a search procedure as much as a model. If the number you quote used a scaffold, the scaffold belongs in the same sentence as the score. Otherwise a later reader will compare your search against someone else’s single sample and call the difference intelligence.

Use a slice you own

For a serving decision, keep a small set drawn from production: real prompts, the failure you care about, and a grader that matches the downstream check. Run it whenever the weights, the prompt, or the decoder changes. Publish the public number too if you want, but pair it with the overlap evidence you actually have and with the decoding settings. The public number is then a comparison point. The private slice is the claim about your traffic.

When those two disagree, believe the slice. The benchmark already told you everything it can, which is how the model behaved on its own items.