Memory
Do the KV-cache arithmetic before you buy the GPU
Context length is a memory product. The weights set the floor. The cache, multiplied by batch and sequence length, decides whether the card can hold the job.
Each generated token appends a key and a value at every layer. Those tensors stay live until the request finishes, and every later token reads them. Parameter count does not capture this. Two models with the same weight footprint can differ by several times in cache cost once grouped-query attention, head dimension, and dtype enter the product.
Bytes per token
For one sequence, in bytes:
kv_bytes_per_token = layers × kv_heads × head_dim × 2 × dtype_bytes
The 2 is key plus value. dtype_bytes is 2 for FP16 or BF16, and 1 for FP8. Grouped-query attention reduces kv_heads relative to the number of query heads. It does not change the formula, only one factor inside it.
Take a concrete decoder: 32 layers, 8 KV heads, head dimension 128, BF16.
32 × 8 × 128 × 2 × 2 = 131,072 bytes = 128 KiB per token
Scale that by the sequence you intend to keep resident:
| Resident tokens | One sequence | Batch of 16 |
|---|---|---|
| 8,192 | 1.0 GiB | 16 GiB |
| 32,768 | 4.0 GiB | 64 GiB |
| 131,072 | 16 GiB | 256 GiB |
The last row does not fit on one 80 GiB card even before weights, activations, and allocator waste. “128K context” on the model card is a maximum the architecture can address. It is not a batch-1 promise at full length, and it is certainly not a promise at the concurrency you need for throughput.
What paging does and does not change
Paged attention stores the cache in fixed blocks instead of one contiguous allocation per request. That removes the large holes left by reservations that never fill. It does not reduce the bytes of a token you have already stored. A full 32K sequence still occupies 4 GiB in the example above. Paging lets you pack many short requests into the holes that a single reserved 32K slab would have wasted.
Prefix caching is the other half. If two requests share a system prompt, the shared prefix can be stored once and referenced twice. The saving is real only for the shared span. Divergent chat history after the first turn is not shareable, so a cache hit rate measured on identical benchmark prompts overstates production.
A buying check
Before comparing cards, write down four numbers from the traffic, not from the model card: median prompt tokens, p95 prompt tokens, average output tokens, and target concurrency. Multiply kv_bytes_per_token by resident tokens per request and by concurrency. Add the weight bytes at the dtype you will actually serve. If that sum exceeds about 70% of device memory, the plan depends on a lower batch, a shorter cap, quantization of the cache, or another GPU. Those are product decisions. They should be made before the purchase order, not after the first out-of-memory error.