Chrome's synced passkeys all decrypt under one 32-byte secret reachable in memor…

Every serving optimization maps to one term in one equation, so they rank before any benchmark runs. The equation is 2 × layers × KV heads × head_dim × bytes_per_number × tokens × batch. The 2 is key and value. Layers count because each layer keeps its own cache. Llama 3 70B at 80 layers, 8 KV heads, head_dim 128, fp16 gives 327,680 bytes, about 320KB per token per request. At 128K context that is roughly 41GB for one request.

Prefill and decode pull against each other. Prefill runs all input tokens in one parallel pass and saturates the math units. Decode emits one token at a time and re-reads the entire cache from HBM every step, saturating the memory bus. On one homogeneous replica pool a phase is always leaving silicon idle, and long prefills head-of-line block other requests' decode steps. What settles prefill/decode disaggregation is inter-token latency degradation on decode-heavy requests while long prefills share the replica.

Where cache traffic overtakes weight traffic

Back-of-envelope, assumptions stated. An 8×H100 node has roughly 26.8 TB/s aggregate HBM. Llama 3 70B fp16 weights are 140GB, streamed once per decode step, about 5.2ms. Cache traffic scales with aggregate in-flight tokens. Set the two equal: 140GB ÷ 320KB is about 440,000 tokens. Past that, batch 4 at 110K each or batch 32 at 14K each, decode moves more cache bytes than weight bytes, and quantizing the cache buys more latency than quantizing weights.

Which levers are free and which are gambles

Paged attention and prefix caching change how the cache is stored and shared; quantization and eviction change what it contains. Only the first is quality-neutral by construction, and it is good engineering. Paged attention cuts fragmentation from 60–80% to under 4% for 2–3x throughput. Prefix caching saves 50–90% of cost and latency on hits. Eight-bit KV quantization is 2x at a quality cost well under 1%. Four-bit is 4x with measurable loss on multi-needle retrieval: fine for chat, wrong for RAG. Eviction with sliding windows and attention sinks is non-causal by construction, since whether a token matters depends on a query that has not arrived. Once evicted, the model behaves as if the token never existed.

Prefix caching fires only on byte-identical prefixes, so one generated_at timestamp or session UUID high in a system prompt invalidates everything downstream. The order that survives is static system prompt, tool schemas, few-shots, retrieved chunks, volatile fields, user turn. Cached-token share is the SLI, since it maps to the bill. GPU utilization is not a decode health signal; KV pool utilization, preemption and recompute rate, prefix hit rate, and bytes moved per decode step are.

ByteByteGo flags one caveat and declines to resolve it: cross-user cache sharing has opened timing side-channels that leak information about other tenants' prompts. The cheapest large cost reduction on the menu has the least-settled isolation story. Whether a provider shares blocks across tenants is usually undocumented, and worth asking before compliance does.

The KV cache is a bandwidth cost, not a storage cost. That is why shrinking it speeds up generation and adding FLOPs does not.