The Stack v3 Grows 9x to 5T Tokens, Strips Risky Licenses

The Signal

The number worth planning around is not the 5T headline. It is the lopsided per-language growth. C++ grew 15x over v2, TypeScript 7.5x, Rust 7x. Python only grew 4.8x, because it was already well-covered. A codebase leaning on systems, infra, or web code was starved of exactly those tokens in v2. That asymmetry is what tells you where a continue-pretrain or fine-tune spike actually moves the needle, rather than where the token budget happened to pile up.

Under the Microscope

Two operational changes matter more than raw scale. First, contents ship inline instead of Software Heritage IDs, which ends the blob-resolution step that cost a day of preprocessing in v2. Second, the dataset filters restrictively-licensed code — the same class of exposure AINews ties to Anthropic's reported $1.5B copyright settlement, the largest in history. A fresh GitHub recrawl through August 2025, a ready-to-train split, and a full bucket for your own dedup round it out. The license filtering is the cheapest insurance against training-data liability, and it tidies the reproducibility story too.

On the training side, vLLM's prime-rl 0.6.0 is a reference architecture for RL-on-agents at scale: FP8, expert parallelism, prefill/decode disaggregation, KV offload, routing. AINews reports it trains GLM-5 on SWE tasks at 131k context with sub-5-minute steps on 28 H200 nodes. Those are vendor-reported throughput numbers, not independently verified. The thing those step times don't tell you is what they replicate on your hardware. The config patterns — disaggregation plus KV offload as the levers for long-context rollout throughput — are the durable takeaway either way.

The third thread is quieter and points at a real convergence. PRO-LONG treats agent memory as a queryable database of full structured interaction history, and reportedly beats bespoke long-horizon memory harnesses on ARC-AGI-3 with fewer tokens. ARC-AGI-3 is a narrow slice, so the Pareto claim of higher completion at fewer tokens is a hypothesis to falsify, not a result to bank. Paired with the D1 dispatcher and the Harness Handbook, the direction is clear: from context-stuffing toward structured, retrievable memory.

In Your Pipeline

For code-model teams, The Stack v3 is the new floor. Prioritize fine-tune experiments on the C++/TypeScript/Rust gains where v2 was thinnest. That is where the marginal token buys the most. For long-horizon agents, PRO-LONG is a cheap spike with a genuine Pareto claim, worth testing against your own memory harness before committing to a rewrite. What none of these numbers tell you is how much holds on your task distribution. A one-sprint eval settles that.

The Stack v3 and vLLM prime-rl are the two items you can act on — everything else is a benchmark claim waiting for your own harness to confirm.

The Signal

Here is the mechanism most cost post-mortems miss. Most production stacks run one main LLM plus several small specialized models: an embedder, a reranker, an extractor. Standard serving (vLLM, TEI) pre-allocates GPU memory per process at startup with zero cross-process visibility, so a 600M-param embedder claims a full GPU it will never fill. That is why a 'move to smaller, better models' initiative can fail to register on the infra bill at all. It is a GPU-sharing architecture gap, not a model-quality tradeoff. Superlinked's open-source SIE (github.com/superlinked/sie) claims to fix it with shared memory pools and on-demand loading behind one API. Worth a one-day POC against your current stack. The footnote: the claim is vendor-sourced and unbenchmarked here.

Under the Microscope

The reason the last hardware upgrade disappointed is the roofline. An A100 delivers ~300 TFLOP/s FP16 but moves only ~2TB/s between HBM and compute, and autoregressive decoding reads the full model weights plus the entire KV cache from memory for every token. Swap in an H100 with 3x the FP16 compute and token generation barely moves. This is a memory-bandwidth problem dressed as a compute-upgrade decision. Two serving techniques attack that directly:

• Paged attention cuts KV-cache memory waste from 60-80% to under 4%, per the original vLLM paper. A stack not already running it is likely leaving 60%+ of KV-cache capacity on the table.
• Speculative decoding is a throughput win with no quality tax. The rejection-sampling scheme is provably distribution-identical to the target model alone, not an approximation.

Caveat: none of these figures are re-benchmarked; the 60-80%→4% number is a paper citation and the others arrive with no fresh throughput data. Treat this as a mental-model refresh, not new evidence.

On the demand side, TLDR Fintech reports Ramp's customer base grew AI token spend 20.7x since June 2025, with early users of its spend-control product finding up to $10K/month from prompt caching and one in three identifying cheaper model swaps for the same task. The $10K figure has no disclosed denominator and the downgrade recommendations ship with no quality-preservation methodology. The thing these numbers don't tell you is where they came from. What they do tell you: enterprise LLM cost management is now its own product category, scaling faster than most teams' observability.

In Your Pipeline

Two experiments fall out of this at near-zero cost. First, run a roofline/arithmetic-intensity profile on the actual workload before any GPU spend, and confirm paged-attention serving is on. That is a measurable, fixable throughput ceiling. Second, instrument prompt-cache hit rate on the highest-volume repeated-context endpoints (RAG retrieval prompts, system prompts, few-shot templates) for a week and measure the token-cost delta. Then build a lightweight model-routing eval: compare output quality at the current tier and one tier down on a sample of production tasks, validated against your own quality bar rather than a vendor's black-box recommendation.

Buy bandwidth, not FLOPs — and don't expect 'switch to smaller models' to cut costs until your serving stack can actually share a GPU across them.