Meta doubled its ads training efficiency and still wastes three FLOPs in every f…

The detail worth internalizing even if you never run several thousand GPUs

Meta shards two parameter populations under two different regimes: 2D FSDP plus Expert Parallelism for dense parameters, and Fully Sharded 2D Model Parallelism for trillions of sparse embedding parameters. Both are mapped topology-aware onto a multi-tiered fabric with SM-free collectives. The transferable finding is narrower than the architecture diagram suggests: one uniform parallelism strategy across a model with a huge embedding table and a modest dense trunk leaves throughput on the floor. At this scale the jobs are comms-bound, not FLOPs-bound, and adding GPUs to a comms-bound job buys more all-reduce, not more steps per second.

Portability, ranked

Where the disclosure thins

No baseline MFU is stated. The ~10–12% prior is an inference from "2x to 20–25%", not a number Meta published. The 4x FLOPs scaling is entangled with the efficiency work, so any model-quality improvement cannot be attributed between more compute and better utilization. Correlated, plausibly causal, not established. And "no CTR/CVR regression" from MXFP8 arrives with no confidence intervals and no slice-level breakdown. The thing that claim doesn't tell you is where the errors went, because low precision fails first on rare slices and on probability calibration, and neither shows up in an aggregate conversion rate.

Meta stated on the record that the honest ceiling for sparse recommenders is low. That is negotiating leverage in your next hardware request, not an excuse.

What to do with it

The cheapest measurement available is the padding ratio. Take one real batch from the highest-cost sequence model in production and compute sum(true_seq_lens) / (batch_size × max_len). Under 0.6 means up to half the attention FLOPs are multiplying zeros. The fix is a varlen kernel, with no quality conversation attached, which is rare enough in this field to be worth doing first. MFU belongs on the training dashboard next to loss and AUC, because without a baseline there is no arguing for or against more GPUs.

Before any linear-attention migration, run a truncation-depth ablation on user-history length: sweep 64 / 256 / 1024 with full attention and plot the AUC curve. BlockAttention's value is entirely conditional on longer history carrying signal in your domain, which is a research-leaderboard property, not a production one. If the curve flattens at 256, O(L) attention buys approximation risk and nothing else. For MXFP8, gate on paired runs with identical seeds and data order, scoring log-loss and expected calibration error across at least five traffic slices including the lowest-volume decile.

Why a request that fits in HBM can still crawl

Prefill is compute-bound; decode is memory-bandwidth-bound, because every emitted token re-reads the whole stored cache out of HBM. That is a bandwidth bill, not a storage bill. Sweeping 42GB of cache per token against one H100's ~3.35TB/s is ~12.5ms of pure memory traffic, an ~80 tok/s ceiling before a single multiply runs. That roofline is our arithmetic, not a published measurement. It is still the number that should govern the next hardware request, because if TPOT is the latency problem, FLOPs are not measuring the bottleneck.

What each optimization actually attacks

• Paged attention. Fragmentation waste from 60–80% down to under 4%, 2–3x throughput, essentially zero quality cost. On vLLM or SGLang that gain is already priced in and does not arrive twice.
• Prefix / prompt caching. 50–90% cost and latency reduction on hits, per OpenAI's and Anthropic's own figures. Realized saving is (hit rate × discount), and hit rate is a function of prompt hygiene alone. One timestamp or session ID near the top of a system prompt forfeits the discount on every call.
• 8-bit KV quantization. 2x, at a claimed well under 1% accuracy cost. 4-bit. 4x, with measurable degradation on multi-needle retrieval specifically, which is the RAG and long-document-agent profile.
• MLA (DeepSeek-V2/V3) holds ~70KB/token against 192–328KB for GQA peers, roughly 4–5x the concurrency at fixed HBM. Locked at training time, awkward with standard attention kernels.

The second lever: stop paying max on easy queries

Medical-reasoning models that vary how much they think per question cut compute 4.7x to 6.4x while holding accuracy "nearly the same." No effect size, no confidence interval, one domain: a hypothesis with a known verification cost. The mechanism holds up, since a fixed thinking budget provisions for the hardest input in the distribution and bills it on every request, and required compute is heavily skewed. A cheap difficulty classifier or self-consistency confidence gate up front, escalating only on low confidence, is the whole design. Pre-register the non-inferiority margin before anyone sees results, because "nearly the same accuracy" is where a three-point regression on a tail segment hides.

What this changes downstream

First, context length and concurrency draw from one budget, since cache is linear in tokens and linear in batch. No configuration gives 128K context and high concurrency for free, and the unmade choice gets made by the scheduler as an OOM. Put the capacity equation in admission control: max concurrent sequences, max context per tier.

Second, the eval harness needs a compression canary. Cache-compression damage is task-localized, so an aggregate benchmark will pass a config that broke mid-document fact extraction. Build a position-stratified multi-needle eval, needles spread across the context rather than clustered at the end, run it paired against the 16-bit baseline, and ship 8-bit only when the interval on the delta excludes a meaningful loss. Then audit for sliding-window eviction anywhere in a RAG path and turn it off there. Token relevance depends on a query that has not arrived, so eviction is non-causal by construction, and a default window in a retrieval path reads as model weakness rather than the config bug it is.

Shrink bytes-per-token and share prefixes aggressively; quantize to 8-bit only behind a multi-needle canary; never evict on retrieval workloads.