LLM Inference at Scale

01What changes when the unit is a fleet

Everything on page 31 still applies. It just stops being sufficient.

Continuous batching and chunked prefill both work inside one engine on one set of GPUs. They interleave prefill and decode so that neither starves the other, which is the best you can do while the two phases share a device.

But they still share a device. A prefill chunk and a decode step compete for the same SMs, the same HBM bandwidth and the same KV block pool. Chunking makes the interference smaller and more even. It does not remove it.

The move at fleet scale is to stop sharing. Give prefill its own GPUs, give decode its own GPUs, and pay a transfer cost to connect them.

Three more consequences follow from a fleet, and each gets a section below:

• Bytes are the budget. Weights in FP8 or four bits, KV cache in FP8 or lower, and both of those change what fits and what moves.
• One model has to serve many jobs. Hundreds of LoRA adapters on one base, or hundreds of MoE experts across many GPUs.
• Throughput stops being the target. A fleet is provisioned against a latency contract, so the metric is goodput.

02Prefill/decode disaggregation, and where the crossover is

The argument for splitting is easy. The arithmetic for when to stop is the useful part.

The two phases contend for the same GPU and also want different machines. Prefill is compute-bound and latency-sensitive, so it wants wide tensor parallelism and small batches. Decode is bandwidth-bound, so it wants the largest batch the KV pool will hold.

A single pool has to pick one configuration for both phases, while two pools can each be sized, sharded and batched for its own phase, and the prefill:decode GPU ratio can follow the traffic.

The systems literature is consistent about the size of the win. DistServe reports serving "7.4x more requests or 12.6x tighter SLO" than the systems it compared against, while staying inside latency constraints for over 90% of requests. Splitwise reports "1.4x higher throughput at 20% lower cost", or "2.35x more throughput with the same cost and power budgets".

At production scale, Moonshot's Mooncake describes a KV-cache-centric disaggregated architecture and reports that it "enables Kimi to handle 75% more requests" under real workloads. NVIDIA's Dynamo packages the same idea as an orchestration layer above vLLM, SGLang and TensorRT-LLM.

The cost, derived

Prefill produces a KV cache and decode consumes it, so splitting the phases means that cache has to cross a wire, and the size of the cache is fixed by the architecture:

The leading 2 is for K and V. $b$ is bytes per element, so 2 in FP16 and 1 in FP8.

Prefill for a prompt of $L$ tokens costs roughly $2PL$ floating-point operations, where $P$ is the parameter count active per token. If the pool achieves $F$ FLOP/s and the link runs at $W$ bytes/s, then the transfer as a fraction of the prefill is:

The prompt length cancels, so the ratio does not depend on how long the prompt was.

This cancellation is why disaggregation scales. Both sides grow linearly in $L$, so a 100,000-token prompt pays the same proportional transfer cost as a 1,000-token one. Attention makes prefill superlinear in $L$, so at very long contexts the ratio gets better.

Setting the ratio to 1 gives the link speed at which shipping the cache costs as much as making it:

Call this the break-even bandwidth. It depends on the model and the GPU and does not depend on the workload.

import numpy as np

# KV bytes per token = 2 (K and V) x layers x kv_heads x head_dim x dtype
def kv_per_token(layers, kv_heads, head_dim, nbytes):
    return 2 * layers * kv_heads * head_dim * nbytes

F = 4.0e14        # assumed achieved prefill rate, FLOP/s. Yours differs.

# name, layers, kv_heads, head_dim, bytes/elem, params active per token
cfgs = [("Llama-3.1-8B ", 32, 8, 128, 2,  8.0e9),
        ("Llama-3.1-70B", 80, 8, 128, 2, 70.0e9),
        ("70B, FP8 KV  ", 80, 8, 128, 1, 70.0e9)]

print("model          KV per token   break-even link")
for name, L, h, d, nb, P in cfgs:
    c = kv_per_token(L, h, d, nb)
    print(f"{name}   {c/1024:6.0f} KiB     {c*F/(2*P)/1e9:6.2f} GB/s")

c70 = kv_per_token(80, 8, 128, 2)
print("\n70B, an 8k-token prompt: KV =",
      round(c70 * 8192 / 2**30, 2), "GiB to move.")
print("shipping it costs this share of the prefill that made it:")
for link, bw in [("NVLink 4, one direction", 450e9),
                 ("PCIe Gen5 x16          ", 64e9),
                 ("InfiniBand NDR 400G    ", 50e9),
                 ("25 GbE                 ", 3.125e9)]:
    print(f"  {link} {c70*F/(2*70e9*bw)*100:7.1f} %")

The link bandwidths above come from vendor specifications: NVLink 4 at 900 GB/s total on an H100 SXM, halved here for one direction; PCIe Gen5 x16 at 128 GB/s bidirectional, also halved; and InfiniBand NDR at 400 Gb/s per port.

The last point deserves emphasis. Disaggregation is the expensive answer to interference. Chunked prefill is the cheap one, and page 31 covers it. Measure that the cheap one has run out before buying the expensive one.

03The KV cache: make it narrower, or move it off the GPU

Two independent attacks on the same problem. One changes the bytes per token, the other changes where those bytes live.

NVIDIA's inference optimization guide states the size directly: "Size of KV cache per token in bytes = 2 * (num_layers) * (num_heads * dim_head) * precision_in_bytes". Under grouped-query attention the head count there is the number of KV heads, which is what section 02 called $n_{\text{kv heads}}$. Three of the four terms are fixed by the architecture. Only the last is yours.

Quantizing the cache

vLLM's quantized KV cache supports fp8_e4m3 and fp8_e5m2, which halves $c$ against FP16. This doubles how many tokens fit in the block pool, and it halves the break-even bandwidth from section 02 at the same time.

Below eight bits the cache stops behaving like a generic tensor. KIVI found that the two halves need opposite treatment: "the key cache should be quantized per-channel" while "the value cache should be quantized per-token". With that asymmetry it reports 2-bit KV at "2.6x less peak memory" and "2.35x ~ 3.47x throughput".

KVQuant attacks the same problem from the accuracy side and reports "< 0.1 perplexity degradation with 3-bit quantization", which it converts into context length: "serving LLaMA-7B with a context length of up to 1 million on a single A100-80GB GPU".

Offloading the cache

The second attack leaves the precision alone and moves the bytes to host memory or NVMe. Whether that helps is decided by arithmetic, and there are two completely different cases.

Case one: KV that decode is actively reading. Every decode step reads the whole cache for every request in the batch. Take 32 requests at 8,192 tokens on the 70B, at 320 KiB per token. That is 85.9 GB read per token generated.

Over H100 HBM3 at 3.35 TB/s, that read takes 25.6 ms. Over PCIe Gen5 x16 at 64 GB/s it takes 1.34 seconds. A 52x slowdown on the inner loop is not a tradeoff, it is a different product.

Case two: KV that nobody is reading yet. A cached prefix from a previous turn, sitting in host RAM. The question is not bandwidth against HBM, it is fetch against recompute, and that is exactly the comparison section 02 already solved:

The same break-even bandwidth, for a question that looks nothing like the first one.

For the 70B, $W^{*}$ was 0.94 GB/s and PCIe Gen5 x16 runs at 64 GB/s, about 68 times higher. Fetching a cached prefix over PCIe is far cheaper than recomputing it, which is the entire case for KV offload.

LMCache productised this. Its paper describes storing KV "out of the GPU memory" and sharing it "across engines and queries", supporting "both cache offloading (prefix reuse across queries) and prefill-decode (PD) disaggregation", and reports "up to 15x improvement in throughput" on workloads like multi-round question answering.

The extreme version of case two is FlexGen, which aggregates GPU, CPU and disk to run OPT-175B on a single 16 GB GPU at "1 token/s ... with an effective batch size of 144". It is a throughput system for offline work, and its latency is what the arithmetic above predicts.

04Multi-LoRA: hundreds of models, one set of weights

A fleet rarely serves one model. It serves one base and a long tail of customer-specific adaptations, and the only affordable way to do that is to not duplicate the base.

Page 29 covers what a LoRA adapter is. The serving question is different: given a batch of requests where each one wants a different adapter, what does that cost?

Start with memory: a LoRA on a weight of shape $(d_{\text{out}}, d_{\text{in}})$ adds two matrices, $A \in \mathbb{R}^{r \times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}} \times r}$, so $r(d_{\text{in}} + d_{\text{out}})$ parameters per adapted matrix.

import numpy as np

# LoRA on a weight of shape (d_out, d_in) adds A (r, d_in) and
# B (d_out, r): r * (d_in + d_out) new parameters.
def adapter_params(r, shapes):
    return sum(r * (d_in + d_out) for d_out, d_in in shapes)

H, KV, LAYERS = 4096, 1024, 32          # Llama-3-8B, 32 q heads, 8 kv
attn = [(H, H), (KV, H), (KV, H), (H, H)]        # q, k, v, o

for r in (8, 16, 64):
    p = adapter_params(r, attn) * LAYERS
    print(f"rank {r:3d}: {p/1e6:6.2f}M params, {p*2/1e6:6.1f} MB in BF16"
          f"   1000 of them = {p*2*1000/1e9:6.1f} GB")

print(f"\nbase model in BF16 = {8e9*2/1e9:.0f} GB;"
      f" 1000 full fine-tunes = {8e9*2*1000/1e12:.0f} TB")

print("\ndecode step reads 16.0 GB of base weights, plus, for n"
      "\nDISTINCT adapters in the batch:")
for r in (16, 64):
    mb = adapter_params(r, attn) * LAYERS * 2 / 1e6
    for n in (1, 32, 128):
        print(f"  rank {r:3d}, {n:3d} distinct: {n*mb/1000:6.2f} GB"
              f"  -> step is {1 + n*mb/16000:5.3f}x longer")

That last result is the whole design constraint. The base path is one big GEMM shared by the entire batch. The LoRA path is not shareable: each sequence needs its own $A$ and $B$ gathered and applied.

Done naively that becomes a Python loop over requests, which destroys the batch. Punica solved it with a kernel: "Segmented Gather Matrix-Vector Multiplication (SGMV)", which "allows batching GPU operations for the concurrent execution of multiple, different LoRA models". It reports "12x higher throughput ... while only adding 2ms latency per token".

S-LoRA added the memory-management half, and its "Unified Paging" puts adapter weights of different ranks and KV tensors of different lengths in one pool, which is PagedAttention generalised to a second kind of variable-size object. It reports up to 4x throughput over prior LoRA serving.

05FP8 and MXFP4: where the scale factors live

Four-bit floating-point formats differ from plain INT4 in how many numbers share a scaling factor.

FP8 came first and is the easy case. The format paper defines two encodings, "E4M3 (4-bit exponent and 3-bit mantissa) and E5M2 (5-bit exponent and 2-bit mantissa)", and reports "effectively matching the result quality achieved by 16-bit training sessions".

The hardware reason to care is on the datasheet. An H100 SXM lists 1,979 TFLOPS for BF16 tensor cores and 3,958 for FP8, both quoted with sparsity and both halved without it. That is exactly 2x, and it comes from the format and not from the silicon generation.

What gets quantized

"The model is in FP8" is three separate decisions, and they have different risk profiles:

• Weights. These are static and quantized once offline. vLLM's llm-compressor FP8 scheme is "static, per-channel quantization on the weights", meaning one scale per output channel, stored alongside them.
• Activations. Their range depends on the input, so the scale is computed at runtime. vLLM uses "dynamic, per-token quantization on the activations".
• KV cache. See section 03. It has a separate flag and a separate risk, and it is the one that breaks long context first.

TensorRT-LLM's precision reference names the same three granularities: "per-tensor" with a single scaling factor for all elements, "per-token", and "per-channel". Finer granularity costs storage and buys tolerance to outliers, and that tradeoff explains the four-bit formats.

Microscaling: the scale moves inside the tensor

The OCP Microscaling (MX) specification v1.0 fixes the granularity in the format itself. MXFP4 is an E2M1 element, four bits, in blocks of 32 that share one E8M0 scale. E8M0 is a bare Float32 exponent with no sign and no mantissa, so the scale is a raw power of two.

Count the bits: four per element plus eight per block of 32 is $4 + 8/32 = 4.25$ bits per weight. NVIDIA's NVFP4 halves the block "from 32 values to 16" and upgrades the scale to E4M3, giving 4.5 bits, plus a per-tensor FP32 scale on top.

import numpy as np

# MXFP4 (OCP Microscaling v1.0): 32 elements share one E8M0 scale
# (a raw power of two); each element is E2M1, four bits.
E2M1 = np.array([0., .5, 1., 1.5, 2., 3., 4., 6.])   # magnitudes

def snap(q, grid):
    i = np.abs(np.subtract.outer(np.abs(q), grid)).argmin(axis=-1)
    return np.sign(q) * grid[i]

def mxfp4(x, blk=32):
    y = x.reshape(-1, blk)
    s = 2.0 ** np.floor(np.log2(np.abs(y).max(1, keepdims=True) / 6))
    return (snap(y / s, E2M1) * s).ravel()

def int4_tensor(x):                    # one scale for the whole tensor
    s = np.abs(x).max() / 7.0
    return np.clip(np.round(x / s), -7, 7) * s

rng = np.random.default_rng(0)
w = rng.normal(0, 1, 4096)
w[137] = 50.0                          # one outlier, as real weights have

for name, q in [("MXFP4, 32-elem blocks ", mxfp4(w)),
                ("INT4, one tensor scale", int4_tensor(w))]:
    e = q - w
    print(f"{name}  rms err {np.sqrt((e**2).mean()):6.3f}"
          f"   weights flushed to zero {(q == 0).mean():6.1%}")

print("\nbits per weight, scale included:")
for nm, elem, blk, sc in [("MXFP4", 4, 32, 8), ("NVFP4", 4, 16, 8)]:
    b = elem + sc / blk
    print(f"  {nm}  {b:.2f} bits   {16/b:.2f}x smaller than BF16")

NVIDIA reports NVFP4 against FP8 on DeepSeek-R1-0528 at "1% or less accuracy degradation", with MMLU-Pro at 85 against 84 and GPQA Diamond at 81 against 80.

The clearest production example is OpenAI's gpt-oss. Its model card states the models "were post-trained with MXFP4 quantization of the MoE weights, making gpt-oss-120b run on a single 80GB GPU ... and the gpt-oss-20b model run within 16GB of memory".

The arithmetic behind that sentence is the 4.25 bits above. 120 billion weights at 4.25 bits each is 63.75 GB, and only the MoE weights are at that width, so the real footprint is somewhat higher. In BF16 the same weights would be 240 GB and would need four cards.

06Decode needs its own attention kernel

FlashAttention is a prefill kernel. Running it at decode time leaves most of the GPU idle, and the reason is a single dimension collapsing to 1.

FlashAttention parallelises over batch, heads and blocks of queries. At decode there is one query per request, so the query dimension is gone and only batch times heads is left to fill the machine.

The Flash-Decoding authors put the consequence plainly: "if the batch size is smaller than the number of streaming multiprocessors (SMs) on the GPU (108 for an A100), the operation will only use a small part of the GPU! ... With a batch size of 1, FlashAttention will use less than 1% of the GPU!"

Count it yourself: a 32-head model at batch 1 offers 32 independent work units against an A100's 108 SMs, under a third of the machine, and no kernel tuning recovers the other two thirds because the work is not there.

The split only pays when the KV sequence is long relative to the batch, because that is precisely when parallelism over batch and heads is scarce and parallelism over sequence is abundant. At batch 256 and 128 tokens of context there is nothing to fix.

FlashInfer generalised this into a kernel library built for serving and not for training, and reports "29-69% inter-token-latency reduction compared to compiler backends for LLM serving benchmark, 28-30% latency reduction for long-context inference". It won a best paper award at MLSys 2025.

Two further decode-specific facts shape these kernels. Grouped-query attention gives each KV head several query heads, so the kernel can load the KV once and issue a small matrix multiply instead of several separate vector ones. And PagedAttention scatters the KV across non-contiguous blocks, so the kernel has to walk a block table rather than stride through an array.

07Expert-parallel MoE serving

A sparse mixture of experts is a model that does not fit on one GPU and does not want to. Serving it well is a communication problem with a load-balancing problem inside it.

DeepSeek-V3 is the reference point: "671B total parameters with 37B activated for each token". Each MoE layer has "1 shared expert and 256 routed experts", of which "8 experts will be activated for each token".

Tensor parallelism would shard every expert across every GPU, which means every GPU does a slice of all 256 experts. Expert parallelism does the opposite: each GPU owns a few whole experts, and tokens travel to them.

This changes what moves. Instead of all-reducing activations after every matmul, you dispatch each token's hidden state to the GPUs holding its 8 experts, compute, then combine the results back, which takes two all-to-all collectives per MoE layer, per step.

The all-to-all, priced

DeepEP benchmarks this at DeepSeek-V3's configuration: "8K tokens per batch, 7168 hidden dimensions, top 8 experts, FP8 dispatching, and BF16 combining". Those numbers are enough to size the traffic.

Dispatch moves $8192 \times 8 \times 7168$ bytes at one byte each, which is 470 MB. Combine moves the same count at two bytes, so 940 MB. DeepEP reports 726 GB/s dispatch and 740 GB/s combine over NVLink, against 90 and 81 GB/s over RDMA.

Divide through and the intra-node case costs about 1.9 ms per MoE layer, while the cross-node case costs about 16.8 ms. DeepEP notes these are logical bandwidths that include local traffic, so treat the ratio as the finding and neither absolute as exact.

That roughly 9x gap is why DeepSeek-V3's router constrains each token to "at most 4 nodes". Node-locality is not a deployment detail bolted on afterwards, it is a term in the routing function.

Imbalance is the cost

Routing is learned, and learned routing is not uniform. The LMSYS large-scale expert-parallelism report states the consequence: "This imbalance forces the system to wait for the slowest GPU computation or communication ... As the number of GPUs (EP size) increases, the imbalance issue gets more severe."

A step finishes when the busiest expert finishes. Mean load is irrelevant; the maximum sets the clock.

import numpy as np

E, K, T = 256, 8, 8192        # experts, experts per token, tokens
rng = np.random.default_rng(0)

def load(scores):             # tokens per expert after top-K routing
    top = np.argpartition(-scores, K, axis=1)[:, :K]
    return np.bincount(top.ravel(), minlength=E)

def report(tag, c):
    print(f"{tag:22s} mean {c.mean():6.1f}  busiest {c.max():5d}"
          f"  imbalance {c.max()/c.mean():5.2f}x")

aff = rng.normal(0, 1, (T, E))                 # router affinities
report("uniform router", load(aff))

pop = rng.normal(0, 0.6, E)                    # some experts are popular
report("popularity-skewed", load(aff + pop))

# DeepSeek-V3's auxiliary-loss-free fix: a per-expert bias, added to
# the score for ROUTING ONLY, nudged down when an expert is overloaded.
b, gamma = np.zeros(E), 0.05
for step in range(60):
    c = load(aff + pop + b)
    b -= gamma * np.sign(c - c.mean())
report("after bias correction", load(aff + pop + b))

Two production mitigations sit on top of that. DeepSeek deploys "redundant experts, which duplicates high-load experts", and SGLang's report finds that "SGLang with default expert imbalance is 20% slower than DeepSeek's profile, while the simulated perfect EPLB case narrows the gap to 6%".

For scale, that same deployment ran on 12 nodes of 8 H100s and reported "52.3k input tokens per second and 22.3k output tokens per second per node for 2000-token input sequences", using prefill/decode disaggregation from section 02 alongside expert parallelism.

08SLA-aware scheduling, and why throughput is the wrong target

Throughput rises monotonically with concurrency, but goodput does not, so running at the top of the first curve puts you well past the top of the second.

DistServe defines the target precisely: "per-GPU goodput, defined as the maximum request rate that can be served adhering to the SLO attainment goal (say, 90%) for each GPU provisioned". Tokens served too slowly to be useful do not count.

The gap between the two metrics is not small, and it is easy to see with the arithmetic already on this page.

import numpy as np

P, W, BW, F = 8e9, 16e9, 3.35e12, 4.0e14   # 8B in BF16, one H100
L_out = 256                                # output tokens per request
TTFT_SLO, ITL_SLO = 2.0, 0.025             # the contract with the user

rng = np.random.default_rng(0)
L_in = rng.lognormal(np.log(900), 1.0, 4000)        # prompt lengths
wait = (np.arange(4000) + .5) / 4000                # place in the cycle

print("  N   thr tok/s   ITL ms   met SLO   goodput tok/s")
for N in (4, 8, 16, 24, 32, 48, 64, 128, 256):
    # one cycle = N prefills plus L_out decode steps at batch N
    cycle = L_out * max(W / BW, 2 * P * N / F) \
            + N * 2 * P * L_in.mean() / F
    thr = N * L_out / cycle                          # tokens/s
    itl = cycle / L_out
    ttft = wait * cycle + 2 * P * L_in / F
    met = ((ttft <= TTFT_SLO) & (itl <= ITL_SLO)).mean()
    print(f"{N:4d} {thr:10.0f} {itl*1e3:8.1f} {met:9.0%} {thr*met:14.0f}")

Once you accept goodput as the target, the scheduler gets choices it did not have before. Four systems illustrate the range:

• FastServe attacks head-of-line blocking with "a novel skip-join Multi-Level Feedback Queue scheduler", reporting up to 31.4x throughput at the same average latency requirement.
• Llumnix reschedules requests across model instances at runtime and reports improving "tail latencies by an order of magnitude" and accelerating "high-priority requests by up to 1.5x".
• QLM manages the queue and not the batch, reporting "SLO attainment by 40-90% and throughput by 20-400%".
• Andes optimises perceived quality of experience directly, reporting up to 4.7x average QoE at fixed GPU count, or "61% GPU resources" saved at fixed QoE.

These policies are built in: vLLM ships a policy setting where "fcfs" means arrival order and "priority" means "requests are handled based on given priority (lower value means earlier handling)".

Preemption is the other half. When the KV pool fills, the scheduler evicts a request and later recomputes its prefill, which is throughput spent twice. vLLM exposes a watermark that keeps a fraction of blocks free specifically to "avoid frequent KV cache eviction and the resulting repeated preemption of requests".

09Build this

Section 02's break-even bandwidth is three measurements away from being a number about your own hardware and not just a number on this page.

10What breaks

At fleet scale the failures are not crashes. They are a number that got worse somewhere you were not looking, for a reason that is nowhere in the logs.

• The KV transfer is serialised, not overlapped. Section 02's percentage assumes the send of layer $i$ overlaps the prefill of layer $i+1$. If it does not, you pay the whole transfer as added TTFT, and disaggregation looks like a regression.
• The pool ratio is tuned for yesterday's traffic. Prompts get longer, the prefill pool saturates, and the decode pool idles while queueing grows. The symptom is rising TTFT with flat inter-token latency, which points straight at the split.
• KV quantization passes every eval you ran. Short-context benchmarks will not show it. The damage lands on long-context retrieval, so it ships, and then a customer with a 100,000-token document finds it for you.
• One hot expert is setting the step time. Throughput falls by the imbalance factor and no component reports an error, because every GPU is healthy and most of them are waiting. Log per-expert token counts or you cannot see it.
• A new router checkpoint invalidates the expert placement. Yesterday's redundant-expert plan was fitted to yesterday's routing distribution. Rebalancing has to be part of deployment, not a one-off.
• One customer's rank-256 adapter taxes the whole fleet. LoRA buffers are sized for the maximum rank, so everyone pays for the largest adapter admitted. Cap it at training time.
• The decode kernel silently fell back. An unusual head dimension, dtype or block size drops you onto a generic path without the split-K, and inter-token latency worsens by a factor you will attribute to anything else.
• Throughput improved and goodput did not. The most common outcome of a batching change. If you are not measuring SLO attainment alongside tokens per second, you cannot tell an improvement from a degradation.
• KV was offloaded to the wrong side of the line. Offloading cached prefixes is a large win. Offloading the cache that decode reads every step costs a factor of about 50, and both are enabled by flags with similar names.

11Where you meet this in the wild

The same techniques, in wildly different combinations, because the binding constraint is different in each.

12Interview questions

13Go deeper

●Now write it yourself

Reading the derivation and being able to produce it are different skills. These are Deep-ML problems that exercise what this page covers — each one is checked against real test cases, not multiple choice.

Matched to this page from Deep-ML's catalogue of 1,380 problems. More at deep-ml.com, and Where to practise covers the other platforms and what each one trains.

14Related pages

Other techniques for this problem

A scoped slice of the full Technique Map — every technique this page covers, grouped by what it solves.