Training gets the headlines; inference pays the bills. A frontier model is trained once and then serves billions of tokens, so the economics of the field — margins, pricing, which products are even viable — are set at inference time. Almost every optimization that matters is, at bottom, a trick to move more tokens through fixed hardware without degrading quality more than the market will tolerate. Understanding the stack means understanding one hardware fact first.

Why memory bandwidth rules

Autoregressive decoding generates one token at a time, and each token requires reading the entire model's weights — plus a growing cache — out of GPU memory. For a single request, the arithmetic is lopsided: the GPU's compute units sit mostly idle while they wait on memory. Decoding is memory-bandwidth-bound, not compute-bound, and that single fact explains most of what follows.

The cache in question is the KV cache. Attention lets each new token attend to every previous one, so rather than recompute the keys and values for the whole sequence at every step, you store them. The KV cache grows linearly with sequence length and batch size and, at long context, comes to dominate memory — often exceeding the size of the weights themselves. It is also why long-context serving is expensive: the cost is not only the tokens you generate but the memory bandwidth to stream an ever-larger cache on every single step. Nearly every inference optimization is either shrinking what must be read (quantization, cache compression) or extracting more useful work from each read (batching, speculative decoding, sparsity).

Quantization

Quantization stores weights — and sometimes activations and the KV cache — in fewer bits than the 16-bit floats used in training. Because decode is memory-bound, halving the bytes per weight roughly doubles throughput and cuts the memory footprint proportionally. It is the highest-leverage single optimization for most deployments.

The formats divide along a few axes. Weight-only methods such as GPTQ and AWQ compress weights to about 4 bits post-training while keeping activations at higher precision; AWQ's insight — protect the small fraction of salient weight channels — makes 4-bit weights nearly lossless for many models and is a common production default. GGUF, the format llama.cpp popularized, offers a spectrum of mixed-precision schemes tuned for CPU and consumer-GPU inference. Activation-aware methods like SmoothQuant, and low-precision floating formats like FP8, quantize both weights and activations, which unlocks faster matrix multiplies on hardware with low-precision tensor cores — FP8 on NVIDIA Hopper, and now FP4 on Blackwell, where frontier models increasingly ship with quantization-aware training so the low precision is baked in during training rather than bolted on afterward.

The tradeoff is always accuracy against footprint, and it is not linear. Going from 16-bit to 8-bit is nearly free; 4-bit is usually a good deal with careful methods; below 4 bits, quality degrades fast and unevenly — reasoning and long-context behavior tend to break before perplexity visibly does, which is why a model that looks fine on a quick eval can fail subtly inside an agent loop. Quantizing the KV cache itself (to 8 or even 4 bits) is a newer lever aimed specifically at long-context memory pressure. The working rule: quantize weights aggressively, activations moderately, and always validate on the actual downstream task rather than on loss.

Speculative decoding

Speculative decoding attacks the latency of sequential generation without changing the output distribution. The idea: a small, cheap draft model proposes several tokens ahead, and the large target model verifies them all in a single forward pass. Because a forward pass over a handful of candidate tokens costs almost the same as a pass over one — remember, decode is memory-bound, so the weights are already being read — you get the large model to check multiple guesses for roughly the price of generating one. Accepted tokens are kept; the first rejected token is corrected and generation continues. Crucially, the accepted output is provably identical in distribution to what the target model would have produced alone: it is a speedup, not an approximation.

The variants differ in where the draft comes from. A separate small model is the classic form; Medusa bolts extra prediction heads onto the target model itself; EAGLE and its successors predict at the feature level and, with tree-structured drafting to raise the acceptance rate, are the current state of the art. Realistic speedups are roughly 2-3x on interactive, low-batch paths. The catch: the benefit shrinks as batch size grows, because a busy server is already using its memory bandwidth efficiently — speculative decoding is most valuable for latency-sensitive, lightly batched workloads, not for maximizing throughput on a saturated GPU.

Batching and continuous batching

If a single request wastes the GPU's compute, the fix is to serve many at once. Batching processes multiple sequences together so each expensive weight read amortizes across many tokens — the single most important lever for throughput and for cost per token. Naive static batching stalls, though: requests finish at different times, and a batch that waits for its slowest member leaves the GPU idle while short requests sit completed.

Continuous batching (also called in-flight batching) solves this by scheduling at the token level — the moment one sequence finishes, a queued request takes its slot, keeping the batch full. Paired with PagedAttention, which manages the KV cache in non-contiguous pages like virtual memory and eliminates the 60-80% waste of pre-allocating a contiguous cache per request, it is the reason modern serving engines reach many times the throughput of a naive generation loop. Underneath sits a throughput-latency tension: bigger batches lower cost per token but raise time-to-first-token for any individual user, so serving systems tune that tradeoff — and often split the compute-bound prefill phase from the memory-bound decode phase across different resources so neither starves the other.

MoE at inference

Mixture-of-experts models change the arithmetic. A MoE layer holds many expert sub-networks but routes each token to only a few, so the model carries a huge total parameter count while activating a small fraction per token — DeepSeek-V3, for instance, has 671 billion parameters but activates roughly 37 billion per token. That decouples capability, which tracks total parameters, from per-token compute, which tracks active parameters — a genuinely favorable trade. The catch moves to memory and systems: every expert must be resident in GPU memory even though most sit idle each step, and at scale experts are sharded across GPUs, turning inference into an all-to-all communication problem where routing imbalance can leave hardware idle. Complementary tricks like DeepSeek's multi-head latent attention — compressing the KV cache into a shared low-rank latent — pair with MoE to keep long-context serving affordable.

The serving stack

The tools that implement all of this have consolidated into a few serious options. vLLM is the widely adopted open default, built around PagedAttention and continuous batching with an OpenAI-compatible API — the pragmatic first choice when you need to serve something today. SGLang centers on RadixAttention, which reuses the KV cache across requests that share a prefix, and tends to win on multi-turn and heavily structured workloads. TensorRT-LLM compiles models to squeeze peak throughput out of NVIDIA hardware, with FP8 and FP4 support built in — fastest on the metal, at the cost of a compilation step and hardware lock-in. llama.cpp, with its GGUF format, owns local and edge inference on CPUs and consumer GPUs. The distinctions matter less than the fact that all four now implement the same core ideas; choosing one is mostly a question of your hardware, your workload shape, and how much you value peak throughput over portability.

The through-line is worth restating: inference optimization is applied economics. Every technique here trades some combination of accuracy, latency, throughput, and engineering complexity, and the right mix depends entirely on whether you are optimizing a chatbot's time-to-first-token, an agent's cost-per-completed-task, or a local model's ability to run on the hardware you own at all. The one constant is the memory wall — until that hardware fact changes, the winning tricks will keep being the ones that read fewer bytes, or wring more value from each byte read.