084 - Speculative Decoding: Making LLMs Generate Faster

When a large language model generates text one token at a time, even a fast GPU hits a sequential bottleneck. This episode explains speculative decoding, an inference technique that prepares several tokens at once and checks them together, and when the extra computation pays off.

Ordinary generation makes the target model run a full forward pass for every single token, since each prediction depends on the one before it. Speculative decoding adds a cheap draft step: a smaller model, a lightweight prediction head, or a pattern-matching method proposes a short run of likely next tokens, and the target model verifies the whole batch in one pass. Using the illustrative example "The capital of France is" followed by candidate tokens "Paris," "and," "it," "is," the episode walks through how the target model accepts a correct prefix, rejects a wrong candidate, and continues generating from that rejection point rather than discarding everything. Done with the right verification and sampling procedure, this preserves the target model's exact output distribution — the speedup doesn't require swapping in a weaker model. The original 2023 ICML paper introducing the technique reported a 2x-3x acceleration in its T5 experiments, a result specific to that setup rather than a universal guarantee.

The episode covers the main ways to build a draft: a separate smaller model (which needs a compatible tokenizer with the target), prediction heads or internal-feature-based methods like EAGLE, multi-token prediction, and N-gram or prompt-lookup methods that reuse matching text already in the prompt without a second model at all. It explains why the benefit depends on acceptance rate, draft cost, and available spare compute, and why speculative decoding can even reduce throughput once a GPU is already busy serving many concurrent requests.

The episode closes with concrete evaluation guidance: measure time to first token, inter-token latency, accepted-draft rate, memory overhead, and total throughput across varied workloads and concurrency levels, since a setup that feels faster for one user can perform worse once the server gets busy.

Sources & References

Leviathan, Kalman, and Matias: Fast Inference from Transformers via Speculative Decoding (ICML 2023) — https://proceedings.mlr.press/v202/leviathan23a.html

vLLM Documentation: Speculative Decoding — https://docs.vllm.ai/en/latest/features/speculative_decoding/index.html

NVIDIA TensorRT-LLM Documentation: Speculative Decoding — https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/features/speculative-decoding.md

NVIDIA Triton Inference Server Documentation: Speculative Decoding with TensorRT-LLM — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Feature_Guide/Speculative_Decoding/TRT-LLM/README.html

Voice narration is AI-generated.