Large language models (LLMs) are powerful but can be slow when handling many simultaneous requests. A key optimization technique involves separating the process into two phases: prefill and decode.
During the prefill phase, the model processes the entire input prompt in parallel, computing key-value (KV) caches once. This step is compute-bound, meaning the GPU's arithmetic units are fully utilized. By batching multiple prompts together, we can maximize throughput because the GPU works efficiently on large matrix operations.
Once the KV caches are built, the decode phase generates tokens one at a time. Here, the bottleneck shifts to memory bandwidth: each new token must read the KV cache for all previous tokens. To handle many requests concurrently, the system must interleave the decode steps of different batches, ensuring that memory access patterns keep the GPU busy.
Engineers use several tactics to optimize these phases:
- Dynamic batching: Group incoming requests into batches for prefill, then split them into sub-batches for decode to balance load.
- Prefix caching: Store KV caches for common input prefixes (e.g., system prompts) to avoid redundant computation.
- Pipeline parallelism: Use multiple GPUs, with one set dedicated to prefill and another to decode, reducing idle time.
- Speculative decoding: Generate multiple token candidates in one pass to skip some decode iterations.
By carefully orchestrating prefill and decode across concurrent requests, systems like vLLM and TensorRT-LLM achieve significant latency reductions and higher throughput. These optimizations are critical for deploying LLMs in real-time applications such as chatbots, code assistants, and content generation tools.
Efficient handling of concurrent requests is the key to making LLMs practical at scale.
Developers should profile their specific workload to choose the right combination of batching and scheduling strategies.