Deploying large language models (LLMs) like Meta's Llama 2 can be computationally intensive and latency-sensitive. To help businesses choose the best configuration on Amazon SageMaker, a new benchmark tested over 60 deployment setups using the Hugging Face LLM Inference Container. The study evaluated Llama 2 models (7B, 13B, and 70B parameters) across various EC2 instance types (g5 and p4d series) under different loads, measuring latency and throughput.
Three key use cases were identified:
- Most Cost-Effective: Using GPTQ 4-bit quantization on a single GPU (e.g., g5.2xlarge) for Llama 2 13B, balancing performance and cost.
- Best Latency: Minimizing per-token latency for real-time services by selecting powerful instances like p4d.24xlarge.
- Best Throughput: Maximizing tokens per second by leveraging larger instances and higher concurrency.
The benchmark data, including raw results and a processed spreadsheet, is publicly available on GitHub. This enables users to replicate tests and make informed decisions for their specific LLM deployment needs.
"We hope to enable customers to use LLMs and Llama 2 efficiently and optimally for their use case."
Key findings include that GPTQ quantization significantly reduces memory footprint without major accuracy loss, allowing smaller instances to handle larger models. For cost-sensitive deployments, Llama 2 13B with GPTQ on a single A10G GPU offers excellent throughput per dollar. For latency-critical applications, the A100-based p4d.24xlarge delivers the fastest token generation.
The full results are available in the spreadsheet and the GitHub repository.