Hugging Face has announced that its Text Generation Inference (TGI) toolkit is now available for AWS Inferentia2 accelerators, enabling efficient deployment of large language models (LLMs) on Amazon Web Services.
This integration allows developers to run popular open-source models like Llama, Mistral, and Falcon on AWS's custom silicon, which is designed for high-performance inference workloads. TGI optimizes text generation with features such as continuous batching, tensor parallelism, and quantization, now fully supported on Inferentia2.
According to the company, users can expect lower latency and cost compared to GPU-based alternatives, making it easier to scale LLM applications in production. The release includes pre-built Docker images and detailed setup guides for seamless deployment.