New Chip Design Aims to Slash Latency for Million-Token AI Contexts

Researchers have introduced AMMA, a multi-chiplet memory-centric architecture designed to dramatically reduce latency when serving large context windows of up to one million tokens in AI models. The architecture addresses the memory bottleneck that plagues current attention mechanisms, where memory bandwidth and capacity limit the ability to process long sequences efficiently. By distributing memory across multiple chiplets and optimizing data flow, AMMA achieves low-latency inference for extended contexts, a critical need for applications like long-document analysis, code generation, and conversational AI. The paper, authored by a team including researchers from academia and industry, presents detailed simulations showing significant performance improvements over traditional monolithic designs. AMMA represents a step toward making billion-parameter models practical for real-time tasks with very long inputs.