Google DeepMind has introduced Decoupled DiLoCo, a novel distributed training architecture that allows large language models (LLMs) to be trained across geographically distant data centers with reduced bandwidth requirements and enhanced fault tolerance.
Unlike traditional tightly synchronized chip-based approaches, Decoupled DiLoCo separates training into independent learner units that can continue operating even when some hardware fails. This decoupling improves communication efficiency and overall system goodput.
Tests using the Gemma 4 model show that Decoupled DiLoCo maintains comparable machine learning performance while offering greater hardware flexibility and resilience. The approach could become a key infrastructure layer for next-generation frontier AI training, enabling more scalable and robust distributed systems.