A new collaborative training method called DeDLOC enables distributed deep learning over the internet, overcoming bandwidth and reliability issues. The approach was successfully used to pretrain sahajBERT, a Bengali language model, with 40 volunteers, achieving near state-of-the-art results.
Modern language models typically require vast computational resources, often beyond the reach of individuals or small organizations. However, a team of researchers has introduced a method that allows volunteers to pool their home computers' GPUs to train large models collaboratively.
The key innovation is a technique called Distributed Deep Learning in Open Collaborations (DeDLOC). Unlike traditional distributed training that relies on fast interconnects, DeDLOC adapts to participants' varying network speeds and hardware capabilities. It works by accumulating gradients from many volunteers before each optimizer step, effectively creating a large batch size. This approach naturally handles peer disconnections: if a volunteer drops out, their contribution is simply excluded, and the training continues with others.
To manage communication efficiently, DeDLOC uses adaptive averaging. It splits the gradient vector into parts based on each peer's internet speed, with faster connections handling larger portions. Some nodes may only send data without receiving, preventing bottlenecks. This flexibility makes the method robust to diverse network conditions.
The researchers demonstrated DeDLOC by training sahajBERT, a Bengali language model, using 40 volunteers. Despite relying on consumer-grade hardware and internet connections, the model achieved performance comparable to much larger models trained on hundreds of high-end GPUs.
"The computational resources we're looking for might already be there," the authors note, drawing parallels to volunteer computing projects like Folding@home.
This work opens the door for more inclusive AI development, where researchers and enthusiasts can contribute to training state-of-the-art models without massive budgets.