Researchers at NTT have demonstrated that scaling the width of neural networks eliminates the need for complex alignment when merging different models. Their findings, presented at ICLR 2026, show that high-dimensional geometry simplifies at scale, enabling weight averaging without parameter permutation. This ensemble effect boosts performance without increasing inference compute or requiring shared training data.
The work addresses a key challenge in model merging: traditionally, combining neural networks required intricate alignment of their internal representations. The new research proves that as model width increases, the geometry of the weight space becomes more amenable to simple averaging, making alignment unnecessary.
This discovery could significantly streamline model integration, allowing practitioners to combine multiple trained models efficiently. Merging models improves performance akin to ensemble methods but without the computational overhead of running multiple models during inference.
The study has implications for distributed learning and model reuse, potentially reducing storage and computational costs.