Hugging Face has released a new library, pytorch_block_sparse, that brings block sparse matrix support to PyTorch, enabling neural networks to be both smaller and faster. The library aims to address the lack of efficient sparse algebra computation in current frameworks.
Sparse matrices replace dense layers that are often overkill and can be pruned without significant loss of precision. In some cases, sparse linear layers can even improve precision or generalization.
The BlockSparseLinear module is a drop-in replacement for torch.nn.Linear, and the BlockSparseModelPatcher allows existing models to be modified on the fly. The library builds on NVIDIA's CUTLASS templates for block-sparse matrix multiplication, leveraging CUDA for high performance.
Currently, sparse matrices are roughly two times slower than their dense counterparts, but a 75% sparse matrix is about 2x faster than dense. Memory savings are significant: 75% sparsity reduces memory consumption by 4x.
Future work includes optimizing the sparsity pattern during learning and leveraging NVIDIA Ampere's 50% sparse pattern for further gains.