In the second part of our series on scaling BERT-like model inference on modern CPUs, we dive into software optimizations that unlock the full potential of Intel's latest Xeon processors. Following our earlier exploration of hardware features like AVX512 and VNNI, this article focuses on leveraging Intel's software stack—from memory allocators to parallelization frameworks—to achieve significant performance gains.
Intel's Ice Lake Xeon CPUs, launched in April, promise up to 75% faster inference on NLP tasks compared to the previous Cascade Lake generation. This boost comes from both hardware improvements, such as the Sunny Cove architecture and PCIe 4.0, and software enhancements. Key components include Intel's oneAPI suite, which provides optimized libraries like oneMKL for linear algebra, oneDNN for deep neural network primitives, and oneTBB for threading. Frameworks like PyTorch and TensorFlow integrate these libraries natively, while Intel-specific versions (e.g., Intel TensorFlow, Intel PyTorch Extension) offer additional tuning.
Deep Dive: Performance Tuning Knobs
At a high level, every ML/DL framework relies on three core ingredients:
- Memory representation of data (vectors, matrices, etc.)
- Efficient parallelization of computations
- Optimized mathematical operators
We'll explore tunable parameters for each, including memory allocators, OpenMP settings, and BLAS libraries. Benchmarks on Ice Lake CPUs demonstrate that simply switching to Intel's memory allocator (jemalloc) can reduce latency by up to 15%, while proper OpenMP configuration yields another 10-20% improvement. Advanced techniques like Bayesian optimization with Intel SigOpt automate the search for optimal settings.
Conclusion: Accelerating Transformers for Production
By combining hardware features with a meticulously tuned software stack, practitioners can deploy BERT-scale models on CPU with latency and throughput suitable for production. The tools and techniques discussed here are readily available and can be applied to other transformer architectures, making CPU inference a viable alternative for cost-sensitive or latency-constrained applications.