Training large language models can be a daunting task, especially when access to top-tier GPUs is scarce. However, Tensor Processing Units (TPUs) offer a powerful alternative, enabling scalable and efficient training. In this guide, we walk through the complete process of training a masked language model from scratch using TensorFlow and TPUs, leveraging the Hugging Face Transformers library.
Introduction
TPU training is a valuable skill, as TPU pods provide high performance and scalability for models ranging from tens of millions to hundreds of billions of parameters. Google's PaLM model, for instance, was trained entirely on TPU pods. While previous tutorials focused on small-scale examples, this guide is designed for dedicated TPU nodes or VMs, offering a realistic, scalable training pipeline.
We rely on TensorFlow's robust TPU support via XLA and TPUStrategy, along with the fact that most TensorFlow models in 🤗 Transformers are XLA-compatible. This minimizes the effort needed to run them on TPUs.
Motivation
Despite years of TensorFlow support in 🤗 Transformers, TPU training has been a pain point due to XLA incompatibility and non-native TF data collators. With a recent push to make the codebase XLA-compatible, users can now train most models on TPUs without hassle. Additionally, as GPU availability becomes increasingly competitive, knowing how to train on TPUs provides an alternative path to high-performance compute.
What to Expect
We will train a RoBERTa base model from scratch on the WikiText dataset (v1). The process includes training a tokenizer, tokenizing the data, uploading it to Google Cloud Storage (GCS) in TFRecord format, and finally training the model. All code is available in this directory.
Getting the Data and Training a Tokenizer
The WikiText dataset is available on the Hugging Face Hub. We load the train split using 🤗 Datasets and train a Unigram tokenizer with 🤗 Tokenizers. The trained tokenizer is then uploaded to the Hub. The tokenizer training code can be found here, and the tokenizer itself here.
💡 Use 🤗 Datasets to host your text datasets; refer to this guide for details.
Tokenizing the Data and Creating TFRecords
After training the tokenizer, we apply it to all dataset splits (train, validation, test) and create TFRecord shards. Spreading data across multiple shards enables massively parallel processing. Each sample is tokenized individually, then batches of samples are concatenated and split into fixed-length chunks (128 tokens) to avoid excessive truncation. These chunks are serialized into TFRecord shards and uploaded to a GCS bucket.
The number of shards is determined by the total dataset length and desired shard size. This approach ensures efficient data loading during TPU training.
Training a Model on Data in GCS
With the data prepared in GCS, we can train the model using a TPU pod. The training script leverages TPUStrategy for distributed training and uses the Hugging Face Transformers TFRobertaForMaskedLM model. Key training configurations include:
- Learning rate schedule with warmup
- Weight decay
- Gradient accumulation if needed
- Mixed precision (bfloat16) for performance
The model is trained for a specified number of steps, with periodic evaluation on the validation set. After training, the model is saved and can be uploaded to the Hugging Face Hub.
Conclusion
Training language models on TPUs offers a scalable and efficient path, especially when GPUs are hard to come by. By following this end-to-end guide, you can train a BERT-sized model from scratch, but the principles apply to much larger models. The combination of Hugging Face Transformers, TensorFlow, and TPUs makes advanced NLP training accessible to more practitioners.