Transformer-based encoder-decoder models, introduced by Vaswani et al. (2017), have seen a surge in popularity with architectures like BART and T5. However, pre-training these models from scratch requires immense computational resources, limiting their development to well-funded organizations.
A 2020 paper by Rothe, Narayan, and Severyn proposes a solution: warm-start encoder-decoder models using pre-trained encoder-only (e.g., BERT) or decoder-only (e.g., GPT-2) checkpoints. This approach bypasses the costly pre-training phase while achieving competitive performance on sequence-to-sequence tasks like summarization and translation, at a fraction of the cost.
This article explains the theory behind warm-starting, reviews effective model combinations from the paper, and provides a practical code example using the Hugging Face Transformers library. The key insight is that initializing an encoder-decoder model with pre-trained components—such as BERT for the encoder and GPT-2 for the decoder—allows the model to leverage existing linguistic knowledge without full pre-training.
"The authors show that such warm-started encoder-decoder models yield competitive results to large pre-trained encoder-decoder models, such as T5 and Pegasus, on multiple sequence-to-sequence tasks at a fraction of the training cost."
Warm-starting democratizes access to powerful sequence-to-sequence models, enabling smaller research groups and companies to build effective systems without massive compute budgets.