SpeechT5, a unified model for spoken language processing, is now available in the Hugging Face Transformers library. Originally developed by Microsoft Research Asia, this model can perform three distinct tasks: text-to-speech, speech-to-text, and speech-to-speech conversion.
How It Works
SpeechT5 uses a Transformer encoder-decoder backbone, similar to other Transformer models. To handle both text and speech inputs and outputs, it employs specialized pre-nets and post-nets. During pre-training, the model learns from a mixture of text-to-speech, speech-to-text, text-to-text, and speech-to-speech data, creating a shared hidden representation space for both modalities.
Text-to-Speech
For text-to-speech, SpeechT5 uses a text encoder pre-net, a speech decoder pre-net, and a speech decoder post-net. This is the first TTS model added to the Transformers library. Users can generate speech by providing text tokens and a speaker embedding, which captures the voice characteristics of a particular speaker. A vocoder, such as HiFi-GAN, is then used to convert the generated spectrogram into an audio waveform.
Voice Conversion and Speech Recognition
Beyond TTS, SpeechT5 can convert speech from one voice to another (speech-to-speech) and perform automatic speech recognition (speech-to-text). These capabilities make it a highly versatile tool for various speech processing applications.
Getting Started
To use SpeechT5, install the latest version of Transformers from GitHub along with the sentencepiece library. Pre-trained models and a HiFi-GAN vocoder are available on the Hugging Face Hub. Interactive demos are also provided for quick experimentation.
"SpeechT5 is flexible, but not that flexible." — The fine-tuned models are task-specific and cannot be swapped between tasks without retraining.