A Multi-Speaker Multi-Lingual Few-Shot Voice Cloning System for the LIMMITS'24 Challenge

Core Concepts

The THU-HCSI team developed a multi-speaker multi-lingual few-shot voice cloning system for the LIMMITS'24 Challenge, which achieved the best speaker similarity MOS of 4.25 and a considerable naturalness MOS of 3.97.

Abstract

The THU-HCSI team developed a multi-speaker multi-lingual few-shot voice cloning system for the LIMMITS'24 Challenge. The system is built upon YourTTS and incorporates several enhancements inspired by VITS2, including a speaker-aware text encoder, a flow-based decoder with Transformer blocks, and noise-injected monotonic alignment search.
For data preprocessing, the team resampled, normalized, and denoised the audios, and mixed up the few-shot data with pre-training data. During fine-tuning, they adopted a speaker-balanced sampling strategy to ensure effective training for target speakers.
The official evaluations in track 1 showed that the proposed system achieved the best speaker similarity MOS of 4.25 and obtained a considerable naturalness MOS of 3.97, outperforming other teams.

Stats

The competition provided two datasets: a TTS training dataset of 560 hours of studio-quality data in 7 Indian languages, and a few-shot dataset of 9 target speakers with about 5 minutes of speech each.

Quotes

"To achieve high speaker similarity and speech quality, we build the system upon YourTTS and incorporate several modifications from VITS2."
"For data preprocessing, we resample, normalize and denoise the audios. For model training, we mix up few-shot data with pre-training data and adopt a speaker-balanced sampling strategy during fine-tuning."

Key Insights Distilled From

The THU-HCSI Multi-Speaker Multi-Lingual Few-Shot Voice Cloning System for LIMMITS'24 Challenge

by Yixuan Zhou,... at arxiv.org 04-26-2024

https://arxiv.org/pdf/2404.16619.pdf

Deeper Inquiries

How can the proposed system be further improved to achieve even higher speaker similarity and naturalness, especially for low-resource languages?

To enhance speaker similarity and naturalness, especially for low-resource languages, the system can benefit from several improvements. Firstly, incorporating more diverse and extensive training data specific to low-resource languages can help the model better capture the nuances and characteristics of these languages, leading to improved speaker similarity. Additionally, fine-tuning the model with data augmentation techniques such as speed perturbation, noise injection, and reverberation can further enhance the system's ability to generalize to different speaking styles and environments, ultimately improving naturalness. Moreover, exploring advanced techniques like transfer learning from related languages or leveraging unsupervised pre-training methods can also boost performance in low-resource language scenarios.

What are the potential challenges in extending the system to support zero-shot voice cloning, as required in track 3 of the LIMMITS'24 Challenge?

Extending the system to support zero-shot voice cloning, as required in track 3 of the LIMMITS'24 Challenge, poses several challenges. One major challenge is the lack of speaker-specific data during training, making it difficult for the model to accurately capture the unique characteristics of unseen speakers. This can lead to issues such as poor speaker similarity and unnatural voice synthesis. Additionally, zero-shot voice cloning requires the model to generalize well across a wide range of speakers, languages, and speaking styles, which can be a complex task due to the inherent variability in human speech. Ensuring robustness and adaptability to unseen data while maintaining high-quality voice synthesis poses a significant challenge in zero-shot voice cloning scenarios.

How can the system's performance be evaluated on real-world applications beyond the challenge, such as personalized voice assistants or audiobook narration?

To evaluate the system's performance in real-world applications beyond the challenge, such as personalized voice assistants or audiobook narration, several key metrics and criteria can be considered. Firstly, conducting user studies and collecting feedback from end-users can provide valuable insights into the system's usability, naturalness, and overall user experience. Objective metrics like word error rate, prosody evaluation, and speaker similarity can also be used to assess the system's performance in different application scenarios. Furthermore, benchmarking the system against industry standards and existing commercial TTS solutions can help gauge its competitiveness and suitability for real-world deployment. Continuous monitoring and feedback loops from users in real-world applications can further refine the system and ensure its effectiveness in meeting the specific requirements of personalized voice assistants or audiobook narration.