Improving English to Ukrainian Machine Translation with Supervised Finetuning and Unsupervised Data Selection

מושגי ליבה

A two-phase approach to build a high-performing English-Ukrainian machine translation system by leveraging supervised finetuning on a large noisy dataset and unsupervised data selection on a high-quality dataset.

תקציר

The authors present a recipe for building an English-Ukrainian machine translation system using a two-phase approach.
In the first phase, they perform supervised finetuning of a large pretrained language model (Mistral-7B-v0.1) on a noisy parallel dataset of 3 million English-Ukrainian sentence pairs. They apply various heuristic filters to the publicly available Paracrawl dataset to control the quality of the training data, including language detection, perplexity thresholding, translation mismatch filtering, and length filtering.
In the second phase, the authors further finetune the model on the high-quality Extended Multi30K dataset. They employ unsupervised perplexity-based data selection, using k-fold cross-validation to identify and remove the most surprising sentences from the training set. This second phase provides an additional performance boost.
The authors' final model, named Dragoman, outperforms previous state-of-the-art encoder-decoder models on the FLORES-101 English-Ukrainian devtest set, achieving 32.3 BLEU. They also explore few-shot translation using pretrained models, but find that their finetuned Dragoman model still outperforms these approaches.
The authors discuss the limitations of their work, including the challenges of tokenization for Ukrainian, the choice of evaluation metric (BLEU), and the need for more nuanced translation quality assessment. They also highlight potential future directions, such as exploring the stability of long-context attention and incorporating community-informed metrics for Ukrainian.

סטטיסטיקה

The Paracrawl dataset contains 13,354,365 English-Ukrainian sentence pairs, with significant noise and quality issues.
After applying heuristic filters, the authors obtain subsets of 1 million, 3 million, and 8 million sentence pairs.
The Extended Multi30K dataset contains 29,000 high-quality English-Ukrainian sentence pairs.

ציטוטים

"To build large language models for Ukrainian we need to expand our corpora with large amounts of new algorithmic tasks expressed in natural language."
"Our decoder-only model named Dragoman beats performance of previous state of the art encoder-decoder models on the FLORES devtest set."

תובנות מפתח מזוקקות מ:

Setting up the Data Printer with Improved English to Ukrainian Machine Translation

by Yurii Paniv,... ב- arxiv.org 04-24-2024

https://arxiv.org/pdf/2404.15196.pdf

שאלות מעמיקות

How can the authors' approach be extended to other language pairs beyond English-Ukrainian

The authors' approach can be extended to other language pairs beyond English-Ukrainian by following a similar two-phase data cleaning pipeline. Firstly, a large parallel dataset for the target language pair needs to be obtained, similar to the Paracrawl dataset used in this study. The dataset can then undergo heuristic filtering based on language detection, perplexity thresholding, translation mismatch filtering, and length filtering to ensure data quality. In the second phase, a high-quality dataset specific to the target language can be selected for further training and fine-tuning. This dataset should be chosen based on its relevance and quality for the specific language pair. By following a systematic approach like the one outlined in the study, researchers can adapt the methodology to different language pairs, enabling the development of machine translation systems for various languages.

What are the potential challenges and considerations in applying unsupervised data selection techniques to a wider range of datasets and domains

Applying unsupervised data selection techniques to a wider range of datasets and domains may pose several challenges and considerations. One challenge is the diversity of data sources and the variability in data quality across different domains. Ensuring that the selected data is representative and of high quality is crucial for the effectiveness of the machine translation system. Additionally, domain-specific nuances and vocabulary may require specialized handling during data selection to improve translation accuracy. Another consideration is the scalability of the approach to handle large volumes of data efficiently. Developing automated processes for data selection and filtering can help streamline the process and make it more scalable. Furthermore, the generalizability of the unsupervised data selection techniques across different languages and domains needs to be evaluated to ensure consistent performance across diverse datasets.

How can the authors' work inform the development of more holistic evaluation frameworks for machine translation, beyond just BLEU scores

The authors' work can inform the development of more holistic evaluation frameworks for machine translation by emphasizing the limitations of traditional metrics like BLEU scores. While BLEU scores provide a quantitative measure of translation quality, they may not always align with human judgment. By highlighting the need for learned metrics and incorporating data from language communities for evaluation, the study underscores the importance of considering qualitative aspects of translation quality. Future evaluation frameworks could integrate a combination of quantitative metrics like BLEU scores with qualitative assessments based on human judgment or curated language data. This hybrid approach can provide a more comprehensive evaluation of machine translation systems, taking into account both the technical performance metrics and the linguistic nuances that impact translation quality.