PORTULAN ExtraGLUE: Machine-Translated Datasets and Models for Benchmarking Portuguese Language Understanding

核心概念

This paper contributes a collection of machine-translated datasets for various language processing tasks in Portuguese, designated as PORTULAN ExtraGLUE, and a set of fine-tuned neural language models as baselines for these tasks.

摘要

The paper presents the PORTULAN ExtraGLUE benchmark, which consists of machine-translated versions of several well-known English language understanding tasks, including single sentence tasks, similarity tasks, inference tasks, question-answering tasks, and reasoning tasks. The authors discuss the challenges and limitations of using machine translation to create these datasets, such as issues with pronoun resolution, gendered nouns, and named entity translation.

To validate the datasets, the authors fine-tune low-rank adaptations (LoRA) of the Albertina language model, a state-of-the-art open encoder model for Portuguese, on 10 of the PORTULAN ExtraGLUE tasks. The resulting fine-tuned models are made available as baselines for future research.

The authors compare the performance of the Albertina LoRA models on the PORTULAN ExtraGLUE datasets to the performance of the multilingual XLM-RoBERTa-XL model and the English DeBERTa-V2-XXLarge model. While the Albertina LoRA models lag behind the English model, they outperform the multilingual model, demonstrating the benefits of using a monolingual model for Portuguese.

The authors acknowledge the limitations of machine-translated datasets and call for future work to improve the benchmark through manual curation and the development of new datasets from scratch to better reflect the Portuguese language and its cultural nuances.

统计

The PORTULAN ExtraGLUE datasets contain a total of 14 tasks, with varying numbers of training, validation, and test samples.
The number of tokens in the Portuguese variants (pt-PT and pt-BR) differs from the original English datasets, with the pt-BR version generally having more tokens.
The authors found an average of 8% machine translation errors, 2% label errors, and 2% low-quality translated samples across the datasets.

引用

"Even though MT datasets have their limitations and pitfalls, our manual analysis has found a relatively reduced amount of (translation and label) errors. We believe this renders our obtained datasets highly useful for assessing the comparative performance of neural language models for Portuguese."
"In future work, it would be important to improve this benchmark with manual curation of the datasets (in particular, the test sets) and expand it with new ones. Additionally, developing new datasets from scratch may better reflect the language and the cultures latent within language variants (which go well beyond European and Brazilian ones)."

从中提取的关键见解

PORTULAN ExtraGLUE Datasets and Models

by Tomá... 在 arxiv.org 04-09-2024

https://arxiv.org/pdf/2404.05333.pdf

更深入的查询

How can the PORTULAN ExtraGLUE benchmark be further improved to better capture the nuances and complexities of the Portuguese language?

To enhance the PORTULAN ExtraGLUE benchmark for Portuguese, several improvements can be considered:

Manual Curation: Incorporating human curation for dataset creation and validation can ensure higher quality translations and labels, capturing the nuances of the Portuguese language more accurately.

Diverse Task Selection: Including a wider range of language understanding tasks that cover various linguistic phenomena specific to Portuguese, such as idiomatic expressions, gendered nouns, and pronoun resolution, can provide a more comprehensive evaluation of language models.

Cultural Context: Introducing tasks that reflect the cultural context of Portuguese-speaking regions, such as specific idioms, cultural references, or regional variations in language usage, can make the benchmark more relevant and representative.

Fine-tuning Strategies: Exploring different fine-tuning approaches, beyond low-rank adaptation, to optimize model performance on Portuguese-specific tasks and datasets.

Leaderboard and Community Engagement: Establishing a public leaderboard for tracking model performance, fostering competition, and encouraging community participation and contributions to the benchmark's evolution.

What are the potential biases and limitations introduced by the machine translation process, and how can they be mitigated in future iterations of the benchmark?

Biases and limitations introduced by machine translation in the PORTULAN ExtraGLUE benchmark include:

Translation Errors: Inaccuracies in translation, especially for gendered nouns, pronoun resolution, and idiomatic expressions, can lead to incorrect labels and impact model performance.

Cultural Nuances: Machine translation may not capture cultural nuances specific to Portuguese, affecting the authenticity of the translated datasets.

Named Entities: Inconsistent translation of proper names and entities can lead to confusion and errors in downstream tasks.

To mitigate these issues in future iterations:

Human Validation: Incorporating human validation and correction of translations to ensure linguistic accuracy and cultural relevance.

Post-Editing: Implementing post-editing processes to refine machine-translated texts and correct errors introduced during translation.

Task-specific Evaluation: Conducting thorough evaluations of translated datasets to identify and address biases and limitations, especially in tasks sensitive to translation quality.

Continuous Improvement: Iteratively refining the translation process based on feedback and evaluation results to enhance the quality and reliability of the benchmark.

What other types of language understanding tasks or datasets could be included in the PORTULAN ExtraGLUE benchmark to provide a more comprehensive evaluation of Portuguese language models?

To broaden the evaluation scope of Portuguese language models, additional tasks and datasets can be included in the PORTULAN ExtraGLUE benchmark:

Sentiment Analysis: Incorporating sentiment analysis tasks in Portuguese to assess models' ability to understand and classify emotions in text.

Named Entity Recognition: Adding datasets for named entity recognition to evaluate models' proficiency in identifying and categorizing entities in Portuguese text.

Text Generation: Including tasks for text generation to test models' capability to generate coherent and contextually relevant text in Portuguese.

Dialogue Systems: Introducing datasets for dialogue systems to evaluate models' performance in engaging and contextually relevant conversations in Portuguese.

Document Classification: Adding tasks for document classification to assess models' ability to categorize and organize textual documents in Portuguese.

By including a diverse range of tasks that cover various aspects of language understanding and processing, the PORTULAN ExtraGLUE benchmark can offer a more comprehensive evaluation of Portuguese language models across different linguistic domains.