PeLLE: Encoder-based Language Models for Brazilian Portuguese Using Open Data

Kernkonzepte

PeLLE introduces large language models for Brazilian Portuguese, emphasizing the importance of data curation and model performance in downstream tasks.

Zusammenfassung

PeLLE presents a family of language models based on RoBERTa architecture trained on open data from the Carolina corpus. The study evaluates model performance in various NLP tasks, highlighting the impact of data size and curation. Results show that larger models outperform smaller ones in some tasks, but curated data benefits specific applications.
The Carolina Corpus is described as a curated dataset with provenance and typological diversity, enabling investigations in linguistics and AI. The corpus construction methodology ensures open-licensed texts with detailed metadata.
Evaluation on NLP tasks like natural language inference and hate speech identification shows competitive results for PeLLE models compared to existing Portuguese-specific models like Bertimbau and Albertina-PTBR. Multilingual models like mBERT also play a role in initialization.
Performance evaluation on datasets like ASSIN, HateBR, and Acórdãos TCU demonstrates the effectiveness of PeLLE models in classification, multiclass classification, and regression tasks. The use of law-related documents enhances model performance on legal-domain tasks.
The study concludes that larger models excel in certain tasks while smaller-but-curated data benefits others. PeLLE's transparency in licensing and reproducibility sets a standard for future language model development using open datasets.

Statistiken

Version 1.2 of Carolina Corpus has 823 million words across more than 2 million texts.
HateBR dataset contains 7,000 Instagram comments annotated for offensive content.
Acórdãos TCU dataset consists of 35,414 samples with an average document length of 225.75 words.

Zitate

"In this work we focus specifically on Encoder-based LLMs."
"Models pretrained on open data can be used with no restriction."
"The use of law-related documents enhances model performance."

Wichtige Erkenntnisse aus

PeLLE

by Guilherme La... um arxiv.org 03-01-2024

https://arxiv.org/pdf/2402.19204.pdf

Tiefere Fragen

How does the use of curated data impact the performance of language models?

Curated data plays a crucial role in enhancing the performance of language models. When data is carefully selected, organized, and cleaned, it leads to higher-quality training sets for language models. Curated data ensures that irrelevant or noisy information is minimized, allowing the model to focus on relevant patterns and relationships within the text. This focused attention results in more accurate predictions and better generalization to unseen data.
By using curated data, language models can learn from high-quality examples that are representative of the target domain or task. This targeted learning helps improve model efficiency and effectiveness in understanding complex linguistic structures, nuances, and contexts present in natural language text. Additionally, curated datasets often contain annotations or labels that provide valuable supervision during training, guiding the model towards learning specific tasks or objectives effectively.
Overall, leveraging curated data sets in pretraining language models leads to improved performance metrics such as accuracy, precision, recall rates across various NLP tasks like classification, sentiment analysis, named entity recognition (NER), among others.

What are the implications of using law-related documents in pretraining language models?

The inclusion of law-related documents in pretraining language models has several significant implications:

Domain-specific Understanding: Incorporating legal texts into pretraining exposes the model to specialized vocabulary and syntax commonly found in legal documents. This exposure enhances the model's comprehension of legal jargon and terminology unique to this domain.

Improved Performance on Legal Tasks: Training with law-related texts equips the model with knowledge about legal concepts and principles necessary for performing well on legal-specific tasks such as contract analysis, case summarization,...

3....