TartuNLP's Submission to the EvaLatin 2024 Shared Task: Detecting Emotion Polarity in Historical Latin Texts

Core Concepts

The TartuNLP team developed a system that achieved the overall first place in the Emotion Polarity Detection task of the EvaLatin 2024 Evaluation Campaign by leveraging two distinct approaches to annotate training data: heuristics-based labeling using a provided polarity lexicon and LLM-generated labels.

Abstract

The TartuNLP team participated in the Emotion Polarity Detection task of the EvaLatin 2024 Evaluation Campaign, where the goal was to label Latin texts from three historical authors with four emotion polarity labels: positive, negative, neutral, or mixed.
The team employed two approaches to annotate the training data:

Heuristics-based annotation: They used the provided polarity lexicon to assign labels to sentences based on rules like the mean polarity of the words in the sentence. This resulted in 15,396 annotated sentences.

LLM-based annotation: They used the GPT-4 model to assign labels to sentences and provide explanations. This resulted in 7,281 annotated sentences.

The team then fine-tuned a multilingual BERT-based model (XLM-RoBERTa) using the adapter framework for parameter-efficient knowledge transfer. They explored both monolingual transfer from a Latin corpus and cross-lingual transfer from an English sentiment analysis task.
The team made two submissions to the shared task: one using the heuristically annotated data (TartuNLP_1) and one using the LLM-annotated data (TartuNLP_2). Both submissions performed competitively, with TartuNLP_2 taking the overall first place.
The ablation study showed that the model trained on the LLM-annotated data benefited more from the monolingual knowledge transfer and outperformed the initial submission. The results suggest that the LLM-based annotations are of higher quality than the lexicon-informed heuristic labels.
The team discussed potential reasons for the relatively low absolute scores, such as the difficulty of assigning emotion polarity labels to expository or narrative texts, and the challenge of distinguishing between mixed and neutral labels. They proposed reframing the task as a multi-label classification problem as a possible improvement.

Stats

The heuristics-based approach resulted in 15,396 annotated sentences, while the LLM-based approach resulted in 7,281 annotated sentences.

Quotes

"The model with LLM-generated labels obtained better results than the model with lexicon-based heuristic labels, although the final results of both submitted systems are relatively close."
"The ablation study showed that the model trained on the LLM-annotated data benefited more from the monolingual knowledge transfer and outperformed the initial submission."

Key Insights Distilled From

TartuNLP at EvaLatin 2024: Emotion Polarity Detection

by Aleksei Dork... at arxiv.org 05-03-2024

https://arxiv.org/pdf/2405.01159.pdf

Deeper Inquiries

How could the task be further improved by incorporating contextual information beyond the sentence level?

Incorporating contextual information beyond the sentence level can significantly enhance the emotion polarity detection task. One approach could be to consider the surrounding sentences or paragraphs to capture the broader context in which a particular sentence appears. This contextual information can provide valuable cues for understanding the emotional tone of the text more accurately. By analyzing the relationships between sentences or paragraphs, the model can better grasp the overall sentiment and emotional flow of the text, leading to more precise emotion polarity predictions.
Furthermore, incorporating metadata such as the author's background, historical context, or genre of the text can also provide additional context for emotion analysis. Understanding the historical and cultural context in which the text was written can help in interpreting the emotional nuances and expressions more effectively. By leveraging metadata and contextual information, the emotion polarity detection system can achieve a deeper understanding of the text and improve the accuracy of emotion classification.

What other techniques, such as multi-task learning or ensemble methods, could be explored to improve the overall performance of the emotion polarity detection system?

To enhance the performance of the emotion polarity detection system, exploring techniques like multi-task learning and ensemble methods can be beneficial.

Multi-task Learning: By incorporating related tasks such as sentiment analysis, emotion recognition, or language modeling into the training process, the model can learn to extract more nuanced features and representations that are beneficial for emotion polarity detection. Multi-task learning allows the model to leverage shared knowledge across tasks, leading to improved generalization and performance on emotion classification tasks.

Ensemble Methods: Ensemble methods, such as combining predictions from multiple models or using different architectures, can help improve the robustness and accuracy of the emotion polarity detection system. Techniques like model averaging, stacking, or boosting can be employed to combine the strengths of individual models and mitigate their weaknesses, resulting in more reliable predictions. Ensemble methods can enhance the overall performance by reducing overfitting and capturing diverse patterns in the data.

Exploring these techniques in conjunction with the existing framework can lead to a more sophisticated and effective emotion polarity detection system with enhanced performance and generalization capabilities.

How might the insights from this work on Latin texts be applied to emotion analysis in other historical or low-resource languages?

The insights gained from working on emotion analysis in Latin texts can be extrapolated to emotion analysis in other historical or low-resource languages in the following ways:

Transfer Learning: The techniques and methodologies developed for emotion polarity detection in Latin texts can be transferred to other historical languages with similar characteristics. By fine-tuning pre-trained models on data from different historical languages, the models can leverage the knowledge gained from Latin texts to improve emotion analysis in other languages.

Data Augmentation: The annotation strategies and data augmentation techniques used for Latin texts can be adapted for other low-resource languages to enhance the training data quality and quantity. By leveraging similar approaches to generate labeled data, the emotion analysis models for low-resource languages can benefit from improved training data.

Contextual Understanding: The importance of contextual information and metadata in emotion analysis, as highlighted in the study on Latin texts, can be applied to other historical or low-resource languages. By considering the broader context, cultural nuances, and historical background of texts in these languages, the emotion analysis models can achieve more accurate and culturally sensitive predictions.

By applying the insights and methodologies developed for Latin texts to other historical or low-resource languages, researchers can advance emotion analysis capabilities in diverse linguistic contexts.