UMBCLU's Semantic Textual Relatedness Models for African and Asian Languages with and without Machine Translation

Core Concepts

The authors developed two models, TranSem and FineSem, to identify semantic textual relatedness between sentence pairs in 14 African and Asian languages, exploring the effectiveness of machine translation and different training strategies.

Abstract

The authors participated in the SemEval-2024 Task 1 on Semantic Textual Relatedness for African and Asian Languages. They developed two models, TranSem and FineSem, to address the task:
TranSem Model:

Uses a Siamese network architecture to encode sentence pairs and train a cosine similarity loss to match the semantic relatedness score.
Experiments with various sentence encoding models, including DistilRoBERTa, and finds that mean pooling works well.
Explores the usefulness of machine translation by translating the training data to English using multiple translation models.
FineSem Model:

Fine-tunes T5 models on the semantic textual similarity (STS) task, using both the untranslated and translated training data.
Compares the performance of individual T5 models fine-tuned on each language, a unified T5 model trained on all languages, and a T5 model trained on the translated and augmented data.
Finds that direct fine-tuning with the translated and augmented data is comparable to the TranSem model using various sentence embeddings.
For the cross-lingual Track C, the authors use the T5 models fine-tuned on the English and Spanish datasets to evaluate the other languages.
The authors' models outperform the official baseline for some languages in both the supervised and cross-lingual settings. They also explore the effectiveness of machine translation and find that it can lead to better performance for certain languages.

Stats

The authors used a batch size of 32 for the TranSem model and 16 for the FineSem model.
Mean pooling performed better than max pooling and CLS token pooling for the TranSem model.
The FineSem model trained on the translated and augmented data performed comparably to the TranSem model using various sentence embeddings.

Quotes

None.

Key Insights Distilled From

UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation

by Shubhashis R... at arxiv.org 04-15-2024

https://arxiv.org/pdf/2402.12730.pdf

Deeper Inquiries

How can the authors further improve the performance of their models, especially for the languages where they did not outperform the baseline

To further improve the performance of their models, especially for the languages where they did not outperform the baseline, the authors can consider the following strategies:

Fine-tuning on Language-Specific Data: Instead of relying solely on translated and augmented data, the authors can fine-tune their models on language-specific datasets. This approach can help capture language nuances and improve performance for languages where the baseline was not surpassed.
Data Augmentation Techniques: Exploring advanced data augmentation techniques, such as back-translation or synthetic data generation, can help in enhancing the model's ability to generalize better across languages with limited training data.
Ensemble Models: Combining the outputs of multiple models or ensembling different architectures can often lead to improved performance. By leveraging diverse models, the authors can potentially boost the overall performance across all languages.
Hyperparameter Tuning: Fine-tuning hyperparameters like learning rate, batch size, and optimizer settings can significantly impact model performance. Systematic exploration and optimization of these parameters can lead to better results.

What other techniques, such as prompt-based learning or few-shot learning, could be explored to enhance the models' ability to handle low-resource languages

To enhance the models' ability to handle low-resource languages, the authors could explore the following techniques:

Prompt-Based Learning: Implementing prompt-based learning can help the models better understand the task and improve their performance, especially in scenarios with limited training data. By providing specific prompts related to semantic textual relatedness, the models can learn to focus on relevant information.
Few-Shot Learning: Incorporating few-shot learning techniques can enable the models to generalize better with minimal training examples. By exposing the models to a small number of examples from low-resource languages, they can learn to make accurate predictions even with limited data.
Zero-Shot Learning: Extending the models to zero-shot learning, where they can perform tasks without explicit training on them, can be beneficial for handling languages with scarce resources. By leveraging transfer learning and cross-lingual embeddings, the models can infer patterns from related languages to make predictions in unseen languages.

How can the authors investigate and mitigate potential biases in the training data and models, which may impact the fairness and robustness of the semantic textual relatedness task

To investigate and mitigate potential biases in the training data and models for the semantic textual relatedness task, the authors can take the following steps:

Bias Analysis: Conduct a thorough analysis of the training data to identify any biases related to social groups, cultural aspects, or language-specific nuances. This analysis can help in understanding the underlying biases that may impact model performance.
De-biasing Techniques: Implement de-biasing techniques such as adversarial training, bias correction layers, or fairness constraints during model training. These methods can help in reducing biases and promoting fairness in the models' predictions.
Diverse Dataset Collection: Ensure diversity in the training data by collecting samples from a wide range of sources and demographics. By including diverse perspectives, the models can learn to make more inclusive and unbiased predictions.
Bias Mitigation Strategies: Develop strategies to mitigate biases at different stages of the model development process, including data preprocessing, model architecture design, and evaluation metrics. Regularly auditing the models for biases and taking corrective actions is essential for ensuring fairness and robustness.