This paper explores the robustness of machine reading comprehension (MRC) models to entity renaming, focusing on entities from low-resource regions such as Africa. The authors propose a method called "EntSwap" to create a perturbed test set, AfriSQuAD2, by replacing entities in the SQuAD2.0 dataset with entities of African origin.
The key findings are:
All MRC models (BERT, RoBERTa, DeBERTa) show a performance drop on the AfriSQuAD2 dataset compared to the original SQuAD2.0 dataset, indicating a lack of robustness to entity renaming.
Larger models like DeBERTa-large and RoBERTa-large perform better than the BERT-large model on AfriSQuAD2, suggesting that increased model capacity can improve robustness to novel entities.
The analysis reveals that entity types like Person, Organization, and Location pose the greatest challenge to MRC model performance when renamed with African entities. This is likely due to the models being trained on a limited set of entities, mostly from high-resource regions.
The authors also find that the MRC models struggle more on questions with answerable spans than on unanswerable questions when the entities are renamed, indicating an over-reliance on entity-specific knowledge.
Overall, the study highlights the need for more diverse and representative datasets to improve the robustness of MRC models to entity renaming, especially for low-resource regions.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Clemencia Si... at arxiv.org 04-18-2024
https://arxiv.org/pdf/2304.03145.pdfDeeper Inquiries