Detecting Template-based Translation in the Egyptian Arabic Wikipedia: An Exploratory Analysis and Automated Identification

Core Concepts

The core message of this paper is to mitigate the problem of template-based translation that occurred in the Egyptian Arabic Wikipedia by identifying these template-translated articles and their characteristics through exploratory analysis and building automatic detection systems.

Abstract

The paper explores the content of the three Arabic Wikipedia editions (Arabic Wikipedia, Egyptian Arabic Wikipedia, and Moroccan Arabic Wikipedia) in terms of density, quality, and human contributions. It highlights how the template-based translation that occurred on the Egyptian Wikipedia produces unrepresentative content.
The key highlights and insights from the exploratory analysis are:

The Egyptian Wikipedia has a greater number of total articles than the Arabic Wikipedia, but a substantial portion of these articles (46%) are under 50 tokens, indicating limited and shallow content.
The Egyptian Wikipedia exhibits lower lexical richness and diversity compared to the other Arabic Wikipedia editions, suggesting the template translation produced poor-quality content.
The Egyptian Wikipedia has a high number of duplicate n-grams, especially for n>=5, indicating the use of templates in the translation process.
The Egyptian Wikipedia's articles are 100% created by humans, but 42.72% of them are automatically template-translated from English without human supervision.
The paper then attempts to build powerful multivariate machine learning classifiers leveraging articles' metadata to detect the template-translated articles automatically. The key findings are:

Supervised classification algorithms, particularly ensemble methods like Random Forest and XGBoost, outperform unsupervised clustering algorithms in detecting the template-translated articles.
The metadata features related to the total edits, total editors, total bytes, total characters, and total words are effective in training the classifiers to identify the template-translated articles.
The paper publicly deploys the best-performing classifier, XGBoost, as an online application called "Egyptian Wikipedia Scanner" and releases the extracted, filtered, and labeled datasets to the research community.
Finally, the paper discusses the negative implications of the template-based translations on the Egyptian Wikipedia, including societal, representation, and performance issues. It argues that such practices could misrepresent the native speakers and their culture and negatively impact the performance of language models and NLP tasks trained on these corpora.

Stats

The Egyptian Arabic Wikipedia edition has nearly 1.6 million total articles, of which 46% (741K) are under 50 tokens per article.
The Egyptian Arabic Wikipedia has the lowest mean values of total characters (610) and total tokens/words (100) compared to the Arabic Wikipedia and Moroccan Wikipedia.
The Egyptian Arabic Wikipedia has a high number of duplicate n-grams, especially for n>=5, indicating the use of templates in the translation process.
42.72% of the articles in the Egyptian Arabic Wikipedia are automatically template-translated from English without human supervision.

Quotes

"We first explore the content of the three Arabic Wikipedia editions in terms of density, quality, and human contributions and utilize the resulting insights to build multivariate machine learning classifiers leveraging articles' metadata to detect the template-translated articles automatically."
"We then publicly deploy and host the best-performing classifier, XGBoost, as an online application called Egyptian Wikipedia Scanner and release the extracted, filtered, and labeled datasets to the research community to benefit from our datasets and the online, web-based detection system."
"We argue that such automatic template-based translations without humans in the loop could misrepresent the Egyptian Arabic native speakers, where instead of the Egyptian people enriching the content of Wikipedia by sharing their voices, opinions, knowledge, perspectives, and experiences, a couple of registered users automated the creation and translation of more than a million and a half million articles (95.56%) from English on their behalf without supervision or revision of the translated articles."

Key Insights Distilled From

Leveraging Corpus Metadata to Detect Template-based Translation

by Saied Alshah... at arxiv.org 04-02-2024

https://arxiv.org/pdf/2404.00565.pdf

Deeper Inquiries

How can the research community work with the Wikimedia Foundation to address the issue of template-based translation and ensure the representation of native speakers' perspectives in Wikipedia?

To address the issue of template-based translation and ensure the representation of native speakers' perspectives in Wikipedia, the research community can collaborate with the Wikimedia Foundation in several ways. Firstly, researchers can work on developing automated tools or algorithms to detect template-translated articles, similar to the approach outlined in the study. By identifying these articles, researchers can flag them for review by human editors to ensure accuracy and cultural relevance. Additionally, researchers can provide insights and recommendations to the Wikimedia Foundation on best practices for content creation and translation to maintain the authenticity and diversity of perspectives on Wikipedia. Collaborative efforts between researchers and the Wikimedia Foundation can lead to the implementation of policies or guidelines to prevent the mass creation of template-translated articles and promote the creation of organic, culturally relevant content by native speakers.

What are the potential biases and stereotypes that could be introduced into language models and NLP systems trained on the template-translated articles from the Egyptian Arabic Wikipedia, and how can these be mitigated?

Training language models and NLP systems on template-translated articles from the Egyptian Arabic Wikipedia can introduce biases and stereotypes into the models. Some potential biases include gender bias, cultural bias, and linguistic bias. For example, the use of off-the-shelf translation tools like Google Translate may introduce inaccuracies in gender representation or cultural nuances. To mitigate these biases, researchers can implement bias detection algorithms to identify and correct biased language in the training data. They can also incorporate diverse datasets from multiple sources to ensure a balanced representation of different perspectives. Additionally, researchers can fine-tune the language models on a more diverse and representative dataset to reduce the impact of biases from template-translated articles.

How can the insights from this study on detecting template-based translation be applied to other low-resource language Wikipedias to ensure the quality and representativeness of the content?

The insights from this study on detecting template-based translation can be applied to other low-resource language Wikipedias to ensure the quality and representativeness of the content. Researchers can develop similar detection systems tailored to the specific characteristics of each language Wikipedia edition. By analyzing metadata, content density, and lexical richness, researchers can identify patterns indicative of template-based translation and flag potentially problematic articles for review. Moreover, researchers can collaborate with local language communities to provide guidelines and training on creating high-quality, culturally relevant content. By sharing the tools, methodologies, and best practices developed in this study, the research community can contribute to improving the overall quality and authenticity of content in low-resource language Wikipedias.