Multilingual Language Models Show Imbalanced Cross-Lingual Knowledge Alignment, with Limited Improvement from Pretraining and Finetuning

Core Concepts

Current multilingual language models exhibit imbalanced cross-lingual knowledge alignment, with high consistency but low conductivity between languages. Multilingual pretraining and instruction tuning can improve basic language abilities and cross-lingual performance and consistency, but fail to substantially enhance the cross-lingual knowledge conductivity.

Abstract

The paper systematically evaluates the cross-lingual knowledge alignment of popular multilingual language models using the proposed CLiKA framework. The key findings are:

Basic language abilities: Multilingual language models show imbalanced basic language abilities across different languages, with better performance in Indo-European languages and poorer abilities in non-Latin languages like Chinese, Arabic and Hebrew.

Factual knowledge alignment:

Performance (PF) level: Models show imbalanced performance on factual knowledge tasks across languages, with much lower scores in non-English languages like Chinese.
Consistency (CT) level: Models exhibit high cross-lingual consistency, with their correct answers in non-English languages highly overlapping with their English answers.
Conductivity (CD) level: However, the cross-lingual knowledge conductivity, i.e. the ability to retrieve knowledge learned in one language using another language, is unsatisfactory for all tested models.

Effect of multilingual pretraining:

Mixed multilingual pretraining improves basic abilities, PF and CT alignment across languages, while continued pretraining in a target language only benefits that language at the cost of others.
Neither mixed nor continued pretraining can substantially improve the cross-lingual knowledge conductivity.

Effect of multilingual instruction tuning:

Multilingual instruction tuning improves basic abilities in the target language, and mitigates the performance drop in factual knowledge caused by instruction tuning.
However, it fails to significantly enhance the CT and CD alignment.
In summary, current multilingual language models exhibit shallow cross-lingual knowledge alignment, and existing techniques like multilingual pretraining and instruction tuning are limited in improving the deeper levels of alignment, especially the cross-lingual knowledge conductivity.

Stats

The dental office handled a lot of patients who experienced traumatic mouth injury, where were these patients coming from? C. hospital
The item was packaged in bubble wrap. What was the cause of this? A. It was fragile.
What administrative division of Egypt is Alexandria in? B. Alexandria Governorate
In what year was Houari Boumediene born? D. 1932

Quotes

"Despite their strong ability to retrieve knowl-edge in English, current large language models show imbalance abilities in different languages."
"Results show that: while both multilingual pretraining and instruction tuning are beneficial for cross-lingual knowledge alignment, the training strategy needs to be carefully designed."
"Also, the overall cross-lingual knowledge alignment, especially in the conductivity level, is unsatisfactory for all tested LLMs, and neither multilingual pretraining nor instruction tuning can substantially improve the cross-lingual knowledge conductivity."

Key Insights Distilled From

Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly

by Changjiang G... at arxiv.org 04-09-2024

https://arxiv.org/pdf/2404.04659.pdf

Deeper Inquiries

How can we design more effective pretraining and finetuning strategies to improve the cross-lingual knowledge conductivity of multilingual language models?

To enhance the cross-lingual knowledge conductivity of multilingual language models, we can implement the following strategies:

Diversified Multilingual Pretraining: Instead of relying solely on parallel data, incorporating diverse monolingual data in the pretraining corpus can help models learn a broader range of linguistic patterns and improve their cross-lingual alignment. This approach can expose models to a wider variety of language structures and nuances, enhancing their ability to transfer knowledge across languages.

Bidirectional Translation Data: Introducing bidirectional translation data, where entities and relations are translated not only from English to other languages but also vice versa, can facilitate better knowledge transfer between languages. This approach can help address the unidirectional bias in current models and improve their cross-lingual knowledge alignment.

Fine-tuning with Multilingual Instruction: Leveraging multilingual instruction tuning, where models are fine-tuned on tasks in different languages or translation-related tasks, can further enhance their cross-lingual knowledge alignment. By exposing models to diverse linguistic contexts during fine-tuning, they can better adapt to different languages and improve their knowledge conductivity.

Attention Mechanism Optimization: Optimizing the attention mechanisms in language models, such as through techniques like LoRA (Low-Rank Adaptation), can help models focus on relevant information during training and inference. By fine-tuning attention mechanisms to prioritize cross-lingual knowledge alignment, models can improve their ability to conduct knowledge across languages effectively.

How can we design more effective pretraining and finetuning strategies to improve the cross-lingual knowledge conductivity of multilingual language models?

The fundamental limitations of current language models that hinder deep cross-lingual knowledge alignment include:

Language Bias: Models trained predominantly on English data may exhibit a bias towards English, leading to imbalanced performance across languages. This bias can limit the models' ability to effectively transfer knowledge between languages and achieve deep cross-lingual alignment.

Linguistic Differences: Variations in linguistic structures, syntax, and semantics across languages pose challenges for models in conducting knowledge effectively. Differences in language families, writing systems, and word order can impact the models' ability to align knowledge accurately.

To overcome these limitations and achieve deeper cross-lingual knowledge alignment, we can:

Enhance Multilingual Training Data: By incorporating a more diverse and balanced set of languages in the training data, models can learn a wider range of linguistic patterns and improve their cross-lingual alignment. This approach can help mitigate language bias and enhance the models' ability to transfer knowledge across languages effectively.

Optimize Model Architectures: Developing model architectures that are more adaptable to diverse linguistic structures and can capture cross-lingual relationships effectively can improve the models' ability to conduct knowledge across languages. Techniques like multi-head attention and cross-lingual parameter sharing can enhance the models' cross-lingual alignment capabilities.

Continuous Evaluation and Improvement: Regularly assessing the models' cross-lingual knowledge alignment and iteratively refining pretraining and finetuning strategies based on the evaluation results can help address limitations and enhance the models' performance in real-world multilingual applications.

Given the findings on the shallow cross-lingual alignment, how can we leverage the strengths of multilingual models while mitigating their weaknesses in real-world multilingual applications?

To leverage the strengths of multilingual models while mitigating their weaknesses in real-world multilingual applications, we can adopt the following strategies:

Task-Specific Fine-Tuning: Tailoring the fine-tuning process to specific multilingual tasks and languages can help optimize the models for real-world applications. By fine-tuning models on task-relevant data in different languages, we can enhance their performance and alignment for specific use cases.

Domain Adaptation: Incorporating domain-specific data during pretraining and fine-tuning can improve the models' performance in specialized domains and enhance their cross-lingual knowledge alignment. By exposing models to domain-specific language patterns, they can better adapt to diverse contexts and improve their effectiveness in real-world applications.

Ensemble Approaches: Combining multiple multilingual models with complementary strengths and weaknesses through ensemble methods can enhance overall performance and alignment in real-world multilingual applications. By leveraging the diversity of ensemble models, we can mitigate individual model limitations and improve overall effectiveness across languages and tasks.