The authors conduct an extensive study on the effectiveness of Low-Rank Adaptation (LoRA) for multilingual summarization across different data availability scenarios, including high-data, low-data, and cross-lingual transfer settings.
In the high-data regime, the authors find that full fine-tuning achieves the best ROUGE-L scores, but LoRA exhibits superior performance in terms of summary faithfulness and conciseness. As the amount of training data decreases, LoRA becomes a better alternative to full fine-tuning, delivering competitive or even superior results while being more computationally efficient.
In the cross-lingual transfer setting, the authors explore zero-shot and few-shot learning. They observe that LoRA consistently outperforms full fine-tuning, especially when only a small number of target language examples are available. The authors also investigate different strategies for few-shot cross-lingual transfer, finding that continued LoRA tuning outperforms full fine-tuning and the dynamic composition of language-specific LoRA modules.
When scaling up to the larger PaLM 2-S model, LoRA achieves on-par performance with full fine-tuning, making it a better choice due to its computational efficiency.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Chenxi White... at arxiv.org 04-02-2024
https://arxiv.org/pdf/2311.08572.pdfDeeper Inquiries