WangchanLion and WangchanX MRC Evaluation Report

Kernekoncepter

Development of WangchanLion for Machine Reading Comprehension in Thai language, focusing on contextual understanding and evaluation.

Resumé

This technical report discusses the development of WangchanLion, a Thai instruction fine-tuned model for Machine Reading Comprehension (MRC). It includes details on the model's training data, evaluation methodology, and comparison with other models. The report also introduces a new evaluation scheme assessing correctness, helpfulness, conciseness, and contextuality.
Abstract:

Development of WangchanLion for MRC in Thai language.
Public release of training data, code, and model weights under Apache-2 license.
Experimental studies using XQuAD and Iapp_wiki_qa_squad datasets.
Proposal of a new evaluation scheme for MRC.
Introduction:

Significance of Large Language Models (LLMs) in AI.
Open-source research interest in LLMs.
Introduction to SEA-LION and other models supporting the Thai language.
Instruction Tuning:

Data sources used for instruction tuning.
Supervised Fine-tuning (SFT) strategy employed.
Hyperparameter settings for fine-tuning WangchanLion.
Machine Reading Comprehension (MRC) Evaluation:

Components of MRC evaluation: context, question, reference answer, response.
Traditional extractive QA evaluation using XQuAD dataset.
Human evaluation method design and results comparison.

Statistik

SEA-LIONは10,652トークンのタイ語ボキャブラリーサイズを持つ。
LLaMA2のトレーニングデータセットは約2.0兆トークンで、そのうち89%以上が英語である。

Citater

"Large Language Models have gained significant attention in recent years." - Wannaphong Phatthiyaphaibun

Vigtigste indsigter udtrukket fra

WangchanLion and WangchanX MRC Eval

by Wannaphong P... kl. arxiv.org 03-26-2024

https://arxiv.org/pdf/2403.16127.pdf

Dybere Forespørgsler

How can the transferability of knowledge from rich-resource languages to low-resource languages be improved?

To enhance the transferability of knowledge from rich-resource languages to low-resource languages, several strategies can be implemented. One approach is to focus on cross-lingual training and fine-tuning techniques that leverage multilingual datasets and models. By incorporating diverse linguistic data during pre-training, models can better capture language patterns across different language families. Additionally, utilizing parallel corpora for translation tasks can facilitate alignment between languages, enabling effective knowledge transfer.

What are the implications of limited vocabulary size on the performance of open-source LLMs?

The limited vocabulary size in open-source Large Language Models (LLMs) poses significant challenges to their performance in low-to-medium resource languages. With a constrained set of tokens, these models may struggle to accurately represent the semantic nuances and complexities present in such languages. This limitation could lead to difficulties in capturing context-specific information and generating coherent responses. Moreover, it may hinder the model's ability to generalize well across various linguistic contexts, impacting its overall effectiveness in understanding and processing text data.

How can automated evaluations like GPT assessments impact future research on language models?

Automated evaluations using tools like GPT assessments offer a scalable and efficient means of evaluating language models' performance across various tasks. By leveraging large pre-trained models for assessment purposes, researchers can quickly analyze model outputs at scale without extensive human intervention. This automation streamlines evaluation processes, allowing for rapid feedback loops and iterative improvements in model development. Furthermore, automated evaluations provide insights into model capabilities beyond traditional metrics, fostering advancements in natural language understanding and generation technologies.