핵심 개념
Constructing a Japanese financial-specific large language model through continual pre-training on domain-focused datasets, which outperforms the original model on Japanese financial benchmarks.
초록
The study aims to construct a Japanese financial-specific large language model (LLM) through continual pre-training. The authors first built Japanese financial-focused datasets containing around 8.1 million documents and 370 million tokens, covering various financial documents such as speeches, reports, and company profiles. They then employed a state-of-the-art Japanese LLM, rinna/nekomata-14b, as the base model and performed continual pre-training using the constructed datasets.
The authors evaluated the tuned model using Japanese financial benchmarks and by comparing the output quality with the original model. The results show that the tuned model outperformed the original model on all the benchmark tasks, indicating that the domain-specific continual pre-training was effective. The output comparison also revealed that the tuned model's outputs tend to be better than the original model's in terms of quality and informativeness, though the tuned model still had issues answering some financial domain-specific questions correctly.
The authors discuss that the domain-specific tuning is effective for LLMs, but the scope for future research includes instruction tuning, expanding the financial dataset coverage, and evaluating the domain-specific tuning for 100-billion-class-parameter models.
통계
The Japanese financial-focused datasets contain around 8.1 million documents and 370 million tokens.
The tuned model achieved better performance than the original model on all the Japanese financial benchmark tasks.
The overall score of the tuned model is 0.4716, which is 0.0381 higher than the original model's score of 0.4335.
인용구
"The tuned model's outputs tend to be better than the original model's outputs in terms of the quality and length of the answers."
"However, the tuned model still has issues to answer correctly for some questions."