Conceitos essenciais
JDocQA is a large-scale Japanese document question answering dataset that requires understanding of both textual and visual information to answer questions.
Resumo
The JDocQA dataset was created by collecting 5,504 Japanese documents in various formats (pamphlets, slides, reports, websites) and annotating 11,600 question-answer pairs on them. The questions cover four categories: yes/no, factoid, numerical, and open-ended. The dataset also includes 1,000 unanswerable questions where the correct answer is not mentioned in the given documents.
The key highlights of the dataset are:
- Multimodal nature: Questions require understanding of both textual and visual elements (figures, tables, charts) in the documents.
- Diverse question types: The dataset includes yes/no, factoid, numerical, and open-ended questions.
- Unanswerable questions: 1,000 questions have no answer in the given documents, testing the model's ability to detect unanswerable cases.
- Multilingual: The dataset is in Japanese, addressing the lack of non-English document question answering resources.
The authors conducted experiments with both text-only and multimodal models, evaluating their performance on the JDocQA dataset. They found that incorporating unanswerable questions during finetuning can help mitigate hallucination in language model outputs.
Estatísticas
The dataset contains 5,504 documents and 11,600 question-answer pairs.
The documents include 1,715 pamphlets, 1,640 slides, 2,086 reports, and 67 websites.
The question types are: 1,855 yes/no, 2,052 factoid, 1,866 numerical, and 5,827 open-ended.
1,788 questions require referencing multiple pages, and 1,000 questions are unanswerable.
Citações
"Incorporating unanswerable questions in finetuning may contribute to harnessing the so-called hallucination generation."
"JDocQA consists of 11,600 question and answer pairs on the collected 5,504 documents as references for answering the question, four different question categories and 1,000 multi-page questions."