QASE Enhanced PLMs: Improved Control in Text Generation for MRC

핵심 개념

Generative PLMs can be enhanced with the lightweight QASE module to improve text generation quality and factual consistency in MRC tasks.

초록

研究では、Question-Attended Span Extraction（QASE）モジュールを使用して、生成的PLMを改善し、MRCタスクでのテキスト生成の品質と事実の整合性を向上させることが示されています。QASEにより、複数のパッセージにまたがる回答や暗黙的な回答など、複雑なシナリオでも正確な回答が生成されます。また、QASEは実世界の知識を活用する能力も向上させます。

통계

QASE-enhanced model outperforms vanilla fine-tuned models on SQuAD, MultiSpanQA, and Quoref datasets.
Flan-T5-LargeQASE surpasses GPT-4 by a significant margin on all three MRC datasets.
Q2 scores show that QASE-enhanced models consistently outperform the vanilla fine-tuned model in terms of factual consistency.

인용구

핵심 통찰 요약

QASE Enhanced PLMs

by Lin Ai,Zheng... 게시일 arxiv.org 03-11-2024

https://arxiv.org/pdf/2403.04771.pdf

더 깊은 질문

どのようにしてQASEモジュールは生成的PLMの性能を向上させるのですか？

QASEモジュールは、質問に関連する回答スパンを特定するために使用されます。このモジュールは、文脈内で潜在的な回答スパンに対するモデルの注意を集中させることで、生成的PLMが適切な回答を生成する際に役立ちます。具体的には、マルチヘッドアテンションメカニズムやシーケンスタグ付けロスなどが組み込まれており、質問とコンテキスト間の関係性を学習し、正確な回答スパンを予測します。これにより、不完全または冗長なフレーズから成る誤った回答や事実と一致しない情報から desviating した生成された回答への対処が可能となります。

この研究結果は、他のNLPタスクへの応用可能性がありますか？

この研究結果は非常に有望であり、他のNLPタスクへも応用可能性があります。例えば、「文章理解」以外でも「文章生成」「機械翻訳」「会話応答生成」など幅広い分野で利用できる可能性が考えられます。QASEモジュールを導入してプリトレーニング済み言語モデル（PLMs）を微調整することで品質や事実内容合意度が向上しました。そのため、「文脈指向型テキスト生成」という新しいNLPタスクへ展開すれば効果的だろうと考えられます。

生成的PLMと抽出型モデルとの比較から得られた洞察は何ですか？

本研究では、抽出型方法では難しかった多くのMRCタスクでも優れた成果を収めました。「Question-Attended Span Extraction (QASE)」 モジュール導入後 fine-tuned PLMs の品質や事実内容合意度が改善され、「SOTA extractive methods」とGPT-4 を超越しました。
これからも引き続き QASE ベースアプローチ を採用すれば 抽出型手法以上 の 成果 も期待されること示唆しています。
Generative models, however, underperform in MRC due to out-of-control generation (Li et al.,2021). This leads to two main challenges: (1) ill-formed generated answers, containing incomplete or redundant phrases, and (2) factual inconsistency in the generated answers deviating from the correct response. In this paper, we address these by introducing a lightweight Question-Attended Span Extraction(QASE) module. We fine-tune multiple open-source generative pre-trained language models(PLMs) on various MRC datasets to assess the module's efficacy in guiding answer generation.
Our contributions include: Developing QASE to improve fine-tuned generative PLMs' quality and factual consistency on MRC tasks matching SOTA extractive methods and surpassing GPT-4; Qase boosts performance without significantly increasing computational costs benefiting researchers with limited resources.
Most current studies on MRC involve predicting the start and end positions of the answer spans from a given context(Ohsugi et al.,2019; Lan et al.,2019; Bachina et al.,2021; Chen et al.,2022 using encoder-only PLM models such as BERT and XLM-Roberta. To handle multi-span setting some studies frame problem as sequence tagging task(Segal etal.,2020),and others explore ways combine models with different tasks(Huetal.,2019; Leeet.al . ,2023 ; Zhanget.al . ,20
While these extractive-based methods mainly utilize encoder-only models there is also research focuses on using generative language model(Yangetal . ,20 ; Lieta l . ,20 ; Sueta l . ).Retrieval-augmented text generation(RAG)a ugments input of PLMs within-domain(Guetal . ,20 ; Westonetal .
Ablation Studies Details
Figure depicts architecture model use ablation studies baseline span extraction module The baseline span extraction module omits component typifying standard architecture finetuning pretrained encoders downstream sequence tagging tasks The baselineembedded FlanT5Large models finetuned configurations FlanT5LargeQAS including learning rate weight decay batch size epoch number GPU type experiment prompting strategies ablation studies Contextfirst prompting default prompting strategy utilize finetuning PLMS both QAS In setting prompt ordered instruction tokens context tokens question tokens Questionfirst prompting qf Following BERTs standard finetuning procedures In setting prompt ordered instruction tokens question tokens SEP context tokens SEP special separator token
Factual Consistency Case Studies
Section demonstrate FlanTLarge model finetuned QAS produces answers greater factual accuracy relation context compared counterpart finedtuned without QAS Specifically observe improvement score SQuAD dataset significant increase MultiSpanQA section includes examples illustrate effectiveness Table showcases FlanTLargeQAS accurately identifies key focus question locates pertinent factual information within context aid needs recognize ESPN Deportes exclusive broadcaster Spanish CBS although mentioned offer Spanishlanguage broadcasting Combining facts leads correct answer ESPN Deportes network broadcast game SpanishFlanTLargeQAS accurately generates answer whereas FlanTLargeFT incorrectly answers CBS likely confusion caused complex sentence structures dispersed information Similarly Sample correctly identifies seeking name force related potential field two locations successfully locates relevant long sentence deconstructs comprehends produce correct contrast incorrectly selects phrase mentioning force Sample asks class commonly not ascribed graph isomorphism problem model deduce implies graph isomorphism NPcomplete once arrives conclusion while does primary evaluation focuses proficiency deriving provided contexts note enhances leverage realworld knowledge acquired pretraining phase presents example phenomenon asked California venue considered Super Bowl correctly associates San Francisco Bay Area California producing accurate other hand erroneously identifies stadium Miami example illustrates improves contextbased application preexisting realworld knowledge questions posed