Khái niệm cốt lõi
Retrieval-Augmented Language Models (RALMs) integrate external information retrieval with large language models to enhance performance across a wide range of NLP tasks, addressing challenges such as hallucination and lack of domain-specific knowledge.
Tóm tắt
This survey provides a comprehensive overview of Retrieval-Augmented Language Models (RALMs) in Natural Language Processing (NLP). It covers the key components of RALMs, including Retrievers and Language Models, and examines the different interaction modes between these components, such as Sequential Single Interaction, Sequential Multiple Interactions, and Parallel Interaction.
The paper classifies and summarizes the various retrieval methods used in RALMs, including Sparse Retrieval, Dense Retrieval, Internet Retrieval, and Hybrid Retrieval. It also discusses the different types of language models employed, including AutoEncoder Language Models, AutoRegressive Language Models, and Encoder-Decoder Language Models.
The survey further explores the enhancements made to RALMs, such as Retriever Enhancement (Retrieval Quality Control and Retrieval Timing Optimization), LM Enhancement (Pre-Generation Retrieval Processing, Structural Model Optimization, and Post-Generation Output Enhancement), and Overall Enhancement. It also covers the sources of retrieved data and the applications of RALMs across various domains.
The evaluation methods and benchmarks used to assess the performance of RALMs are discussed, emphasizing the importance of robustness, accuracy, and relevance. Finally, the survey acknowledges the limitations of existing RALMs, particularly in retrieval quality and computational efficiency, and provides recommendations for future research directions.
Thống kê
"Large Language Models (LLMs) have catalyzed significant advancements in Natural Language Processing (NLP), yet they encounter challenges such as hallucination and the need for domain-specific knowledge."
"Recent methodologies have integrated information retrieved from external resources with LLMs, substantially enhancing their performance across NLP tasks."
Trích dẫn
"Retrieval-Augmented Language Model (RALM) is the process of refining the output of the LM with retrieved information to obtain a satisfactory result for the user."
"The sequential single interaction process involves finding the Top-K relevant documents z to input x through a retriever Pη(z|x), where η is a parameter of the retriever. Subsequently, the language model Pθ(yi|x, z, yr) receives input x along with relevant documents z and outputs the i-th token yi."
"In the parallel structure, the retriever and the language model work independently for the user input x. The output y is then determined by weighted interpolation."