Główne pojęcia
Retrieval-augmented language models (RAG) can reduce generation risks with certified guarantees.
Streszczenie
Abstract:
- Large language models (LLMs) face trustworthiness issues.
- Retrieval-augmented language models (RAG) aim to enhance credibility.
- C-RAG framework certifies generation risks for RAG models.
Introduction:
- LLMs exhibit emergent abilities but suffer from unreliable generations.
- RAG models aim to mitigate generation risks through in-context learning.
- Theoretical analysis of generation risks for RAG models is explored.
Conformal Generation Risks of RAG Models:
- Constrained generation protocol for RAG models proposed.
- Conformal risk analysis for RAG models under the protocol.
- Certifies generation risks based on test statistics from calibration samples.
Theoretical Analysis of C-RAG:
- RAG achieves lower conformal generation risk compared to LLMs.
- Benefits correlated with quality of retrieval model and transformer.
- Theoretical results presented in a certification framework.
Out-of-Distribution Test Samples:
- Conformal risk analysis under distribution shifts for general bounded risk functions.
Evaluation:
- C-RAG evaluated on four NLP datasets with different retrieval models.
- Conformal generation risks validated and compared across models.
- Soundness and tightness of conformal generation risks under distribution shifts demonstrated.
Statystyki
RAG achieves a lower conformal generation risk than a single LLM when the quality of the retrieval model and transformer is non-trivial.