Core Concepts
Generative retrieval can be understood as a special case of multi-vector dense retrieval, where both methods compute relevance as a sum of products of query and document vectors and an alignment matrix.
Abstract
The paper examines the connection between generative retrieval (GR) and multi-vector dense retrieval (MVDR) models. It shows that GR and MVDR share the same framework for measuring the relevance of a document to a query.
Key highlights:
- The logits in the loss function of GR can be reformulated to a product of document word embeddings, query token vectors, and an attention matrix, corresponding to the unified MVDR framework.
- GR employs distinct strategies for document encoding and the alignment matrix compared to MVDR. Specifically, GR uses simple document embeddings, which can be improved using techniques like prefix-aware weight-adaptive (PAWA) decoding and non-parametric (NP) decoding.
- The alignment matrix in GR is dense and learned, while MVDR typically uses a sparse alignment matrix computed using heuristic algorithms. GR also exhibits document-to-query alignment, in contrast to the query-to-document alignment in MVDR.
- Both GR and MVDR alignment matrices exhibit a low-rank property and can be decomposed into query and document components.
The paper provides a theoretical foundation for understanding the underlying mechanisms of GR and its connection to the state-of-the-art MVDR models, which can lead to further improvements in retrieval models.