Core Concepts
Long-document ranking models may underperform due to positional bias, impacting zero-shot and fine-tuning scenarios.
Abstract
The content delves into evaluating Transformer models for ranking long documents, comparing them with FirstP baselines. It explores the impact of positional bias on model performance and introduces a new collection, MS MARCO FarRelevant, to address bias issues. The experiments reveal differences among models, highlighting the challenges of processing longer document contexts and the implications of positional bias on model performance.
Abstract
Evaluated 20+ Transformer models for long-document ranking.
Compared models with FirstP baselines and identified positional bias.
Introduced MS MARCO FarRelevant collection to address bias issues.
Introduction
Transformer models advanced NLP and IR.
Chunk-and-aggregate approaches proposed for long documents.
Evaluation of models lacking systematic comparison with FirstP baselines.
Methods
Neural ranking models overview.
Description of long-document BERT models.
SplitP and LongP models for processing long documents.
Experiments
Evaluation on MS MARCO, TREC DL, and Robust04 datasets.
Results show marginal improvements over FirstP baselines.
Impact of positional bias on model performance.
Stats
We evaluated 20+ Transformer models for ranking long documents.
LongP models showed little to no improvement compared to FirstP baselines.
PARADE Attn model achieved the biggest gain of 5% over FirstP.
Models were at least 2x slower than respective FirstP baselines.
Quotes
"We found evidence of a substantial positional bias in relevant passages."
"LongP models showed little to no improvement compared to their respective FirstP baselines."
"PARADE Attn model achieved the biggest average gain of 5% over FirstP baselines."