Belangrijkste concepten
Providing counterfactual queries as explanations to help users understand and interact with search engine relevance decisions.
Samenvatting
This paper proposes a method called CounterFactual Editing for Search Result Explanation (CFE2) to generate counterfactual queries as explanations for search result relevance. The key insights are:
Counterfactual explanations, which explain an observed event using some counterfactual events, can be more effective than factual explanations in reducing cognitive load and providing actionable insights for users.
The authors formulate the problem of generating counterfactual queries as explanations for pairwise relevance relations within a search engine result page. The goal is to find a counterfactual query that would rank a lower-ranked document higher than the initially higher-ranked document.
The authors propose desiderata for counterfactual explanations in the search context, including effectiveness in flipping the ranking order, closeness to the initial query, fluency of the counterfactual query, and low latency. They design corresponding automatic evaluation metrics.
The CFE2 method iteratively edits the initial query by masking important tokens and using a language model to predict replacement tokens, until a counterfactual query is found that can flip the ranking order. Experiments on multiple public search datasets show CFE2 outperforms baselines in both automatic and human evaluations.
CFE2 has additional strengths, including being model-agnostic, generating counterfactuals with minimal modifications, using off-the-shelf language models, and being computationally lightweight.
Statistieken
The initial query and document pair (q, d) have a higher relevance score than the initial query and counterfactual document pair (q, d').
The generated counterfactual query q' has a higher relevance score with the counterfactual document d' than with the initially higher-ranked document d.