Multidimensional Evaluation of Search System Explainability Using Psychometrics and Crowdsourcing

Core Concepts

Explainability of search systems is a multidimensional concept comprising both positive utility factors and negative roadblock factors, as identified through a psychometric study leveraging crowdsourcing.

Abstract

The study aimed to establish a user-centric definition of search system explainability by leveraging psychometrics and crowdsourcing. The researchers conducted a comprehensive literature review to identify 26 potential aspects of explainability, which were then used to design a questionnaire.
Through exploratory factor analysis (EFA) on a sample of 200 crowdsourced responses, the researchers identified two key factors underlying explainability:

Utility: This factor encompasses positive attributes such as plausibility, justifiability, trustworthiness, informativeness, acceptability, understandability, and transferability. These aspects represent the overall usefulness and effectiveness of the explainable search system.

Roadblocks: This factor includes negative attributes such as lack of decomposability, global and local interpretability, simulatability, faithfulness, algorithmic transparency, trustworthiness, causality, uncertainty, units of explanation, visibility, and counterfactuals. These aspects represent critical barriers that prevent users from fully understanding the search system's decision-making process.

The researchers then confirmed this two-factor model through confirmatory factor analysis (CFA) on a held-out set of 259 crowdsourced responses. The CFA results showed that the proposed hierarchical two-factor model had a good fit to the data, outperforming alternative models.
The identified dimensions of explainability can be used to guide the design and evaluation of explainable search systems, enabling targeted improvements to address both positive utility and negative roadblock factors. The methodology introduced in this work can also be applied to other IR domains and the wider NLP and ML communities.

Stats

The search system should work well in different search tasks.
I would use this search engine in my everyday life.
The results page provides me enough information to find the answers I am looking for effectively.
The presentation of the results leads me to believe the results are ordered correctly.
The results match my expectations and I agree with them.
I trust that the results are ordered correctly and the system will order results correctly for other queries.
I can easily understand the contents of the results page.
If I change the query, I do not know how it will affect the result ordering.
I do not understand why the results are ordered the way they are and would not be able to recreate the orderings myself.
I think I need more information to understand why the given query produced the displayed results.
I do not understand the document properties that cause some results to be ordered higher than others.
I am unable to see and understand how changes in the query affect the result ordering.
The result interface does not help me understand the true decision making process of the search engine ranker.
I do not understand why each result is ordered in a certain place.
I'm unable to follow how the search engine ordered the results.
It's difficult for me to break down each of the search engine's components and understand why the results are ordered the way they are.
I do not know how confident the search engine is that its displayed orderings are correct.
I do not trust that the results are ordered correctly and that the system will correctly order results for other queries.
The format and amount of information provided in the result interface is not enough to help me understand why the results are ordered the way they are.

Quotes

If I change the query, I do not know how it will affect the result ordering.
I do not understand why the results are ordered the way they are and would not be able to recreate the orderings myself.
The result interface does not help me understand the true decision making process of the search engine ranker.

Key Insights Distilled From

Evaluating Search System Explainability with Psychometrics and Crowdsourcing

by Catherine Ch... at arxiv.org 05-07-2024

https://arxiv.org/pdf/2210.09430.pdf

Deeper Inquiries

How can the identified dimensions of explainability be used to guide the design of novel explainable search interfaces that balance utility and address critical roadblocks?

The identified dimensions of explainability, namely utility and critical roadblocks, can serve as a foundational framework for designing novel explainable search interfaces. By understanding these dimensions, designers can create interfaces that not only provide valuable information to users but also mitigate potential obstacles that hinder the user experience.
To balance utility, designers can focus on aspects such as plausibility, justifiability, trustworthiness, informativeness, acceptability, understandability, and transferability. These dimensions emphasize the importance of providing accurate, trustworthy, and relevant information to users in a way that is easily understandable and acceptable. By incorporating features that enhance these aspects, search interfaces can ensure that users find the system useful and reliable for their information needs.
On the other hand, addressing critical roadblocks involves tackling factors like decomposability, global interpretability, local interpretability, simulatability, faithfulness, algorithmic transparency, causality, uncertainty, units of explanation, visibility, and counterfactuals. These dimensions highlight the challenges users may face in understanding the decision-making processes of the search system and the transparency of its operations. Designers can work on making the system more interpretable, transparent, and accountable by providing explanations that clarify how results are generated and ordered.
By incorporating features that enhance both utility and critical roadblocks, designers can create a balanced and effective explainable search interface. This approach ensures that users not only receive valuable information but also understand how the system operates, leading to a more trustworthy and user-friendly search experience.

How might the explainability dimensions differ for search systems in specialized domains (e.g., medical, legal) compared to general web search, and what are the implications for system design and evaluation?

Explainability dimensions for search systems in specialized domains such as medical or legal contexts may differ from those in general web search due to the unique requirements and complexities of these domains. In specialized domains, users often have specific information needs that require a higher level of accuracy, reliability, and domain expertise. As a result, the dimensions of explainability may need to be tailored to address these specialized requirements.
In medical search systems, factors like accuracy, completeness, trustworthiness, and causality may take precedence due to the critical nature of medical information. Users in this domain require precise and reliable information to make informed decisions about healthcare. Additionally, factors like model fairness and ethical considerations become crucial in ensuring that the search results are unbiased and aligned with medical standards and guidelines.
In legal search systems, factors such as global interpretability, local interpretability, visibility, and counterfactuals may be more prominent. Legal professionals need to understand the reasoning behind search results, the impact of different queries on the outcomes, and the transparency of the system's decision-making process. Ensuring that the system provides clear and interpretable explanations can enhance user trust and confidence in the search results.
The implications for system design and evaluation in specialized domains involve tailoring the explainability dimensions to meet the specific needs of users in those domains. Designers must consider the unique challenges and requirements of medical and legal professionals when designing explainable search interfaces. Evaluation methods should focus on assessing the system's ability to provide accurate, transparent, and trustworthy information relevant to the specialized domain, ensuring that the system meets the high standards of these professional users.