Majority Voting of Pathologists Improves Appropriateness of AI Reliance in Mitosis Detection

Core Concepts

Majority voting of pathologists' AI-assisted decisions can significantly improve the appropriateness of AI reliance, leading to better precision and recall in the detection of mitoses compared to one pathologist collaborating with AI.

Abstract

The study examines how majority voting of pathologists' decisions can enable appropriate AI reliance in the high-stakes medical task of detecting mitoses in tumor images. Key highlights: 32 pathology professionals from 10 institutions participated in a multi-stage user study, first detecting mitoses manually and then with AI assistance. Majority voting decisions from as few as 3 pathologists showed significantly higher relative AI reliance (9% increase) and relative self-reliance (31% increase) compared to one pathologist collaborating with AI. Majority voting decisions also achieved better precision (up to 9% increase) and recall (up to 4% increase) in mitosis detection compared to one pathologist-AI collaboration. Majority voting was more likely to achieve complementary team performance (i.e., outperforming both human and AI) in recall compared to one pathologist-AI collaboration. While the utilization of the provided XAI features was relatively low, pathologists frequently activated the AI recommendations, with 21/29 participants having over 50% AI active time. This work suggests that harnessing group decision-making of AI-assisted medical professionals can be an effective strategy to enable appropriate AI reliance in high-stakes medical tasks.

Stats

The mean AI activation rate was 99.21%. The mean AI active time percentage was 71.39%. The mean XAI activation rate was 14.54%. The mean XAI activation time was 4.31 seconds.

Quotes

"Majority voting decisions from as few as 3 pathologists showed significantly higher relative AI reliance (9% increase) and relative self-reliance (31% increase) compared to one pathologist collaborating with AI." "Majority voting decisions also achieved better precision (up to 9% increase) and recall (up to 4% increase) in mitosis detection compared to one pathologist-AI collaboration." "Majority voting was more likely to achieve complementary team performance (i.e., outperforming both human and AI) in recall compared to one pathologist-AI collaboration."

Key Insights Distilled From

Majority Voting of Doctors Improves Appropriateness of AI Reliance in Pathology

by Hongyan Gu. ... at arxiv.org 04-09-2024

https://arxiv.org/pdf/2404.04485.pdf

Deeper Inquiries

How can the majority voting approach be extended to other high-stakes visual search tasks beyond pathology, such as detecting explosives from X-ray scans or assessing disaster damage from satellite imagery?

The majority voting approach can be extended to other high-stakes visual search tasks by adapting the methodology to suit the specific requirements of each task. For tasks like detecting explosives from X-ray scans or assessing disaster damage from satellite imagery, a similar multi-institutional user study involving professionals with relevant backgrounds can be conducted. The key lies in synthesizing decisions under AI assistance from a group of experts in the respective fields. By aggregating the decisions of multiple experts, the majority voting process can help in enhancing the accuracy and reliability of AI recommendations in these critical tasks. Additionally, the use of explainable AI (XAI) can provide insights into the decision-making process, further improving the transparency and trustworthiness of the majority voting results.

What are the potential drawbacks or limitations of the majority voting approach, and how can they be addressed to ensure its widespread adoption in medical decision-making?

One potential drawback of the majority voting approach is the increased complexity and time required to gather and synthesize decisions from multiple experts. This can lead to challenges in coordinating and aligning the opinions of diverse professionals. Additionally, there may be instances where conflicts arise among the experts, leading to delays or disagreements in the decision-making process. To address these limitations, clear guidelines and protocols should be established for conducting the majority voting process. Training sessions can be provided to ensure that participants understand the methodology and their roles in the decision-making process. Furthermore, the use of advanced AI algorithms to assist in aggregating and analyzing the decisions can streamline the process and improve the efficiency of the majority voting approach.

Given the relatively low utilization of the provided XAI features, how can the design of XAI be improved to better support pathologists in understanding and appropriately relying on AI recommendations?

To improve the utilization of XAI features and better support pathologists in understanding and relying on AI recommendations, several design enhancements can be implemented. Firstly, the XAI interface should be user-friendly and intuitive, providing clear and concise explanations of AI recommendations. Interactive elements such as tooltips, interactive visualizations, and contextual help can enhance the user experience and facilitate the interpretation of AI outputs. Additionally, personalized XAI settings tailored to individual preferences and expertise levels can increase engagement and relevance for pathologists. Incorporating real-time feedback mechanisms and incorporating XAI explanations directly into the workflow can also encourage pathologists to actively utilize and trust the XAI features. Continuous user feedback and iterative design improvements based on user interactions and preferences can further refine the XAI design to meet the specific needs of pathologists in medical decision-making.