Leveraging Inter-Observer Eye-Gaze Consistencies to Support Mitosis Detection AI Training in Pathology

Core Concepts

This study investigates using eye-gaze data from non-medical participants as a cost-effective approach to support the training of convolutional neural networks (CNNs) for mitosis detection in pathology images, focusing on the task of detecting mitoses in meningioma specimens.

Abstract

The study consists of three main parts:

Collection of meningioma pathology images and ground truth annotations:

1,000 high-power field (HPF) images were collected, including 500 positive (with mitoses) and 500 negative (without mitoses) samples.
A test whole slide image (WSI) containing 1,298 non-background HPFs and 380 mitoses was also selected.
Ground truth annotations were provided by pathology professionals.

Eye-tracking user study:

14 non-medical participants were recruited and trained to detect mitoses in the HPF images.
Participants' eye-gaze data were collected while they viewed the 800 HPF images.
The eye-gaze data were processed to extract consistent fixation areas shared among participant groups, which were used as pseudo-labels.

CNN training and evaluation:

Three sets of labels were used to train EfficientNet-b3 CNNs: heuristic-based, eye-gaze-based, and ground truth.
The CNNs were evaluated on the test WSI, and their performance was compared in terms of precision, recall, and F1 score.

The results showed that the eye-gaze-based CNNs closely followed the performance of ground-truth-based CNNs and significantly outperformed the heuristic-based approach. This suggests that eye-gaze data can provide meaningful information to support the training of pathology AI models, potentially addressing the challenge of acquiring high-quality annotations from medical professionals.

Stats

The test WSI contained 1,298 non-background HPFs and 380 mitoses.
The 14 participants spent an average of 2.63 seconds viewing each HPF image, which was shorter than the average time of 10.27 seconds they spent annotating the images in the post-training survey.

Quotes

"Although primarily focused on mitosis, we envision that insights from this study can be generalized to other medical imaging tasks."
"Results indicated that CNNs trained with our eye-gaze labels closely followed the performance of ground-truth-based CNNs, and significantly outperformed the baseline."

Key Insights Distilled From

Supporting Mitosis Detection AI Training with Inter-Observer Eye-Gaze Consistencies

by Hongyan Gu,Z... at arxiv.org 04-03-2024

https://arxiv.org/pdf/2404.01656.pdf

Deeper Inquiries

How can the quality of eye-gaze-based labels be further improved to achieve performance on par with ground truth annotations?

To enhance the quality of eye-gaze-based labels and align their performance with ground truth annotations, several strategies can be implemented:

Refinement through Iterative Processes: Implementing iterative processes where eye-gaze labels are refined based on feedback loops can help improve accuracy. By incorporating feedback from pathologists or utilizing multiple rounds of eye-gaze data collection and label generation, the labels can be iteratively refined to reduce errors and inconsistencies.

Integration of Multiple Modalities: Combining eye-gaze data with other modalities, such as color information or additional contextual cues, can provide complementary information for more accurate labeling. By integrating multiple sources of data, the labels can be enriched and refined, leading to higher quality annotations.

Validation and Verification Mechanisms: Implementing validation and verification mechanisms to cross-check the accuracy of eye-gaze labels can help identify and correct errors. By comparing eye-gaze labels with ground truth annotations or utilizing consensus among multiple observers, the quality of the labels can be validated and improved.

Utilizing Advanced Machine Learning Techniques: Leveraging advanced machine learning techniques, such as active learning or semi-supervised learning, can help optimize the training process with eye-gaze labels. By incorporating these techniques, the models can learn from the eye-gaze data more effectively, leading to improved performance.

Incorporating Domain-Specific Knowledge: Integrating domain-specific knowledge and expertise, such as insights from pathologists or domain specialists, can enhance the quality of eye-gaze labels. By leveraging domain knowledge to guide the labeling process, the labels can capture relevant information critical for accurate AI model training.

What are the potential limitations or biases introduced by using non-medical participants instead of pathologists in the eye-tracking study, and how can future research address this?

Using non-medical participants in an eye-tracking study instead of pathologists can introduce several limitations and biases:

Lack of Domain Expertise: Non-medical participants may lack the domain expertise and knowledge required to accurately identify and annotate medical images, leading to potential errors in labeling mitoses or other pathology features.

Differences in Interpretation: Non-medical participants may interpret and focus on different aspects of the images compared to pathologists, potentially resulting in discrepancies in eye-gaze patterns and annotations.

Biases in Gaze Patterns: Non-medical participants may exhibit different gaze patterns or behaviors while viewing medical images, which could introduce biases in the eye-gaze data and impact the quality of labels generated.

To address these limitations and biases, future research can consider the following approaches:

Incorporating Pathologist Oversight: Including pathologists in the study design and data validation process can help ensure the accuracy and reliability of the eye-gaze data and labels. Pathologists can provide guidance, validate annotations, and offer insights to improve the quality of the study.

Training Non-Medical Participants: Providing training sessions and educational materials to non-medical participants can help familiarize them with the task of mitosis detection and improve their understanding of the pathology images. This can enhance their ability to provide more accurate annotations.

Comparative Analysis with Pathologists: Conducting comparative analyses between annotations from non-medical participants and pathologists can help identify discrepancies and biases. By evaluating the agreement between the two groups, researchers can assess the reliability of the eye-gaze data and address any inconsistencies.

Gradual Transition to Pathologist Involvement: Gradually transitioning from non-medical participants to pathologists in the study can help bridge the gap in expertise and ensure a smooth integration of domain knowledge. This phased approach can facilitate a seamless transition while maintaining data quality.

Given the promising results in mitosis detection, how could this approach be extended to support the development of AI models for other pathology tasks, such as tumor grading or disease diagnosis?

The successful application of eye-gaze data in mitosis detection opens up opportunities to extend this approach to support the development of AI models for various other pathology tasks, such as tumor grading and disease diagnosis. Here are some ways to expand this approach:

Task-Specific Label Generation: Tailoring the eye-gaze label generation process to specific pathology tasks, such as tumor grading or disease diagnosis, can provide task-specific cues for AI model training. By capturing relevant visual patterns and features through eye-gaze data, the models can learn to identify and classify different pathologies accurately.

Multi-Modal Data Fusion: Integrating eye-gaze data with other modalities, such as histopathological images, clinical data, or genetic information, can enrich the training data and provide a comprehensive view of the pathology tasks. By fusing multiple sources of data, AI models can benefit from a more holistic understanding of the underlying pathologies.

Transfer Learning and Generalization: Leveraging transfer learning techniques and generalization strategies can enable the AI models trained on eye-gaze data for mitosis detection to adapt to other pathology tasks. By transferring knowledge learned from one task to another, the models can generalize their learning and apply it to new challenges.

Collaboration with Domain Experts: Collaborating with pathologists and domain experts in the development and validation of AI models for different pathology tasks is crucial. By involving experts in the process, researchers can ensure the clinical relevance and accuracy of the models, leading to more reliable diagnostic outcomes.

Validation and Clinical Trials: Conducting validation studies and clinical trials to evaluate the performance of AI models trained with eye-gaze data for tumor grading or disease diagnosis is essential. By testing the models in real-world clinical settings and comparing their performance against standard practices, researchers can assess their efficacy and potential for clinical integration.