Uncovering Annotation Inconsistencies in Death Investigation Notes to Improve Suicide Cause Attribution

Core Concepts

Annotation inconsistencies in unstructured death investigation notes can lead to misattributed suicide causes, hindering effective suicide prevention strategies. This study proposes an empirical NLP approach to detect these inconsistencies, identify problematic instances, and demonstrate the effectiveness of correcting them.

Abstract

This study aimed to uncover annotation inconsistencies in unstructured death investigation notes from the National Violent Death Reporting System (NVDRS) and their impact on suicide cause attribution. Key highlights: Demonstrated the existence of data annotation inconsistencies across states, leading to performance disparities in suicide crisis prediction systems. Introduced a method to identify problematic instances responsible for these inconsistencies through a cross-validation-like paradigm. Found that for Ohio, 14.8% of the annotations for Family Relationship Crisis, 13.9% for Physical Health Crisis, and 1.5% for Mental Health Crisis were potential mistakes; for Colorado, 7.7%, 4.9%, and 2.0% respectively. Showed that removing the problematic instances improved model performance and generalizability, with an average increase of 2.2% in the average micro F1 scores on the test set of other states. Manually rectified 159 potential mistakes in Ohio's Family Relationship Crisis annotations, finding 89 to be actual mis-labelings, which led to a 4.2% increase in the average micro F1 score on the test set of other states and a 3.5% increase on Ohio's test set. Analyzed the risk of bias in the data annotations, observing differences in the Odds Ratios for demographic subgroups (age, race, sex) before and after removing the identified mistakes. The findings highlight the importance of addressing annotation inconsistencies in unstructured death investigation notes to improve the accuracy and reliability of suicide cause attribution, ultimately supporting more effective suicide prevention strategies.

Stats

The National Violent Death Reporting System (NVDRS) dataset contains 267,804 recorded suicide death incidents from 2003 to 2020 across all 50 U.S. states, Puerto Rico, and the District of Columbia.

Quotes

"Recent studies suggested the annotation inconsistencies within the NVDRS and the potential impact on erroneous suicide-cause attributions." "Our results showed that incorporating the target state's data into training the suicide-crisis classifier brought an increase of 5.4% to the F1 score on the target state's test set and a decrease of 1.1% on other states' test set."

Key Insights Distilled From

Uncovering Misattributed Suicide Causes through Annotation Inconsistency Detection in Death Investigation Notes

by Song Wang,Yi... at arxiv.org 03-29-2024

https://arxiv.org/pdf/2403.19432.pdf

Deeper Inquiries

How can the proposed NLP approach be extended to other types of unstructured clinical notes beyond death investigation reports to improve data quality and consistency across various healthcare domains?

The proposed NLP approach can be extended to other types of unstructured clinical notes by adapting the methodology to suit the specific characteristics of the new data sources. Here are some ways to achieve this extension: Data Preprocessing: Just like in the death investigation reports, preprocessing steps such as tokenization, sentence segmentation, and handling of special characters may be necessary for other clinical notes. Feature Engineering: Identify relevant features or entities in the new clinical notes that are crucial for the specific healthcare domain. This may involve creating custom dictionaries or ontologies to capture domain-specific information. Model Training: Utilize pre-trained language models like BioBERT or other transformer-based models that have shown effectiveness in processing clinical text data. Fine-tune these models on the new dataset to improve performance. Annotation Consistency: Implement a systematic approach to detect and correct annotation inconsistencies in the new dataset. This may involve developing rules or algorithms to identify discrepancies and ensure data quality. Bias Detection: Extend the bias analysis to the new dataset to identify potential biases introduced during the manual annotation process. This can help in understanding and mitigating biases in the data. Validation and Generalization: Validate the NLP models on diverse datasets from various healthcare domains to ensure generalizability and robustness. This will help in developing models that can handle different types of clinical notes effectively. By following these steps and customizing the approach to the specific characteristics of the new clinical notes, the proposed NLP methodology can be extended to improve data quality and consistency across various healthcare domains.

How can the insights from this study be leveraged to develop more robust and generalizable NLP models that can handle diverse data sources and annotation inconsistencies in the context of public health research and policy development?

The insights from this study can be leveraged to develop more robust and generalizable NLP models by incorporating the following strategies: Transfer Learning: Utilize transfer learning techniques to adapt pre-trained models to new datasets and tasks. Fine-tuning models like BioBERT on diverse data sources can improve their performance and generalizability. Ensemble Learning: Implement ensemble learning methods to combine predictions from multiple NLP models trained on different datasets. This can help in handling diverse data sources and improving model robustness. Domain Adaptation: Apply domain adaptation techniques to adjust NLP models to specific characteristics of different data sources. This can help in addressing annotation inconsistencies and biases in public health research datasets. Continuous Learning: Implement a continuous learning framework where NLP models are updated and re-trained periodically with new data. This ensures that the models stay relevant and effective in handling evolving data sources. Interpretability and Explainability: Enhance the interpretability of NLP models to understand how they make predictions and identify potential sources of errors. This can help in improving model performance and addressing annotation inconsistencies. Collaborative Annotation: Foster collaboration among annotators from diverse backgrounds to ensure a comprehensive and unbiased annotation process. This can help in mitigating biases and improving data quality in public health research and policy development. By incorporating these strategies and leveraging the insights from this study, more robust and generalizable NLP models can be developed to handle diverse data sources and annotation inconsistencies in the context of public health research and policy development.