Core Concepts
The author presents the Tri-branch Neural Fusion (TNF) approach for classifying multimodal medical images and tabular data, addressing label inconsistency in classification through a tri-branch framework and ensemble method.
Abstract
The paper introduces TNF for multimodal medical data classification, highlighting the challenges of label inconsistency. TNF's tri-branch framework manages separate outputs for image and tabular modalities, integrating likelihoods to improve accuracy over traditional methods. Experiments validate TNF's superiority across various architectures and datasets.
Traditional methods in multi-modality medical data classification often merge features from distinct input modalities, leading to reduced accuracy due to label inconsistencies. To address this challenge, the TNF approach implements a tri-branch framework that manages three separate outputs: one for image modality, another for tabular modality, and a third hybrid output that fuses both image and tabular data. The final decision is made through an ensemble method that integrates likelihoods from all three branches.
In diagnosing diseases like Alzheimer’s, clinicians use diagnostic tools combined with various information sources to make accurate diagnoses. Applying multimodal learning in multi-modality medical data analysis is a recent trend. TNF's flexibility allows it to work even if one modality is missing during the inference stage.
The contributions of the paper include proposing TNF as a high-performance classification structure combining fusion and ensemble methods for multimodal medical data classification. Two approaches named label masking and maximum likelihood selection are proposed to tackle label inconsistency in multimodal classification tasks effectively.
Experiments on multiple datasets demonstrate the generality of TNF, showcasing its superiority over individual fusion or ensemble methods. The results highlight TNF's effectiveness in improving accuracy metrics by 1% to 5% compared to traditional fusion and ensemble methods.
Stats
Traditional methods rely on single-label approaches.
TNF implements a tri-branch framework.
TNF integrates likelihoods from all three branches.
Experiments illustrate TNF's superiority over traditional fusion and ensemble methods.
TNF achieved superior performance across various architectures and datasets.