Core Concepts
Reasonably observing and improving fine-grained cooperation between modalities enhances multimodal learning.
Stats
One primary topic of multimodal learning is to jointly incorporate heterogeneous information from different modalities.
Most models often suffer from unsatisfactory multimodal cooperation, which cannot jointly utilize all modalities well.
Some methods are proposed to identify and enhance the worse learnt modality, but they are often hard to provide the fine-grained observation of multimodal cooperation at sample-level with theoretical support.
Quotes
"Our methods reasonably observe the fine-grained uni-modal contribution and achieve considerable improvement."
"Our method can reasonably valuate fine-grained modality contribution, and targetedly enhance the learning of low-contributing modality."