
The modality gap, a separation of image and text embeddings in the shared representation space, and the bias towards objects over other factors, such as attributes, are two key challenges in contrastive vision-language representation learning. The driving factor behind both phenomena is the information imbalance between images and their captions.


coremsg

investigating-the-modality-gap-and-object-bias-in-contrastive-vision-language-representation-learning


Investigating the Modality Gap and Object Bias in Contrastive Vision-Language Representation Learning


title_rewrite


This paper proposes an improved probabilistic cross-modal embedding model, PCME++, to effectively capture the inherent ambiguity in image-text matching datasets caused by multiplicity and abundant false negatives. PCME++ introduces a new closed-form probabilistic distance, optimization techniques using pseudo-positives and mixed sample data augmentation, and demonstrates superior performance on various benchmarks.


improved-probabilistic-image-text-representations-for-mitigating-ambiguity-in-vision-language-datasets


Improved Probabilistic Image-Text Representations for Mitigating Ambiguity in Vision-Language Datasets
