The paper introduces a framework for building Concept Bottleneck Models (CBMs) from pre-trained multi-modal encoders like CLIP. The key contributions are:
The Concept Matrix Search (CMS) algorithm, which uses CLIP's capabilities to represent both images and text in a joint latent space to improve the interpretability of CLIP's predictions without any additional training.
A framework for creating CBMs from pre-trained multi-modal encoders. This framework includes novel architectures and training methods that leverage contrastive learning and Gumbel tricks to create sparse and interpretable inner representations in the CBM.
Three variants of the CBM framework are proposed: Sparse-CBM, Contrastive-CBM, and ℓ1-CBM, each with different objective functions for training the Concept Bottleneck Layer (CBL).
The authors show that their Sparse-CBM outperforms prior CBM approaches on several datasets, demonstrating the benefits of sparse inner representations for interpretability and accuracy. They also provide extensive analysis on the impact of the concept set on the CMS algorithm's performance.
toiselle kielelle
lähdeaineistosta
arxiv.org
Tärkeimmät oivallukset
by Andrei Semen... klo arxiv.org 04-05-2024
https://arxiv.org/pdf/2404.03323.pdfSyvällisempiä Kysymyksiä