核心概念
The authors propose a novel framework for building Concept Bottleneck Models (CBMs) from pre-trained multi-modal encoders like CLIP. Their approach leverages Gumbel tricks and contrastive learning to create sparse and interpretable inner representations in the CBM, leading to significant improvements in accuracy compared to prior CBM methods.
要約
The paper introduces a framework for building Concept Bottleneck Models (CBMs) from pre-trained multi-modal encoders like CLIP. The key contributions are:
-
The Concept Matrix Search (CMS) algorithm, which uses CLIP's capabilities to represent both images and text in a joint latent space to improve the interpretability of CLIP's predictions without any additional training.
-
A framework for creating CBMs from pre-trained multi-modal encoders. This framework includes novel architectures and training methods that leverage contrastive learning and Gumbel tricks to create sparse and interpretable inner representations in the CBM.
-
Three variants of the CBM framework are proposed: Sparse-CBM, Contrastive-CBM, and ℓ1-CBM, each with different objective functions for training the Concept Bottleneck Layer (CBL).
The authors show that their Sparse-CBM outperforms prior CBM approaches on several datasets, demonstrating the benefits of sparse inner representations for interpretability and accuracy. They also provide extensive analysis on the impact of the concept set on the CMS algorithm's performance.
統計
The authors report the following key metrics:
"Sparse-CBM (ours) achieves 91.17% accuracy on CIFAR10, 74.88% on CIFAR100, 71.61% on ImageNet, 80.02% on CUB200, and 41.34% on Places365."
"Concept Matrix Search (ours) achieves 85.03% accuracy on CIFAR10, 62.95% on CIFAR100, 77.82% on ImageNet, 65.17% on CUB200, and 39.43% on Places365."
引用
"We show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models. Which means that sparse representation of concepts activation vector is meaningful in Concept Bottleneck Models."
"By introducing a new type of layers known as Concept Bottleneck Layers, we outline three methods for training them: with ℓ1-loss, contrastive loss and loss function based on Gumbel-Softmax distribution (Sparse-CBM), while final FC layer is still trained with Cross-Entropy."