The paper introduces a framework for building Concept Bottleneck Models (CBMs) from pre-trained multi-modal encoders like CLIP. The key contributions are:
The Concept Matrix Search (CMS) algorithm, which uses CLIP's capabilities to represent both images and text in a joint latent space to improve the interpretability of CLIP's predictions without any additional training.
A framework for creating CBMs from pre-trained multi-modal encoders. This framework includes novel architectures and training methods that leverage contrastive learning and Gumbel tricks to create sparse and interpretable inner representations in the CBM.
Three variants of the CBM framework are proposed: Sparse-CBM, Contrastive-CBM, and ℓ1-CBM, each with different objective functions for training the Concept Bottleneck Layer (CBL).
The authors show that their Sparse-CBM outperforms prior CBM approaches on several datasets, demonstrating the benefits of sparse inner representations for interpretability and accuracy. They also provide extensive analysis on the impact of the concept set on the CMS algorithm's performance.
Sang ngôn ngữ khác
từ nội dung nguồn
arxiv.org
Thông tin chi tiết chính được chắt lọc từ
by Andrei Semen... lúc arxiv.org 04-05-2024
https://arxiv.org/pdf/2404.03323.pdfYêu cầu sâu hơn