Audio-Visual Learning

insight - Audio-Visual Learning

Audio-Visual Generalized Zero-Shot Learning Using Pre-Trained Large Multi-Modal Models

Our proposed framework leverages features from pre-trained multi-modal models CLIP and CLAP to achieve state-of-the-art performance on audio-visual generalized zero-shot learning benchmarks.

Text-Guided Visual Sound Source Localization in Multi-Source Mixtures

The core message of this paper is to leverage the text modality as an intermediate feature guide using tri-modal joint embedding models (e.g., AudioCLIP) to disentangle the semantic audio-visual source correspondence in multi-source mixtures for improved visual sound source localization.

Efficient and Scalable Audio-Visual Representation Learning with Siamese Vision Transformers

A single shared vision transformer backbone can effectively process both audio and visual inputs, leading to an efficient and scalable audio-visual pretraining framework that outperforms prior approaches using separate audio and visual encoders.

Leveraging Equivariance for Audio-Visual Contrastive Learning with EquiAV Framework

EquiAV introduces a novel framework leveraging equivariance for audio-visual contrastive learning, outperforming previous methods across various benchmarks.

About

Products | Resources

Insights