Whisper-Flamingo: Adapting Whisper for Improved Audio-Visual Speech Recognition and Translation by Integrating Visual Features
Whisper-Flamingo, a novel model integrating visual features from AV-HuBERT into the Whisper model using gated cross attention, achieves state-of-the-art performance in both audio-visual speech recognition and translation, demonstrating significant improvements in noisy conditions.
Audio-Visual Cross-Modal Alignment for Visual Speech Recognition: Leveraging Audio to Enhance Lip-Reading Accuracy
This research paper introduces AlignVSR, a novel method for visual speech recognition (VSR) that leverages audio information to significantly improve the accuracy of lip-reading by aligning audio and visual modalities through a two-layer alignment mechanism.
Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module
The proposed DCIM-AVSR model introduces an efficient asymmetric architecture that prioritizes the audio modality while treating the visual modality as supplementary, enabling more effective integration of multi-modal information through the Dual Conformer Interaction Module (DCIM).
BRAVEn: Self-Supervised Learning of Robust Visual and Auditory Speech Representations from Raw Audio-Visual Data
BRAVEn, an extension to the RAVEn method, learns strong visual and auditory speech representations entirely from raw audio-visual data, achieving state-of-the-art performance among self-supervised methods in various settings.
© 2024 by Linnk AI