Unified Template Filling for Multimodal Multimedia Event Argument Extraction
A unified template filling framework that connects textual and visual modalities via natural language prompts to effectively address the event argument extraction task.
Open-vocabulary Multimodal Emotion Recognition: Expanding Emotion Modeling Beyond Basic Categories
This paper proposes a new paradigm called Open-vocabulary Multimodal Emotion Recognition (OV-MER) that enables the prediction of any number and category of emotions, advancing emotion recognition from basic to more nuanced emotions.
Multimodal Shannon Game: Exploring the Impact of Visual Context on Next-Word Prediction
The addition of visual information, in various forms, improves both self-reported confidence and accuracy for next-word prediction in both humans and language models.
簡単だが強力な音声付き動画生成のためのベースライン: オーディオとビデオの拡散モデルの効果的な適応による統合生成
提案手法は、既存の音声と動画の拡散モデルを効果的に統合し、時系列調整とクロスモーダル条件付けの新しいメカニズムを導入することで、高品質かつ時間的に整合性の取れた音声付き動画を生成することができる。
Multimodal Prompt Tuning for Efficient Zero-shot Instruction Learning
Multimodal Prompt Tuning (MMPT) is a novel approach that effectively integrates visual and textual prompts into the vision encoder and language processor, respectively, to enable efficient and accurate multimodal adaptation for zero-shot instruction learning.
Visual Prompting in Multimodal Large Language Models: A Comprehensive Survey
This paper presents a comprehensive survey on visual prompting methods in multimodal large language models (MLLMs), covering visual prompt generation, integration into MLLM perception and reasoning, and model alignment techniques.
Multimodal Foundation Models Outperform Audio-Only Models for Non-Verbal Emotion Recognition
Multimodal foundation models (MFMs) like LanguageBind and ImageBind outperform audio-only foundation models (AFMs) for non-verbal emotion recognition (NVER) tasks by better capturing subtle emotional cues through their joint pre-training across multiple modalities.
Towards Comprehensive Evaluation of Multimodal Language Models: Introducing OmniBench
OmniBench is a novel benchmark designed to rigorously evaluate multimodal large language models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously.
Evaluating and Benchmarking the Multimodal Reasoning Capabilities of Large Foundation Models
Current state-of-the-art large foundation models exhibit varying strengths and weaknesses in multimodal reasoning capabilities, with no single model outperforming others across all tasks. Detailed evaluation reveals opportunities for improvement in areas like geometric reasoning, benefiting from multimodal input, and grounding information retrieval.
オンラインパーツ市場における ImageBind の多様なエンベディング空間の分析
ImageBindを使用して、オンラインオートパーツ広告の画像とテキストを融合したマルチモーダルな埋め込み表現を生成し、その意味的な質を分析した。さらに、純粋な音声埋め込みとの相関関係を示すことで、ImageBindの潜在的な応用分野を示唆した。
© 2024 by Linnk AI