Blink: Multimodal Large Language Models Struggle with Core Visual Perception Tasks

Core Concepts

Existing multimodal large language models (LLMs) exhibit significant limitations in core visual perception abilities, performing far below human levels on a range of classic computer vision tasks.

Abstract

The paper introduces Blink, a new benchmark designed to evaluate the visual perception capabilities of multimodal large language models (LLMs). Blink consists of 14 classic computer vision tasks, ranging from low-level pattern matching to high-level visual understanding. These tasks are reformatted into multiple-choice question-answering problems, with visual prompts and answer choices. The key findings are: While humans can solve the Blink tasks with 95.7% accuracy on average, even the best-performing multimodal LLMs like GPT-4V and Gemini Pro achieve only around 50% accuracy, merely 13.17% and 7.63% better than random guessing, respectively. This highlights a significant gap between human and machine visual perception abilities. Multimodal LLMs perform relatively better on spatial reasoning, art style, and counting tasks, but struggle on pixel-level and crop-level tasks like relative depth estimation, reflectance comparison, and visual correspondence. Specialist computer vision models significantly outperform multimodal LLMs on Blink tasks, suggesting potential pathways for future improvements in multimodal perception. Experiments show that reducing Blink to text-only questions using dense image captions can yield comparable or better performance than using multimodal LLMs, indicating the limitations of existing multimodal benchmarks in comprehensively evaluating visual perception. The authors believe Blink can serve as an effective testbed for bridging the gap between traditional notions of perception and the modern generative capabilities of multimodal LLMs.

Stats

Humans achieve 95.7% average accuracy on Blink. GPT-4V achieves 51.26% accuracy, only 13.17% better than random guessing. Gemini Pro achieves 45.72% accuracy, only 7.63% better than random guessing. Specialist computer vision models outperform multimodal LLMs by 18% to 57% on specific tasks.

Quotes

"While humans can solve the Blink tasks with 95.7% accuracy on average, even the best-performing multimodal LLMs like GPT-4V and Gemini Pro achieve only around 50% accuracy, merely 13.17% and 7.63% better than random guessing, respectively." "Specialist computer vision models significantly outperform multimodal LLMs on Blink tasks, suggesting potential pathways for future improvements in multimodal perception."

Key Insights Distilled From

BLINK: Multimodal Large Language Models Can See but Not Perceive

by Xingyu Fu,Yu... at arxiv.org 04-19-2024

https://arxiv.org/pdf/2404.12390.pdf

Deeper Inquiries

How can the insights from specialist computer vision models be effectively integrated into the training of multimodal LLMs to improve their visual perception abilities?

Specialist computer vision models excel in specific tasks due to their focused training and specialized architectures. To integrate their insights into the training of multimodal Large Language Models (LLMs) for improved visual perception abilities, several strategies can be employed: Knowledge Distillation: Transfer knowledge from specialist models to multimodal LLMs by distilling the expertise of the former into the latter. This process involves training the LLMs to mimic the behavior and predictions of the specialist models on specific tasks. Fine-tuning with Specialized Data: Fine-tune the multimodal LLMs using datasets that are tailored to the tasks where specialist models perform well. By exposing the LLMs to task-specific data, they can learn to improve their performance in those areas. Architectural Enhancements: Modify the architecture of multimodal LLMs to incorporate components or modules inspired by the design of specialist models. This can include adding specific layers or mechanisms that are known to be effective for certain tasks. Ensemble Learning: Combine the predictions of multimodal LLMs with those of specialist models through ensemble learning techniques. By leveraging the strengths of both types of models, the overall performance can be enhanced. Task-Specific Training: Train the multimodal LLMs on specific tasks where specialist models excel, focusing on improving their performance in those areas. This targeted training can help bridge the gap in visual perception abilities. By implementing these strategies, multimodal LLMs can benefit from the expertise of specialist computer vision models, leading to enhanced visual perception capabilities across a range of tasks.

What are the key limitations of current multimodal LLMs that prevent them from achieving human-level performance on core visual perception tasks?

Current multimodal Large Language Models (LLMs) face several limitations that hinder their ability to achieve human-level performance on core visual perception tasks: Lack of Specialization: Multimodal LLMs are designed to handle a wide range of tasks, leading to a lack of specialization in specific visual perception abilities. This generalization can limit their performance on tasks that require nuanced understanding. Limited Training Data: Multimodal LLMs may not have access to sufficient training data for fine-tuning on specific visual perception tasks, resulting in suboptimal performance in those areas. Inherent Bias: LLMs may exhibit biases in their training data, leading to skewed perceptions and incorrect interpretations of visual information. These biases can impact their performance on diverse tasks. Complexity of Multimodal Integration: Integrating visual and textual information in a coherent manner is a challenging task for LLMs. The complexity of processing and understanding both modalities simultaneously can lead to errors in perception. Difficulty in Spatial Reasoning: Tasks that require spatial reasoning, such as relative depth estimation or multi-view reasoning, can be particularly challenging for LLMs. Understanding spatial relationships accurately is a complex cognitive task that may exceed the current capabilities of these models. Limited Contextual Understanding: While LLMs excel in language processing, their ability to contextualize visual information within the broader context of a scene or image may be limited. This can impact their performance on tasks that require holistic visual understanding. Addressing these limitations through targeted training, architectural enhancements, and data augmentation can help improve the visual perception capabilities of multimodal LLMs.

How can the Blink benchmark be extended or adapted to further push the boundaries of multimodal perception and encourage the development of more capable models?

To further push the boundaries of multimodal perception and foster the development of more capable models, the Blink benchmark can be extended or adapted in the following ways: Incorporating Progressive Complexity: Introduce tasks of increasing complexity to challenge multimodal LLMs at different levels of visual perception. This can include tasks that require advanced reasoning, fine-grained analysis, and abstract understanding. Dynamic Visual Prompts: Experiment with dynamic visual prompts that adapt based on model performance. By providing feedback through varying prompts, models can learn to focus on specific aspects of the visual input and improve their perception abilities. Interactive Elements: Introduce interactive elements in the benchmark where models can actively engage with the visual stimuli. This can include tasks that require manipulation of images, interactive reasoning, or dynamic interactions with the visual content. Real-time Feedback Mechanisms: Implement real-time feedback mechanisms that provide instant evaluation and corrective feedback to models during inference. This can help models learn from their mistakes and improve their performance iteratively. Domain-Specific Extensions: Extend Blink to include tasks from specific domains such as medical imaging, satellite imagery analysis, or robotics. By diversifying the task domains, models can be trained on a broader range of visual perception challenges. Collaborative Benchmarking: Foster collaboration among researchers and practitioners to contribute new tasks, datasets, and evaluation metrics to Blink. By crowdsourcing expertise, the benchmark can evolve to encompass a wider spectrum of visual perception abilities. By incorporating these enhancements, Blink can serve as a dynamic and evolving benchmark that pushes the boundaries of multimodal perception and drives innovation in the development of more advanced and capable models.