Core Concepts
This study introduces ViTAS, a novel adapter that effectively leverages the informative, general-purpose features extracted by vision foundation models to significantly improve stereo matching accuracy and generalizability across diverse datasets.
Abstract
The article presents a comprehensive exploration of adapting vision foundation models (VFMs) to the task of stereo matching. The key highlights are:
Existing stereo matching networks primarily rely on convolutional neural networks (CNNs) for feature extraction, which limits their performance. The authors argue that VFMs, particularly those based on vision Transformers (ViTs), can provide more informative and general-purpose visual features.
The authors propose ViTAS, a modular adapter that consists of three key components:
Spatial Differentiation Module (SDM) to initialize multi-scale feature pyramids from VFM tokens.
Patch Attention Fusion Module (PAFM) to efficiently aggregate multi-scale contextual information.
Cross-Attention Module (CAM) to incorporate stereo contextual information.
Combining ViTAS with a cost volume-based stereo matching back-end yields ViTAStereo, which achieves state-of-the-art performance on the KITTI Stereo 2012 dataset, outperforming the previous best network by up to 11.25% in terms of percentage of error pixels.
Extensive experiments demonstrate the superior generalizability of ViTAStereo across diverse real-world datasets, including KITTI, Middlebury, and ETH3D, compared to other state-of-the-art stereo matching networks.
The authors argue that cost volumes remain essential for developing generalizable stereo matching networks, as opposed to solely relying on cross-attention mechanisms, which can suffer from scale ambiguity issues.
Stats
The article does not provide any specific numerical data or statistics to support the key logics. The performance improvements are reported in terms of percentage improvements in evaluation metrics like end-point error (EPE) and percentage of error pixels (PEP).
Quotes
"This study serves as the first exploration of a viable approach for adapting VFMs to stereo matching."
"ViTAStereo achieves the top rank on the KITTI Stereo 2012 dataset and out-performs the second-best network StereoBase by approximately 7.9% in terms of the percentage of error pixels, with a tolerance of 3 pixels."
"Additional experiments across diverse scenarios further demonstrate its superior generalizability compared to all other state-of-the-art approaches."