Revisiting Google Landmarks v2 Dataset: Importance of Avoiding Class Overlap Between Training and Evaluation Sets for Image Retrieval

Concepts de base

Removing class overlap between the training set (Google Landmarks v2 clean) and the evaluation sets (Revisited Oxford and Paris) leads to a dramatic drop in performance across state-of-the-art image retrieval methods, highlighting the critical importance of avoiding such overlap. We introduce CiDeR, a single-stage, end-to-end pipeline that detects objects of interest and extracts a global image representation without requiring location supervision, outperforming previous state-of-the-art methods on both the original and the new, revisited dataset.

Résumé

The paper revisits the Google Landmarks v2 clean (GLDv2-clean) dataset, which is widely used for training state-of-the-art image retrieval models. The authors identify and remove landmark categories that overlap between the GLDv2-clean training set and the Revisited Oxford and Paris (ROxf and RPar) evaluation sets, creating a new version called RGLDv2-clean.

The authors then reproduce several state-of-the-art image retrieval methods using the same backbone network and training settings, and compare their performance on the original GLDv2-clean and the new RGLDv2-clean datasets. The results show a dramatic drop in performance when training on RGLDv2-clean, despite the small fraction of images and categories removed, highlighting the critical importance of avoiding class overlap between training and evaluation sets.

To address the challenge of detecting objects of interest and ignoring background clutter, the authors introduce CiDeR, a single-stage, end-to-end pipeline that uses an attention-based approach to localize objects and extract a global image representation. CiDeR does not require any location supervision, unlike previous detect-to-retrieve (D2R) methods that often involve complex two-stage training and indexing pipelines.

Experiments show that CiDeR outperforms previous state-of-the-art methods on both the original GLDv2-clean and the new RGLDv2-clean datasets, demonstrating the effectiveness of the proposed approach.

Stats

The authors report that the Revisited Oxford (ROxf) and Revisited Paris (RPar) evaluation sets have landmark overlap with the original GLDv2-clean training set for 51% and 54% of the queries, respectively.
The new RGLDv2-clean dataset has 1,565 fewer images from 18 landmark categories compared to the original GLDv2-clean.

Citations

"Not only is there a dramatic drop in performance, but it is inconsistent across methods, changing the ranking."
"We outperform previous state-of-the-art on both existing training sets and the new RGLDv2-clean."

Idées clés tirées de

On Train-Test Class Overlap and Detection for Image Retrieval

by Chull Hwan S... à arxiv.org 04-03-2024

https://arxiv.org/pdf/2404.01524.pdf

Questions plus approfondies

How can the proposed CiDeR approach be extended to handle more complex scenes with multiple objects of interest

The CiDeR approach can be extended to handle more complex scenes with multiple objects of interest by incorporating multi-object detection capabilities. Currently, CiDeR focuses on detecting a single object of interest using spatial attention mechanisms. To handle multiple objects, the model can be modified to detect and localize multiple objects within an image. This can be achieved by enhancing the attention mechanism to identify and segment multiple objects simultaneously. Additionally, the model can be trained on datasets that contain images with multiple objects and their corresponding annotations. By adjusting the architecture and training process to accommodate multiple objects, CiDeR can effectively handle complex scenes with multiple objects of interest.

What are the potential limitations of the attention-based localization mechanism used in CiDeR, and how could it be further improved

The attention-based localization mechanism used in CiDeR may have some limitations that could be further improved. One potential limitation is the sensitivity of the model to background clutter or irrelevant features in the image. The attention mechanism may not always accurately focus on the object of interest, especially in cluttered scenes. To address this limitation, the model can be enhanced with more sophisticated attention mechanisms that are robust to background distractions. Techniques like self-attention or hierarchical attention can help the model better focus on relevant objects while ignoring irrelevant background elements. Additionally, incorporating contextual information and feedback mechanisms can improve the accuracy and robustness of the attention-based localization mechanism in CiDeR.

Given the importance of avoiding class overlap between training and evaluation sets, how can this principle be applied to the development of other computer vision datasets and benchmarks

To apply the principle of avoiding class overlap between training and evaluation sets to the development of other computer vision datasets and benchmarks, several strategies can be implemented. Firstly, dataset curators should carefully analyze and filter out images that contain objects or classes present in the evaluation set to ensure no overlap. This process may involve manual inspection, automated filtering, or a combination of both. Additionally, dataset creators should establish clear guidelines and protocols for dataset collection to prevent inadvertent class overlap. Regular audits and updates to the dataset can help maintain the integrity of the class separation. Furthermore, benchmark organizers should prioritize datasets that adhere to the no class overlap principle to ensure fair and unbiased evaluations of computer vision models. By promoting transparency and adherence to best practices in dataset creation, the computer vision community can uphold the importance of avoiding class overlap in training and evaluation sets.