Efficient and Effective Open-Vocabulary Object Detection via Self-Training and Split-Fusion Detection Head

Core Concepts

The core message of this work is to tame self-training for open-vocabulary object detection (OVD) by addressing two key challenges: noisy pseudo labels from vision-language models and frequent distribution changes of pseudo labels. The authors propose a split-and-fusion (SAF) detection head and a periodic update strategy to effectively leverage self-training for OVD.

Abstract

The paper proposes a method called Self-training And Split-and-fusion head for open-vocabulary Detection (SAS-Det) to address the challenges of applying self-training to open-vocabulary object detection (OVD).
The key insights are:

Noisy pseudo labels (PLs) from vision-language models (VLMs) used in OVD can degrade the performance, especially when using self-training. The authors introduce a split-and-fusion (SAF) detection head to handle this issue.

The SAF head splits the standard detection head into an "open-branch" and a "closed-branch".
The closed-branch is trained solely on ground truth boxes and class labels of base categories, mitigating the impact of noisy PLs.
The open-branch is trained on class labels of both ground truth and PLs, acquiring complementary knowledge.
The predictions from the two branches are fused at inference to boost the performance.

Unlike closed-set object detection, the distribution of PLs in OVD is solely determined by the teacher model. Frequent updates to the teacher model can lead to unstable training due to changing PL distributions.

The authors propose a periodic update strategy to the teacher model, reducing the frequency of PL distribution changes.

Extensive experiments on COCO-OVD and LVIS-OVD benchmarks demonstrate that SAS-Det outperforms recent OVD methods by a clear margin. The ablation studies validate the effectiveness of the proposed components.

Stats

SAS-Det outperforms recent OVD models of the same scale by 4.3 APnovel50 on novel categories and 3.7 APbase50 on base categories on COCO-OVD.
On LVIS-OVD, SAS-Det with ResNet50x4 backbone achieves 29.1 APr on rare categories, improving by 8.2 over the previous best.
Removing the external RPN and training a Faster R-CNN detector directly with PLs from SAS-Det achieves similar performance as the full SAS-Det model.
The time cost of pseudo labeling in SAS-Det is nearly 4 times faster than PB-OVD and 3 times faster than VL-PLM.

Quotes

"Unlike semi-supervised object detection, OVD has no ground truth for the target categories. Thus, the distribution of the target data (i.e., PLs) is fully determined by the teacher. The EMA update changes the PLs' distribution in each iteration and leads to a constantly shifting training target that has been shown hard to optimize (i.e., destabilizing the training) in deep Q-learning [35]."
"The closed-branch is trained solely on ground truth boxes and class labels of base categories, mitigating the impact of noisy PLs. The open-branch is trained on class labels of both ground truth and PLs, acquiring complementary knowledge."

Key Insights Distilled From

Taming Self-Training for Open-Vocabulary Object Detection

by Shiyu Zhao,S... at arxiv.org 04-16-2024

https://arxiv.org/pdf/2308.06412.pdf

Deeper Inquiries

How can the proposed SAF head be further improved to better handle the noise in pseudo labels?

The proposed SAF head can be further improved to better handle the noise in pseudo labels by incorporating additional mechanisms for noise reduction and robustness. Here are some potential enhancements:

Dynamic Thresholding: Implement a dynamic thresholding mechanism that adapts to the confidence scores of the pseudo labels. Instead of using a fixed threshold, the threshold value can be adjusted based on the distribution of confidence scores to filter out noisy pseudo labels effectively.

Adaptive Loss Functions: Introduce adaptive loss functions that prioritize the correction of noisy pseudo labels during training. By assigning higher weights to instances with uncertain or noisy labels, the model can focus more on learning from reliable pseudo labels.

Uncertainty Estimation: Incorporate uncertainty estimation techniques to quantify the uncertainty associated with each pseudo label. By considering the uncertainty in the training process, the model can assign less weight to noisy pseudo labels and prioritize learning from more reliable instances.

Consistency Regularization: Implement consistency regularization techniques to enforce consistency between predictions from the open-branch and closed-branch. By penalizing inconsistencies in predictions, the model can learn to filter out noisy pseudo labels that lead to conflicting predictions.

Ensemble Learning: Utilize ensemble learning methods by training multiple instances of the model with different initializations or architectures. By aggregating predictions from multiple models, the ensemble can reduce the impact of noisy pseudo labels and improve overall performance.

How can the potential limitations of the periodic update strategy be addressed, and how can it be extended to other self-training scenarios beyond OVD?

The periodic update strategy, while effective in stabilizing the training process in OVD, may have limitations in scenarios with different characteristics. Here are some ways to address these limitations and extend the strategy to other self-training scenarios:

Adaptive Update Frequency: Implement an adaptive update frequency mechanism that dynamically adjusts the frequency of updates based on the convergence of the model. By monitoring the training progress, the system can determine the optimal timing for updates to prevent underfitting or overfitting.

Regularization Techniques: Introduce regularization techniques such as weight decay or dropout during the periodic updates to prevent overfitting and improve generalization. By adding regularization, the model can maintain stability during updates and avoid drastic changes in the learned representations.

Cross-Validation: Incorporate cross-validation to validate the effectiveness of the periodic update strategy on validation data. By evaluating the performance of the model at each update interval, the system can ensure that the updates are beneficial and do not lead to performance degradation.

Transfer Learning: Extend the periodic update strategy to other self-training scenarios by leveraging transfer learning techniques. By transferring the knowledge gained from OVD to different tasks, the periodic update strategy can be adapted and fine-tuned for specific domains or datasets.

Can the SAS-Det framework be applied to other vision-language tasks beyond object detection, such as visual question answering or image captioning?

Yes, the SAS-Det framework can be adapted and applied to other vision-language tasks beyond object detection, such as visual question answering (VQA) or image captioning. Here's how the SAS-Det framework can be extended to these tasks:

Visual Question Answering (VQA): In VQA, the SAF head can be modified to handle the multimodal nature of the task, where both image features and textual inputs are combined. The split-and-fusion architecture can be adapted to process both visual and textual information, with branches specialized for handling different modalities. This can help improve the model's performance by leveraging complementary knowledge from both modalities.

Image Captioning: For image captioning tasks, the SAF head can be integrated into the encoder-decoder architecture commonly used in image captioning models. The split-and-fusion mechanism can be applied to the encoder to handle noisy visual features or textual inputs. By splitting the encoding process into branches and fusing the representations, the model can better capture the semantics of the image and generate more accurate captions.

Cross-Modal Retrieval: The SAS-Det framework can also be extended to cross-modal retrieval tasks, where the goal is to retrieve relevant information across different modalities. By adapting the SAF head to handle cross-modal representations, the model can learn to align visual and textual features effectively, improving retrieval performance.

Overall, the modular and flexible nature of the SAS-Det framework makes it suitable for adaptation to various vision-language tasks beyond object detection, enabling improved performance and robustness in multimodal learning scenarios.