The paper introduces the ATtentiOn Mixer (ATOM) framework for efficient dataset distillation. The key contributions are:
ATOM utilizes a mixture of spatial and channel-wise attention to capture both localization and contextual information in the feature matching process. Spatial attention helps guide the learning based on consistent class localization, while channel-wise attention captures the contextual information associated with the class.
ATOM demonstrates superior performance across various computer vision datasets, including CIFAR-10/100 and Tiny-ImageNet, especially in scenarios with a low number of images per class. It outperforms previous dataset distillation methods by a significant margin.
ATOM maintains the performance improvement on cross-architectures, including classic CNNs and Vision Transformers, and applications such as neural architecture search. The channel-only variant of ATOM also provides a good trade-off between performance and computational complexity.
Extensive ablation studies are conducted to evaluate the impact of different attention mechanisms and their balance in the ATOM framework. The results show that channel-wise attention plays a crucial role in capturing relevant information for efficient dataset distillation.
Overall, the ATOM framework provides an effective and efficient approach to dataset distillation, addressing the limitations of previous methods in terms of computational costs, performance, and cross-architecture generalization.
To Another Language
from source content
arxiv.org
ข้อมูลเชิงลึกที่สำคัญจาก
by Samir Khaki,... ที่ arxiv.org 05-03-2024
https://arxiv.org/pdf/2405.01373.pdfสอบถามเพิ่มเติม