Efficient Fine-Tuning of Vision Transformers via Salient Channel Selection

Tuning only a small subset of task-specific channels in vision transformers can achieve competitive performance compared to full fine-tuning, while significantly reducing the number of trainable parameters.