Efficient Head-level Adaptation of Vision Transformers using Taylor-expansion Importance Scores

A simple and effective method, HEAT, that efficiently fine-tunes Vision Transformers at the head level by leveraging Taylor-expansion Importance Scores to identify and mask redundant attention heads.