Decayed Identity Shortcuts Improve Self-Supervised Abstract Feature Learning

Decaying the contribution of identity shortcuts in residual connections can substantially improve the quality of abstract features learned by self-supervised masked autoencoders.