Improving Algorithm, Model, and Data Efficiency of Self-Supervised Learning through Non-Parametric Instance Discrimination and Self-Distillation

The authors propose an efficient single-branch self-supervised learning method based on non-parametric instance discrimination, with improved feature bank initialization, gradient-based update rule, and a novel self-distillation loss to enhance algorithm, model, and data efficiency.