Core Concepts
Masked image modeling (MIM) can serve as a framework for self-supervised learning of visual representations that aligns with the focused nature of biological perception through eye movements and attention shifts.
Abstract
The paper investigates key components of masked image modeling (MIM) through the lens of biological vision. It proposes that eye movements and the focused nature of primate vision can be cast as a generative, self-supervised task of predicting and revealing visual information.
The main findings are:
A peripheral masking strategy, inspired by foveal visual perception, leads to strong representations and is more biologically plausible than common random patch-wise masks. This peripheral masking approach implicitly decorrelates latent space neurons, reminiscent of sparse coding.
Data augmentation through crop-and-resize transformations is crucial for the peripheral masking strategy, but not for random patch-wise masks, suggesting the model requires exposure to a variety of prediction tasks from each object.
Restricting the pretraining loss to the main object, rather than the background, does not negatively impact representation quality, indicating that on-object predictions are sufficient for classification.
The networks also reconstruct image content in areas where no loss was calculated, suggesting a holistic percept that fits with the proposed role of visual predictions in creating a stable visual representation.
Overall, the results support the idea that MIM, cast as a framework for predicting and revealing visual information through simulated eye movements, can serve as a candidate mechanism for self-supervised learning in the brain, potentially complementing previously proposed non-generative methods.