Efficient and Theoretically Grounded Nonparametric Modern Hopfield Models

This work presents a nonparametric framework for constructing efficient and theoretically grounded modern Hopfield models, which serve as powerful alternatives to attention mechanisms in deep learning. The proposed sparse-structured modern Hopfield models achieve sub-quadratic complexity while retaining the appealing properties of their dense counterparts, including fixed point convergence, exponential memory capacity, and connection to transformer attention.