A Dynamical Model Explaining Neural Scaling Laws

Core Concepts

The core message of this paper is that a dynamical mean field theory (DMFT) model of a randomly projected linear model trained with gradient descent can reproduce many empirically observed neural scaling laws, including the different scaling exponents for training time, model size, and compute budget.

Abstract

The paper introduces a solvable model of network training and generalization based on a randomly projected linear model trained with gradient descent. The key aspects of this model are:

1. The model uses a random projection of the infinite-width neural tangent kernel (NTK) eigenfunctions to represent the finite-width NTK features. This mismatch between the teacher and student models is the key ingredient that leads to the observed scaling laws.

2. The authors derive a DMFT description of the learning dynamics in terms of correlation and response functions, which can be solved exactly in the Fourier domain.

3. For power-law structured features, the model exhibits power-law scaling of test loss with training time, model size, and dataset size. Importantly, the authors show that the time and model exponents are different in general, leading to an asymmetric compute-optimal scaling strategy where training time increases faster than model size.

4. The theory explains why ensembling is not compute-optimal, as it provides less benefit than increasing model size. The authors also observe that feature learning networks can obtain better power-law scalings compared to the linearized model.

5. The model captures the gradual buildup of overfitting effects over time, as well as the different scaling exponents at early vs. late training time observed in realistic deep learning settings.

6. The authors validate the key predictions of their theory on a realistic image classification task using a ResNet architecture, demonstrating excellent agreement with the observed scaling laws.

Stats

The test loss scales as L(t, N) ≈ L0 + at t^(-rt) + aN N^(-rN), where t is training time, N is model size, and the exponents rt and rN depend on the dataset and architecture.
The compute-optimal scaling strategy requires scaling both model size and training time, with the training time scaling faster than model size.
Larger models train faster in the early stages of training, but can exhibit diminishing returns in the data-limited regime.

Quotes

"We obtain the above scaling by approximating the loss as a sum of the three terms in equation (14) and a constant, see Appendix N. This analysis suggests that for features that have rapid decay in their eigenspectrum, it is preferable to allocate greater resources toward training time rather than model size as the compute budget increases."
"Our theory can explain these observations as it predicts the effect of ensembling E times on the learning dynamics as we show in App. H. The main reason to prefer increasing N rather than increasing E is that larger N has lower bias in the dynamics, whereas ensembling only reduces variance."

Key Insights Distilled From

A Dynamical Model of Neural Scaling Laws

by Blake Bordel... at arxiv.org 04-15-2024

https://arxiv.org/pdf/2402.01092.pdf

Deeper Inquiries

How would the scaling laws and compute-optimal strategy change if the random projection matrix A had a more structured form, rather than iid Gaussian entries

If the random projection matrix A had a more structured form instead of iid Gaussian entries, it would significantly impact the scaling laws and the compute-optimal strategy. The choice of the structure of A would directly influence the eigenvalues and eigenvectors of the resulting model. For example, if A had a structured form that aligned with the underlying data distribution or task, it could lead to more efficient learning and better generalization. This alignment could potentially result in faster convergence, improved performance, and different scaling behaviors compared to the random Gaussian matrix.

Can the theory be extended to capture the effects of architectural choices, such as depth, skip connections, or normalization layers, on the observed scaling laws

The theory presented in the context can be extended to capture the effects of architectural choices on the observed scaling laws. Architectural choices such as depth, skip connections, or normalization layers can have a significant impact on the learning dynamics and generalization of deep learning systems. By incorporating these architectural elements into the model, the theory can provide insights into how different network architectures affect the scaling laws, convergence rates, and generalization performance. Analyzing the interactions between architectural choices and the learning dynamics can offer valuable guidance for designing more efficient and effective deep learning systems.

What are the implications of the theory's predictions for the design of efficient deep learning systems, particularly in resource-constrained settings

The implications of the theory's predictions for the design of efficient deep learning systems, especially in resource-constrained settings, are profound. By understanding the scaling laws and compute-optimal strategies, researchers and practitioners can make informed decisions when designing neural networks. In resource-constrained settings, the theory can guide the allocation of computational resources between model size and training time to maximize performance. Additionally, the theory can help in optimizing architectural choices to improve efficiency and effectiveness, leading to the development of more resource-efficient deep learning systems. Overall, the theory's predictions can inform the design of deep learning models that achieve better performance with limited resources.