This piece first appeared in German. The English version is a rewrite rather than a line-by-line translation, and the German original stays online in the archive: Warum überparametrisierte neuronale Netzwerke nicht Overfitten.

In 2019 Belkin, Hsu, Ma and Mandal published a result that left learning theory without an explanation. Push a model past the point where it fits its training data perfectly, they showed, and the test error starts falling again. The textbook says it should rise, and keep rising.

Modern deep learning models routinely carry more parameters than they have training data points, which puts them in what the field calls the interpolation regime. By classical statistics they ought to fail catastrophically on anything new. They do not, and why they do not is one of the most interesting open questions in the mathematics of learning.

The rule that broke

Picture building a house with a hundred times more columns than the roof needs. Every engineering handbook would call that unstable, inefficient and liable to collapse under its own weight. In modern AI it is the recipe.

Statistics has always followed Occam's razor, which says to take the simplest model that explains the data. Too many parameters produce overfitting, where the model memorizes the training set and fails on new input. It is the student who learns the answers on the problem sheets by heart, never the principle underneath, and then meets an exam with different questions.

What the textbook said

Every statistical model sits between two sources of error. A model that is too simple carries high bias: it underestimates the complexity of the data, a straight line drawn through a curve, and it underfits. Too complex, and the model carries high variance instead: it fits every bit of measurement noise and loses the global trend, passing perfectly through each noisy point and predicting nothing new correctly.

The result is a U-shaped curve. Test error falls as complexity rises, because the model is learning real patterns, and then it climbs again once the model starts reading noise as signal. The textbook prescription was to calibrate complexity to land at the bottom of the U. Not too simple, not too complex.

For decades that was the basis of model selection, regularization and cross-validation. It worked until models grew large enough to cross the line of perfect data fit.

Double descent

Belkin and colleagues showed systematically what practitioners working with deep networks had been observing without a name for it. Beyond the interpolation threshold, the error curve has two valleys and not one. Between them sits a peak, exactly at the threshold, where the model is just large enough to fit the data perfectly and not yet large enough to be flexible. Nakkiran and colleagues confirmed the phenomenon empirically in 2021 in architectures that are actually deployed, among them ResNets and transformers.

What is the interpolation threshold? A model interpolates when it hits every training point exactly and the training error is zero. Below the threshold it cannot fit the data completely. Above it, infinitely many mathematical functions describe the training data perfectly, so the model has freedom in which one to pick. That freedom is the whole story.

When a network is enormous, there is no single perfect solution. There is a space of them. The network does not choose arbitrarily within that space; it chooses a surprisingly smooth, structurally simple one. Why it does that is the question.

The optimizer has a preference

The lead character here is the optimization algorithm rather than the architecture. Stochastic gradient descent is usually described as a purely technical instrument for finding the minimum of a loss function. That description is correct and incomplete. Gradient descent is a filter with a mathematical preference.

Soudry, Hoffer, Nacson, Gunasekar and Srebro proved in 2018 that on separable data, gradient descent steers implicitly toward the minimum-norm solution: the one with the least mathematical complexity, the flattest curvature, the smallest weights. Among all the functions that fit the training data perfectly, it structurally prefers the smoothest.

The geometry of the loss surface makes that visible. Hochreiter and Schmidhuber argued as early as 1997 that the width of a minimum decides generalization. A narrow, deep minimum means a small perturbation in the input, a slightly different test example, produces a large swing in the output. Broad and flat, the minimum absorbs exactly that, and a model that lands there generalizes better.

Picture an enormous terrain of hills where every point on the ground is a different solution that explains all the training data. Gradient descent does not find the deepest, narrowest hole, which would be overfitting. It rolls into the widest, flattest valley, and flat valleys generalize systematically better on new data.

This implicit bias comes from no explicit regularization, from no L2 penalty and no dropout. It is inherent in the algorithm, a by-product of how gradient descent searches the parameter space. Overparameterization is what gives the algorithm room to exercise the preference. A small network has no choice. Given a large one, the algorithm can pick the smoothest solution from among all the perfect ones.

What this means for scaling

The practical consequence is fundamental. Overparameterization is not a bug, it is a feature. More parameters mean more mathematical freedom to find a smooth, stable solution. That explains why the apparently naive scaling recipe of recent years, more data and more parameters and more compute, has worked as reliably as it has.

The limits of the explanation deserve equal weight. Double descent can be observed experimentally and proved mathematically in special cases. No complete, universal theory of deep learning exists today: nothing that accounts for all of generalization behavior, for why particular architectures suit particular tasks, for how data volume and model size should be matched.

We are riding a technology whose mathematical foundation is being built mid-flight. That is not an argument against using it. It is an argument for methodological modesty, and for taking an interest in the questions that are still open.

Sources

Grouped by the section they support.

Opening

  • Belkin, Hsu, Ma and Mandal (2019), Reconciling modern machine-learning practice and the classical bias–variance trade-off, Proceedings of the National Academy of Sciences 116 (32), 15849–15854, doi.org/10.1073/pnas.1903070116, preprint arXiv:1812.11118. Supports the double descent curve, in which raising capacity beyond the point of interpolation improves performance again.

The rule that broke

  • No external source. The house analogy and Occam's razor are the author's framing.

What the textbook said

  • Belkin et al. (2019). Supports the textbook U-shaped bias–variance curve that double descent extends.

Double descent

  • Belkin et al. (2019). Supports the interpolation threshold and the peak in test error around it.
  • Nakkiran, Kaplun, Bansal, Yang, Barak and Sutskever (2020), Deep Double Descent: Where Bigger Models and More Data Hurt, ICLR 2020, arXiv:1912.02292. Supports double descent in ResNets, standard convolutional networks and transformers.

The optimizer has a preference

  • Soudry, Hoffer, Nacson, Gunasekar and Srebro (2018), The Implicit Bias of Gradient Descent on Separable Data, Journal of Machine Learning Research 19 (70), 1–57, jmlr.org. Supports gradient descent on unregularized logistic regression with linearly separable data converging to the direction of the maximum-margin solution. The result is proven for linear predictors.
  • Hochreiter and Schmidhuber (1997), Flat Minima, Neural Computation 9 (1), 1–42, idsia.ch. Supports the link between flat minima, simpler networks and better generalization.
  • Keskar, Mudigere, Nocedal, Smelyanskiy and Tang (2017), On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, ICLR 2017, arXiv:1609.04836. Supports small-batch stochastic gradient descent converging to flat minima that generalize better.
  • Dinh, Pascanu, Bengio and Bengio (2017), Sharp Minima Can Generalize For Deep Nets, ICML 2017, proceedings.mlr.press. Argues against flatness as an explanation for generalization in deep networks. The terrain analogy is an illustration.

What this means for scaling

  • Belkin et al. (2019) and Nakkiran et al. (2020). Support double descent as observed and, in special cases, proven. The point that no complete theory of deep learning exists yet, and the call for methodological modesty, are the author's own.
The link has been copied!