If you train a model for too long, it may overfit it's training data. Not surprising, this has been know for like forever. But did you know you can detect the signatures of overfitting in the layer weight matrices directly, without needing access to any data (train or test) ?
In our recent paper (with hari kishan prakash ), - : - , we show this explicitly in 2 different classic grokking experiments. And the overfitting we see is very different from what has been seen before!
Right. If the dynamics of training are governed by RG flow, then the best optimization path should remove redundant directions, as specified by the RG operator(s)
We present a SemiEmpirical Theory of Learning (SETOL) that explains the remarkable performance of State-Of-The-Art (SOTA) Neural Networks (NNs). We provide a formal explanation of the origin of the fundamental quantities in the phenomenological theory of Heavy-Tailed Self-Regularization (HTSR): the heavy-tailed power-law layer quality metrics, alpha and alpha-hat. In prior work, these metrics have been shown to predict trends in the test accuracies of pretrained SOTA NN models, importantly, without needing access to either testing or training data. Our SETOL uses techniques from statistical mechanics as well as advanced methods from random matrix theory and quantum chemistry. The derivation suggests new mathematical preconditions for ideal learning, including a new metric, ERG, which is equivalent to applying a single step of the Wilson Exact Renormalization Group. We test the assumptions and predictions of SETOL on a simple 3-layer multilayer perceptron (MLP), demonstrating excellent agreement with the key theoretical assumptions. For SOTA NN models, we show how to estimate the individual layer qualities of a trained NN by simply computing the empirical spectral density (ESD) of the layer weight matrices and plugging this ESD into our SETOL formulas. Notably, we examine the performance of the HTSR alpha and the SETOL ERG layer quality metrics, and find that they align remarkably well, both on our MLP and on SOTA NNs.
SETOL is a theory of NN layer convergence. It argues that the individual layers of NN converge at different rates, and the 'Ideal' state of convergence can be detected simply by looking at the spectral properties of the layer weight matrices.
SETOL derives the weightwatcher HTSR layer quality metrics (alpha, alpha-hat) from first principles using techniques from statistical mechanics and quantum chemistry.
SETOL also shows that when an NN layer is 'Ideal', it satisfies the so-called Wilson Exact Renormalization Group condition (called the TraceLog condition)
In other words, SETOL provides empirical layer quality metrics that can be used to determine how well a model is trained or fine-tuned. It can help AI engineers get their AI models to the best state possible.
And while this work uses techniques from theoretical physics and chemistry, you don't need know any mathematical physics to read the paper; it is fully self-contained.
Most importantly, all of the experiments are 100% reproducible, and you can test the theory yourself on your own AI models using the open-source weightwatcher tool.
WeightWatcher (w|w) is an open-source, diagnostic tool for analyzing Deep Neural Networks (DNN), without needing access to training or even test data. It is based on theoretical research into Why Deep Learning Works and uses the Theory of Heavy-Tailed Self-Regularization (HT-SR), published in JMLR and Nature.
In our recent paper (with hari kishan prakash ), - : - , we show this explicitly in 2 different classic grokking experiments. And the overfitting we see is very different from what has been seen before!
paper: https://arxiv.org/abs/2602.02859