>By all accounts, deep neural networks like VGG have way too many parameters and should overfit.
I keep seeing this claim pop up but I've never seen a citation to support it. My understanding is that e.g. statistical learning theory lets you prove theorems of the form "IF it has this number of parameters or fewer, THEN it won't overfit". First year logic is enough to know this theorem tells you nothing in the scenario where the model has lots of parameters. Can anyone provide a solid citation to support the claim that models with many parameters "should" overfit?
BTW, another example of a model with lots of parameters which doesn't overfit is random forests / gradient boosting. I'm wondering if this "should overfit" claim is less of a global property of machine learning models in general, and more just a property of the particular models that this set of researchers have most familiarity with (SVMs).
Yes usual results in statistical learning theory are upper bound on the generalization error, so if you have an "overparametrized" (number of parameters is actually a useless metric but that's another discussion altogether) then the best you can say is "well I expect the error to be at most A FREAKING LOT", which is not a very interesting result.
I keep seeing this claim pop up but I've never seen a citation to support it. My understanding is that e.g. statistical learning theory lets you prove theorems of the form "IF it has this number of parameters or fewer, THEN it won't overfit". First year logic is enough to know this theorem tells you nothing in the scenario where the model has lots of parameters. Can anyone provide a solid citation to support the claim that models with many parameters "should" overfit?
BTW, another example of a model with lots of parameters which doesn't overfit is random forests / gradient boosting. I'm wondering if this "should overfit" claim is less of a global property of machine learning models in general, and more just a property of the particular models that this set of researchers have most familiarity with (SVMs).