Is this true? My understanding was that in fine tuning, you’d only re train some of the layers. And even if you re train all the layers, the starting point for the layers is not random. If it really was all forgotten then fine tuning would not be orders of magnitude faster...
Gradient decent optimizes performance of a model on a given dataset. If you stop training on one dataset and start training on another one your model will become more optimized for the second dataset and less optimized for the first. This will usually result in degraded performance on classes of data found more commonly in the first dataset but not the second. This is what people mean by "forgetting". It doesn't matter how much of the model you fine-tune, the effect is still present though the effect size varies.
