Here is a quick graph I just generated from data I had. Horizontal axis is average bit depth, vertical is accuracy. PTQ is Post Training Quantisation, QAT is Quantisation Aware Training. It's a simple ResNet trained on CIFAR-10, I don't have the resources to train anything bigger.
If you do it during training you automatically check that the compressed model still do what you want after compression, and can even correct some small issues by continuing the training on the smaller model.
Edit: In general the more a compression function understands of what your goals are the better, so it is naturally advantageous to make the compression function look like training since then it is fully aware of what to optimize for.
