I find the nomenclature in this article a bit weird.
> Another disadvantage of backpropagation is its tendency to become stuck in the local minima of the loss function. Mathematically, the goal in training a model is converging on the global minimum, the point in the loss function where the model has optimized its ability to make predictions.
"Backpropagation" is the method how to compute the gradient of the weights with respect to a loss function. But the article repeatedly uses the term as if it was the whole optimization algorithm, running into local minima.
Wikipedia: "The term backpropagation strictly refers only to the algorithm for computing the gradient, not how the gradient is used; however, the term is often used loosely to refer to the entire learning algorithm, including how the gradient is used, such as by stochastic gradient descent."
As someone who uses this in their day job, I have no problem using loose terminology to describe a well known procedure. Most of the time when I am referring to the act of optimization, it doesn’t matter exactly what method I am using, and I can use backprop as a stand in. If I’m talking about the technical details of my work, I will state the specific optimization strategy. Everyone on my team does similar things, and no one is confused or misled. Use rigorous language when necessary, and use colloquial language when appropriate.
> Use rigorous language when necessary, and use colloquial language when appropriate.
Do you think a peer-reviewd publication is formal enough to warrant precise language? These publications are not only read by specialists in the field. I use automatic differentiation in my daily work, but I'm not familiar with machine learning. Thus I am very confused when "backpropagation" is used to mean an optimization algorithm.
EDIT: It is as if physicists used the term "special relativity" to talk about "quantum mechanics" because, after all, quantum mechanics happens in Lorentzian spacetime. Now for specialists of quantum physics it may make sense, since they are using "special relativity" to distinguish it from fancier quantum theories that combine field theory with GR. But for normal people it would be certainly misleading. Using "backpropagation" to include optimization has the same feeling.
That's bad. There are gradient descent techniques that are feed-forward, and understand the domains where those are appropriate and the domains where backprop is appropriate is I think important, especially as you try to do things like mix machine learning techniques with other differentiable programming strategies.
> Another disadvantage of backpropagation is its tendency to become stuck in the local minima of the loss function. Mathematically, the goal in training a model is converging on the global minimum, the point in the loss function where the model has optimized its ability to make predictions.
"Backpropagation" is the method how to compute the gradient of the weights with respect to a loss function. But the article repeatedly uses the term as if it was the whole optimization algorithm, running into local minima.