> _"they obviously are not smart enough to import tensorflow as tf"_
Yeah. This is part of why DistilBERT (and the fact that you can do pretty well without BERT) is interesting, to me. It seems like for a very long time there have been people complaining about certain individuals and orgs throwing cash/compute at problems to look good rather than solve anything. The difference is nowadays it's starting to be less of a fringe view, thank heavens.
The most interesting thing about NLP (as someone who works in it), is precisely that it is very, very hard to get anywhere. And that in turn is why the field keeps turning up so many new NN designs: the flipside the author rightly points out is that this has to happen for data as well, if we aren't to fool ourselves about our progress.
> The difference is nowadays it's starting to be less of a fringe view, thank heavens.
Well put. The timing is good too, because NLU-is-just-around-the-corner hype is starting to have some really negative social consequences. Yesterday a disturbing article about automated test scoring in the states was trending on HN.
Agreed that understanding lags behind when money and cheaper computation (god I hate that term "compute" used as a verb, learn to English my dudes) drives the SOTA rather than actual understanding (on both researchers' and computers' part).
There's a reason we haven't seen a hard takeoff with BERT writing BERT+1 in even better Python and it probably isn't because nobody is trying, it's because BERT is too stupid to write a computer program.
Yeah. This is part of why DistilBERT (and the fact that you can do pretty well without BERT) is interesting, to me. It seems like for a very long time there have been people complaining about certain individuals and orgs throwing cash/compute at problems to look good rather than solve anything. The difference is nowadays it's starting to be less of a fringe view, thank heavens.
The most interesting thing about NLP (as someone who works in it), is precisely that it is very, very hard to get anywhere. And that in turn is why the field keeps turning up so many new NN designs: the flipside the author rightly points out is that this has to happen for data as well, if we aren't to fool ourselves about our progress.
Great read.