Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

TF-IDF is a feature extracted from the data, much like a simple count of words, but it is not learned. It is simply computed. An example of learned features are word embeddings where it is necessary to train on data to obtain them.

If you want to apply machine learning to search then you need clickstream data, embeddings, or learned feature weights.



Word embeddings are also "simply computed." If you use GloVe, then the vectors are obtained by factoring a matrix of co-occurence counts.

The difference between machine learning and "simple" feature extraction is mostly just in the choice of metaphors used to describe the computation, not in any fundamental properties.


Right. Naive Bayes is considered to be a machine learning algorithm, but also consists of just "simple counting".


Your distinction between TFIDF as simply computed vs embeddings as learned is odd and artificial. Both are computations from data, but TFIDF has an understandable closed form and word embeddings do not. As for machine learning, it has to do with improvement and doesn't even necessarily need data.


I think you’re right. It’s the improvement in the learning process that’s the important bit. TF-IDF lacks that.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: