Right now, we are planning on adding more metrics in the future. We also have considered adding the ability of using vector embeddings instead of LLMs for analyzing the responses (which should bring some efficiency gains). Plus, we might add the ability to analyze the distribution of vectors in your vector db to compare it to the distribution of vectors for user queries.
We also have a UI for visualizing all of these metrics that is free at the moment with no paid tier at all. We are planning to add more features into that UI for better visualizations along with improving the ability to keep track of different versions of your LLM system for comparison.
This reminds me of the very cool research of Chris Harrison, a professor at Carnegie Mellon. Lots of interesting work thinking about different ways to interact with computers. http://www.chrisharrison.net/index.php/Research/Welcome
Some fairly widely-used techniques include LSI, LDA, and word2vec or doc2vec. There a lot of different techniques out there! I'm one of the creators of Tagger News, and we used LDA with python's Gensim package. Here's a good tutorial: https://radimrehurek.com/gensim/tut2.html
Blaze has already spent alot of time on this usecase and seems quite promising for a whole host of other reasons. Perhaps it would be best to combine forces?
The idea of putting the column names into the namespace is very interesting, and blaze looks pretty promising overall as a way to connect to big data sources.
