The context window is limited. Using half your context window for tools means you have a 50% smaller context window.
On a large and complex system (not even a mini ERP system or even a basic bookkeeping system, but a small inventory mgmt system) you are going to have a few dozen tools, each with a description of parameters and return values.
For anything like an ERP system you are going to have a few thousands of tools, which probably wouldn't even fit in the context before the user supplied prompt.
This is why the only use case this far for genAI is coding: with a mere 7 tools you can do everything.
> The problem of overflowing context is solved by RAGs, though.
No, it isn't.
It's mitigated with RAGs, but RAGs add to the context, and what they add might be irrelevant is all the retriever module is doing is plain text search.
If the retriever module is performing an embeddings/vector search on a properly prepared dataset you may have more luck, but it's still a piss-poor experience compared to simply putting all the tools into the context.
Of course, I'm not an expert, so I welcome corrections.
On a large and complex system (not even a mini ERP system or even a basic bookkeeping system, but a small inventory mgmt system) you are going to have a few dozen tools, each with a description of parameters and return values.
For anything like an ERP system you are going to have a few thousands of tools, which probably wouldn't even fit in the context before the user supplied prompt.
This is why the only use case this far for genAI is coding: with a mere 7 tools you can do everything.