I spent a lot of time researching LLM routing last year and also came to the conclusion that it's generally not worth the effort. It's too hard to understand the difficulty of a query a priori.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.
I think routing should be pushed 'down the stack' so to speak.
What I mean is that most tasks can be recursively fragmented into smaller tasks, and once you've hit suitable leaf nodes -- where the task is very granular -- you can begin to deterministically show which models perform better or worse for that specific task. Then, when your agent is running a workflow, you may use various models for different steps in a workflow. For example, some models may excel at exploration, some at determining a good architectural fit for an implementation, some at actually writing the implementation, and so on.
But you don't know until you define your 'work' taxonomy, and still further, you won't know until you have a statistically significant number of runs on a given chunk of work. Once you have that, though, you can hone in on models that excel at one specific task or another and prefer those the majority of the time (say ~80%) and hold back the remainder work as a 'test corpus' just in case a different or new model does even better.
This is something I've kept in the back of my head as I've been working through my agent harness primitives -- specifically enabling different models per chunk of work.
this idea of smaller tasks is me simply asking "quick question, foo?" to my fable 5 on max effort, the model just does it, and even gives a more complex answer on misunderstanding or a actually hard question
You also have problems with short prompts whose results depend highly on the understanding of nuance. The router is going to have to mostly solve the prompt to decide which model to send it to.
What you actually want is a model that can conclude either "I know the answer to this with confidence" and answer, or "I think I don't know the answer to this, I should ask another model and I know which one". But I don't think LLMs can really bring their uncertainty to the surface in that way yet? Their internal confidence can be measured and returned, so you could probably front a more powerful model with a knowledgeable assistant, but they can't consciously mark their own homework?
> Their internal confidence can be measured and returned
It can? I was under the impression that confidence was either self-reported by the LLM or assessed by having another model interpret the output response. If there's a confidence score at the level of the actual model math, that's news to me.
You certainly can't ask an LLM for its confidence level because it will generate that answer as it sees fit.
My understanding was that models can be post-trained to assess their own confidence on short prompts within a level of accuracy (there was an article about this a few weeks ago that I can't find), but can't use it to reason, and that research has shown that internally they effectively have a measurable sense of truth but can't surface it:
I mangled what I was trying to say but the point I guess is, it is effectively in there, and researchers can see it and therefore score it, but it is not something the LLM can use.
The problem seems to me (layman's understanding at best) that the LLM is inherently confident within the words of its answer, because of what the model is trained to do and how it is trained. So you can never get an accurate "I don't know this" from a model while it is answering; it is bullshitting.
(One of the things that is most interesting to me at the moment is asking a small local LLM what it knows about a topic and then testing it. It can have no idea of what it wasn't trained on, but any question about what it knows is seemingly going to trigger it to work through all the summary word associations it did find in training, so it can sort of summarise its knowledge that way, with some likelihood of success, without any sense of introspection)
Ahh you found the link I couldn't find with a search.
Yes — they claim that it is likely to be able to assess confidence in whether the statement it just made is correct. But it isn't going to be capable of that while it is answering.
How does one spend a lot of time researching model routing? Did you evaluate different routing providers? Maybe you could build a rorouterter that lets you easily switch to any router, ideally if there's a vuln in any of them you can get infected even if you don't use it.
I mean can't you just directly test some variations of the query against many models at once and pick the cheapest model that hits your accuracy goal? If it's a one time run then ofc this is all pointless, but for ongoing tasks it makes sense. This seems more logical to me than making another AI model of some sorts intuit the right LLM for the job.
If the entire universe were turned into a giant computer and did nothing but generate uuids until its heat death, how many bits would you need for the ID space?
"But are you worried that every human on Earth will be hit by a meteorite right now? That probability is also non-zero, yet it is so infinitesimally small that we treat it as an impossibility."
This might be a bad example because one meteorite could take out the world and given enough time is likely to.
They have 6000 employees located in Ireland including most of their support staff for the EU. They just opened another office in Dublin. They are also there for 45 years already, so not sure where "just a po box" comes from.
The issue wasn't that the CEO was missing when they dropped by the Panama address. Rather it was that the (supposed) registered agent didn't know who they were.
If I say "contact my lawyer" and you show up to his office and the receptionist informs you that I'm not a client ...
So what? A registered agent is literally the agent registered to accept process service. The registered agent is clearly not the corporate headquarters, a branch office, or anything other than a business whose purpose is to accept lawsuits, subpoenas, and other legal and official notices.
Well, that's the same thing the NPR article is reporting on, which is the equivalent legal concept in Panamanian law.
Polymarket's actual offices are remote-first, but they have a small space set up in New York City which is the closest thing they have to an HQ. It's basically set up like a coworking / hotel sort of space.
It's an interesting "problem". The cities we have now exist because businesses and people want to be located in the same geographical area to maximize, well, doing business.
Now the opposite is happening. Businesses have no incentive being located in the same physical area they do business in. In fact, they have opposite incentives. The closer they are to their customers and workers, the less they can do things with impunity.
The modern world is like Jean Baudrillard's vision of hell. Back in 1991 he wrote "The Gulf War did not take place", commenting on what was at the time a new development of 24/7 live media coverage of the war. Media saturation created a hyperreality where images about the war replaced the thing itself. How far we have come. We are so complacent here that war exists only as stream of symbols and sounds streaming out of our screen. I think many do not truly believe it is real.
So far only the Nasdaq-100 has gone along with SpaceX's special weighting chicanery. The biggest fund which tracks the Nasdaq-100 is QQQ. Suffice to say, if you have money in QQQ you should be re-considering that position.
Either way, no, high frequency trading firms are going to beat you to the punch. And shorting elons other company, just because it's over valued by traditional metrics, didn't work out that great for most traders.
This is a slightly tongue-in-cheek way of saying that if you believe a security is severely mispriced then there is a straightforward way to express that opinion.
Shorting is really not that straightforward. It is a avery advanced topic because it mandates the use of leverage. Many (most) investors are long-only, especially the ones being taken advantage of here.
> especially the ones being taken advantage of here.
This is a great argument why buying an index is a poor choice for a long term investor. You can avoid a great deal of shenanigans by randomly purchasing stocks and holding them for 50 years. Even a 0.02% annual fee costs you 1% of your long term returns over that timescale.
Index investing is a great choice for a long term investor who cares about simplicity, which should be the vast majority of them. Actually the best thing about holding individual stocks is probably the increased opportunities for TLH, but the nightmare of holding and managing hundreds of securities in your account is very seldom worth what you save on fees or deferred taxes.
You still owe the taxes on dividends generated by an index fund that automatically reinvests dividends. There’s ways around this like a 401k, but you can also own individual stocks inside a 401k or rollover to an account that lets you do so.
I don't know how that's a response to what I wrote. I'm not saying that an ETF is more tax efficient, although it is more tax efficient than a mutual fund. If you are worried about the tax drag of dividends then there are ETFs that seek to track spx, but without receiving dividends, by selling and rebuying around ex div dates. They're not established enough to trust with a lot of money, imo, but maybe it will become more normal one day.
Tesla has been this for a long time since the price fundamentally does not reflect the quality or future of the company, yet the road is littered with dead Tesla bears.
Efficient markets hypothesis breaks down with Musk companies.
One way to think about this is you have a binomial distribution with p=0.8 and n=number of lying friends. Each time you increase n, you shift the probability mass of the distribution "to the right" but if n is even some of that mass has to land on the "tie" condition.
There are a few markets like Jacksonville and Atlanta where there is a lot of institutional ownership, but outside of those few cases impact of Blackrock et al on housing markets is effectively nil.
To summarize the article: buying the Yes side of this market is like shorting treasuries. It's not a bet that treasuries will default, but rather a macro bet about that demand for cash (i.e. interest rates) will increase.
One specific challenge I was seeing is that difficulty depends a lot on what information is retrievable by the agent. Consider the question "what is the 5-state busy beaver number?" (https://en.wikipedia.org/wiki/Busy_beaver). In 2023 this would be a Mythos-tier research problem, but a solution was proved in 2024 so today any minimally intelligent model with a web search tool can just fetch the answer. You don't know which queries will be basic summarization and which will be deep reasoning until you get going.