Most AI products I review still run on a single model. The team compared a few candidates, picked the one that scored best, and now sends it everything: the password reset, the contract review, the sentiment label. That choice is easy to defend and gets more expensive as traffic grows, because many of those requests could be handled by plain code, a cached answer or a model that costs a tenth as much. Routing sends each request to the cheapest component that can finish it.
The catch is the word "finish." A route is judged on cost per resolved task, and that number is only as honest as your definition of resolved. Intercom's Fin agent shows how easily the definition slips: if a customer goes quiet for 24 hours after Fin's last reply, Fin counts an "assumed resolution" and bills $0.99, whether the customer was helped or gave up.
Intercom is open about this; its help center advises customers to set escalation rules so frustrated users don't trigger the charge. The vendor wrote the definition, and the definition sets the bill.
The price per token doesn't predict the bill
A token price covers one call. Finishing a task usually takes several, and reasoning models bill their thinking at output rates, several times the input rate.
Artificial Analysis, which measures what each model costs to finish the tasks in its Intelligence Index, shows how far the two numbers drift apart. In its September launch analysis, Claude Sonnet 5.5 cost about $7.60 per task at maximum effort, using around 193,000 output tokens per task, plus input. At low effort, the same model at the same $2 and $10 per million tokens reportedly cost about $0.41 per task. A request that needed low effort and got max paid more than ten times as much, so effort level is a routing decision too.
Benchmarks still leave out the largest cost in most production systems I see: the person who checks, fixes or redoes the model's work.
Write down what "done" means
Cost per resolved task = (model spend on every attempt + routing and evaluation overhead + human time) ÷ tasks that met your definition of done
The numerator comes from invoices and logs. Someone has to decide the denominator, and if nobody inside the company does, a vendor contract or a dashboard default will.
A useful definition is checked after the fact: a ticket that stays closed for seven days, a pull request nobody has reverted after 30 days, an extracted record that passes validation downstream. Each workflow gets its own, and the system has to be able to fail it.
A bar like that will stop projects you want to ship. My team paused an internal agent that screened incoming contracts against legal criteria. Moving from one model to an LLM council, with several models checking each other, improved accuracy, though the error rate on legal text stayed too high. A looser definition of done would have let it ship, and on a cost report it would have looked cheap.
That looseness is what makes routing expensive. Move traffic to a cheaper model, count its misses as resolved, and the dashboard shows savings. The customers it failed come back as second tickets, and nobody charges that work to the route.
Many requests don't need a model
I think of routing as a ladder, and each request should stop at the lowest rung that resolves it:
Deterministic code: rules, lookups and validation for anything with a known answer.
A cached or retrieved answer for questions the system has seen before.
A small model for narrow steps whose output is easy to check.
A frontier model for planning, ambiguous input and diagnosis.
A person, who starts from the model's work instead of a blank ticket.
The ladder needs a rule for moving up. The most practical one I know is a check the system can run on every answer: the output parses against its schema, the tests pass, or a cheap verifier model agrees. A request starts low and climbs only when the check fails. Classifying requests upfront works when request types are easy to tell apart, but a misrouted request then has no way back. A cascade pays for failed first attempts, so the check has to be cheap and the first rung has to succeed most of the time.
For the upper rungs, my rule is to use expensive models to think, not to type. On a cloud migration, a reasoning model drafts the plan, engineers review it, cheaper models implement the clearly specified steps, and the reasoning model returns for the ambiguous parts. On a migration my team ran that way, work estimated at 130 to 140 engineering hours took roughly 60 to 70. Human time, the largest line in the formula, fell by about half.
What routing costs, and what it saves
The first hidden cost is the prompt cache. Most production prompts repeat a long prefix on every call: the system prompt, reference documents, the conversation so far. Providers let you cache that prefix and charge far less to reuse it. On Anthropic's API, a cache read on most models costs a tenth of the normal input price, and a standard five-minute cache write costs 1.25 times it. Each cache belongs to one model, so a router that moves a long conversation between models pays the write fee again and loses the discount. Check your cache hit rate before adding a second model; a cache miss is a billing event.
Published router savings depend on the test. RouteLLM, from LMSYS, cut cost by more than 85% on MT Bench and 35% on the GSM8K math benchmark while keeping 95% of GPT-4's quality. LLMRouterBench, published in Findings of ACL 2026, ran 33 models across more than 400,000 test cases and found that several recent routers, commercial ones among them, failed to beat a simple baseline with any consistency. Rules go first, and a learned router has to beat them on your own traffic.
Each route also needs its own test set, rerun whenever a prompt or a model changes. Provider updates move cost as well as quality: when Anthropic released Claude Opus 4.7 in April 2026, the price stayed at $5 and $25 per million tokens, while a new tokenizer meant the same input could count as up to 1.35 times as many tokens. Before you switch a route, replay a recent slice of real traffic through it or run it in shadow next to the current setup, and compare cost per resolved task on your own requests.
Where to start
For a team running everything on one model today, I would go in this order:
Define what "resolved" means for one workflow, including how long you wait before believing it.
Log the full cost of each task (every attempt, cache read and write, and reviewer minute) tagged with its route, and calculate cost per resolved task before changing anything.
Add rules, validation and caching before adding a second model.
Route one category of requests at a time, and keep a slice of traffic on the old setup so you can tell savings from drift.
Token prices will keep falling, and vendors will keep advertising them. The figure to bring to a CFO is the cost of a task your own team agreed was finished, and a customer's 24 hours of silence doesn't count. Routing is one way to lower that figure. Prompt caching and faster human review are others, and each should be judged by the same number.
