The hard part of model routing isn’t the router
The hard part of model routing isn’t the router
Marc Gerard
·
Product Lead
TLDR
Model routing is not mainly a routing problem—it is a task-to-model fit problem. Companies often send every request to the same LLM model, even when cheaper models could handle many tasks just as well. The real challenge is understanding what each task requires, defining the quality bar, and identifying the least expensive model that can consistently meet it. Routers execute those decisions; the valuable intelligence layer is knowing which model should handle which work in the first place. Nebuly Routing Intelligence is being built to solve this by identifying recurring tasks, evaluating task-to-model fit, and recommending the lowest-cost model that can consistently meet the required quality bar.
Your CFO asks why Claude Opus spend doubled last month. Your dashboard can show which application and API key spent the money. It cannot tell you whether the work actually needed Opus.
That distinction is becoming one of the most important questions in AI infrastructure.
Model selection is no longer a technical detail buried inside an application. It is a resource-allocation decision made thousands—or millions—of times each month. Yet most companies still treat it as a static configuration: choose a capable model, connect it to the product and let every request flow through it.
That was a reasonable way to ship the first version. It is an expensive way to operate the tenth.
The safest default is becoming the most expensive one
Today, the same frontier model may analyze a complex codebase, summarize a meeting transcript, extract a company name and classify a support ticket.
The outputs may all be excellent. That does not make the allocation efficient.
When an expensive model performs work that a smaller model could complete just as well, the waste is almost invisible. There is no outage, error message or obvious quality problem. The system works exactly as intended; the company simply pays more than it needed to.
The price difference can become material quickly. At the time of writing, Anthropic charges $1 per million input tokens and $5 per million output tokens for Claude Haiku 4.5. Claude Opus 4.8 costs $5 and $25 respectively—five times more for the same token volume.
For one million monthly requests averaging 1,000 input tokens and 500 output tokens, that is approximately $3,500 on Haiku and $17,500 on Opus. The difference is $168,000 per year, before caching, discounts and other pricing adjustments.
That premium may be entirely justified for difficult reasoning, complex coding or high-stakes analysis. It is harder to defend when a large share of the workload consists of classification, extraction, formatting or routine summaries.
The problem is not that the company is using too much AI. It is that the company has no reliable way to determine how much intelligence each task requires.
Routers solved execution, not judgment
The case for model routing is already well established.
Stanford’s FrugalGPT showed that model cascades could match the performance of the strongest individual model with cost reductions of up to 98% on the workloads it evaluated. RouteLLM later demonstrated that simpler requests could be sent to cheaper models while stronger models were reserved for harder queries, reporting savings of up to 85% while retaining 95% of GPT-4 benchmark performance.
Those numbers should not be treated as a promise. Production savings depend on the workload, the models being compared and the quality standard the product must maintain.
But the underlying conclusion is difficult to dispute:
The right model is not the same for every request.
The infrastructure is rapidly catching up with that insight. OpenRouter’s Auto Router, for example, analyzes a prompt and selects from a pool of models based on factors such as task type, complexity, capability and the desired cost-quality trade-off.
This makes routing easier to execute. It does not remove the need for a routing strategy.
A router can choose between models. It cannot automatically know what your company considers an acceptable result, how much an error costs, whether a response resolved the user’s task or which workflows require a higher standard than their prompts suggest.
The hard part is not sending traffic to another model. The hard part is deciding, with evidence, that the traffic belongs there.
The unit of AI economics is the task
Most AI cost dashboards are organized around providers, models, tokens, applications and API keys. These dimensions explain where the money went, but not whether it was well spent.
The missing dimension is the task.
For every recurring workload, teams should be able to answer four questions:
What was the model being asked to accomplish?
Which model handled the task?
Did the result meet the required quality standard?
Could another model have achieved the same outcome for less?
We think of this as task-to-model fit: the degree to which the capability and cost of a model match the actual requirements of the work.
A good routing policy therefore does not ask, “Which model costs less?”
It asks:
What is the least expensive model that consistently clears the quality bar for this task?
That quality bar cannot come from a generic leaderboard alone. It must reflect the organization’s own prompts, data, users and consequences of failure. A short internal summary and a regulated customer response may look similar at the prompt level while requiring very different standards.
From model observability to model intelligence
This is the problem we are building Nebuly around.
The customer signal is clear: teams do not only want to compare how much they spend on Claude, Gemini, ChatGPT or an internal model. They want to understand which kinds of work belong on each one.
Nebuly Intelligence is being developed to analyze real AI interactions, identify recurring tasks and connect those tasks with the models, costs and outcome signals behind them. The purpose is to recommend the best-fit model for each workload and estimate what the organization could save while holding the required output quality constant.
That final condition matters.
A recommendation to move a workflow to a lighter model should be treated as a testable hypothesis, not an automatic downgrade. Representative historical interactions can be replayed against the alternative model. Deterministic checks, human review, user outcomes or calibrated model-based evaluations can then determine whether the quality threshold is preserved.
Suppose a company discovers that high-volume summarization represents a significant share of frontier-model spend. If a smaller model produces equivalent results on representative examples, the potential saving can be calculated at current traffic levels. If it fails the quality test, the workload stays where it is and no saving should be claimed.
The goal is not to reduce model quality. It is to stop paying for capability that does not improve the result.
Routing policies should be managed like a portfolio
For AI platform teams, this requires a different operating model.
First, instrument work at the task level rather than relying only on model and token totals. “Opus consumed 500 million tokens” is an accounting fact. “Opus consumed 500 million tokens summarizing internal documents” is an optimization opportunity.
Second, measure quality by task and model. A global model score hides the variation that routing depends on. One lightweight model may perform extremely well on extraction and poorly on nuanced writing; another may show the opposite pattern.
Third, track potential savings at constant quality. Cost reduction without a quality constraint is easy but dangerous. The useful metric is how much spend can be removed while keeping the outcome above the required threshold.
Finally, treat routing as a continuous decision rather than a one-time configuration. Models improve, prices change, prompts evolve and the mix of user requests shifts. A model that is the best fit today may not remain the best fit six months from now.
The routing policy should move with the market.
Intelligence should be allocated, not standardized
Frontier models will continue to matter. As their capabilities improve, companies will find increasingly valuable problems for them to solve.
The mistake is standardizing frontier-level intelligence across every problem simply because it is the safest default.
AI cost optimization is not about using weaker models. It is about buying the right amount of intelligence for each task.
The router can execute that decision. The valuable layer is knowing what the decision should be.
That is the layer we are developing at Nebuly, and we will share what we learn as we validate it against real workloads.
Nebuly
Nebuly is the ROI platform for enterprise AI. It connects to the AI agents your business runs on, the assistants your customers interact with, and the tools your employees use every day, including Claude, ChatGPT, and Copilot, and translates that activity into business value. How much time is being saved across teams. What revenue your AI is influencing. What adoption and AI proficiency look like in practice, across departments and geographies. All aggregated at the organizational level, never tied to individuals.
If you need clarity on what your AI investment is actually delivering, book a demo.
FAQs
What is task-to-model fit?
The degree to which a model's capability and cost match what a task actually requires. It lets teams route each recurring task to the least expensive model that consistently clears its quality bar.
Why does running one frontier model for everything cost more?
When an expensive model does work a smaller model could do just as well, there is no outage or error to signal it. The system looks fine while the bill runs higher than the work required. Across one million monthly requests, the gap between Claude Haiku 4.5 and Claude Opus 4.8 is roughly $168,000 a year.
Can a router decide which model to use on its own?
A router can send traffic to another model, but it cannot know your acceptable result, the cost of an error, or which workflows need a higher standard than their prompts suggest. Routing solves execution. Deciding which tasks belong where is a separate layer.
How much can model routing save?
Stanford's FrugalGPT reported up to 98% cost reduction on the workloads it evaluated; RouteLLM reported up to 85% savings while keeping 95% of GPT-4 benchmark performance. Real savings depend on your workload, the models compared, and the quality bar you hold.
How do you move a task to a cheaper model without losing quality?
Treat it as a testable hypothesis. Replay representative interactions against the lighter model, then confirm quality with deterministic checks, human review, or calibrated evaluations. If it passes, calculate the saving at current traffic. If it fails, the task stays where it is.
What is Nebuly Routing Intelligence?
It analyzes real AI agent interactions, identifies recurring tasks, and connects them to the models, costs, and outcomes behind them. It recommends the best-fit model for each workload and estimates the saving while holding output quality constant.
Stay up to date on what we're learning, building, and seeing as enterprise teams deploy and measure AI agents in production.
