When your application sends a request to an LLM, someone has to decide which model handles it. In a simple setup, that decision is hardcoded: every request goes to the same model at the same provider. In a production system handling thousands of requests per minute across different use cases, that approach leaves performance and money on the table.
Intelligent model routing is the practice of dynamically selecting the best model and provider for each request based on real-time conditions and configurable priorities. This article explains how it works under the hood.
What Is Model Routing?
Model routing is the decision layer that sits between your application and the LLM providers. For each incoming request, the router evaluates the available models and selects the one that best matches the current optimization goals.
The simplest form of routing is a static mapping: "always use Claude Sonnet." The most sophisticated form considers real-time latency measurements, per-token cost differences, quality benchmarks, provider health status, and organizational constraints like budget limits — all evaluated in microseconds before the request is forwarded.
The value of routing increases with the number of models and providers you have access to. With one model, there is nothing to route. With ten models across four providers, the routing decision can meaningfully impact cost, speed, and reliability on every single request.
The Three Dimensions of Routing
Every routing decision involves balancing three competing objectives:
Latency
How fast will this model respond? Latency varies not just between models but between providers of the same model, and it fluctuates throughout the day based on load. A model that responds in 200 milliseconds at 2 AM might take 2 seconds at peak hours.
Optimizing for latency means measuring actual response times continuously and routing to the provider that is currently fastest. This is critical for real-time applications like chatbots, autocomplete, and interactive agents where users are waiting.
Cost
How much will this request cost? Pricing varies dramatically across models and providers. A request that costs $0.001 on a small model might cost $0.05 on a frontier model — a 50x difference. For high-volume workloads, this difference translates directly to thousands of dollars per month.
Optimizing for cost means selecting the cheapest model that can handle the request adequately. This requires understanding the complexity of each request and the capability of each model, which is where quality scoring comes in.
Quality
How good will the response be? Not all models produce equivalent output. Frontier models handle nuanced reasoning, complex instructions, and edge cases better than smaller models. But for straightforward tasks, the quality difference is negligible.
Optimizing for quality means routing complex requests to capable models and simple requests to efficient ones. This is the hardest dimension to score because quality is task-dependent and subjective, but benchmark data and empirical testing provide usable signals.
How EWMA Scoring Works for Latency Tracking
Static latency benchmarks are nearly useless for routing decisions because provider performance changes constantly. What you need is a real-time signal that adapts to current conditions while smoothing out noise from individual request variance.
This is exactly what EWMA — Exponentially Weighted Moving Average — provides.
EWMA calculates a running average of observed latencies where recent measurements carry more weight than older ones. The formula is straightforward:
EWMA_new = alpha * latency_observed + (1 - alpha) * EWMA_previous
The alpha parameter (typically between 0.1 and 0.3) controls how quickly the average adapts to new data. A higher alpha means the score reacts faster to recent changes but is more sensitive to outliers. A lower alpha provides more stability but adapts more slowly.
In practice, the router maintains an EWMA score for each model-provider combination. Every response updates the score. When a routing decision is made, the router compares current EWMA scores to identify which provider is genuinely faster right now — not which was faster an hour ago or which has the best published benchmark.
This approach naturally handles transient slowdowns, gradual degradation, and recovery without any manual intervention or threshold configuration.
Weighted Routing Strategies
In production, you rarely want to optimize for just one dimension. A pure cost optimization would always pick the cheapest model, even when response quality is unacceptable. A pure latency optimization would ignore cost entirely. The solution is weighted routing.
A weighted strategy assigns a priority to each dimension, expressed as a percentage:
| Strategy | Latency | Cost | Quality | Best For |
|---|---|---|---|---|
| Balanced | 40% | 40% | 20% | General workloads |
| Speed-first | 70% | 10% | 20% | Real-time chat, autocomplete |
| Budget | 10% | 70% | 20% | Batch processing, internal tools |
| Quality-first | 10% | 20% | 70% | Customer-facing generation, code |
The router computes a composite score for each available model by normalizing each dimension to a 0-1 scale and applying the weights:
score = (w_latency * latency_score) + (w_cost * cost_score) + (w_quality * quality_score)
The model with the highest composite score wins the request. This happens in microseconds and is fully configurable per project or per API key.
Automatic Failover: Detecting Failures and Rerouting
Provider outages and degradations are not edge cases — they are regular events. Any production system that depends on a single provider without failover is accepting unnecessary downtime risk.
Effective automatic failover has three components:
Detection. The router monitors response codes, latency spikes, and timeout rates for each provider in real time. A single failed request might be a transient error. A pattern of failures — three 500 errors in ten seconds, or latency exceeding 5x the EWMA baseline — triggers a provider health downgrade.
Rerouting. When a provider is marked degraded, the router removes it from the candidate pool and redistributes traffic to healthy alternatives. This happens transparently — your application receives a response from a different provider without any code change or retry logic on your side.
Recovery. The router periodically sends probe requests to degraded providers. When consistent healthy responses return, the provider is gradually reintroduced to the pool. This prevents a recovered provider from being immediately overwhelmed with full traffic and avoids premature re-inclusion after a brief false recovery.
How Router One Implements Intelligent Routing
Router One's routing engine is built around these principles as first-class features, not afterthoughts.
Every request to the unified POST /llm.invoke endpoint is evaluated against the current EWMA scores, cost tables, and quality baselines for all available models. Routing weights are configurable at the project and API key level through the dashboard or API, so different workloads can have different optimization priorities within the same organization.
Failover is automatic and requires zero configuration. The moment a provider degrades, traffic shifts. The moment it recovers, traffic rebalances. The full decision trace — which models were considered, which scores they received, and why the winner was chosen — is logged and visible in the observability dashboard for every request.
This gives you not just intelligent routing, but full transparency into how every routing decision was made.
Start Routing Smarter
If you are sending all your LLM requests to a single model at a single provider, you are paying more than you need to, running slower than you could be, and accepting reliability risk you do not have to. Intelligent routing fixes all three.
Try Router One's routing engine with a free account at router.one. Configure your weights, watch the EWMA scores adapt in real time, and see the difference intelligent routing makes on your first day.