What is an LLM router?
Last updated 2026-08-19
An LLM router is a layer that chooses which model serves each request instead of sending everything to one hardcoded model. The goal is economic: easy requests move to cheaper models, hard ones keep the frontier, and the bill drops without a quality drop. Routers differ in one load-bearing way: what authorizes a substitution. It is either a heuristic, a learned prediction, rules you wrote, or, on Finest, a sealed and published evidence record per task class, with everything unproven served exactly as you requested it.
What a router actually does
Every router performs three steps. It classifies the request, by task type, difficulty, or learned features. It selects a configuration, meaning a model plus the settings that change behavior: decoding, effort, caching mode. And it commits, serving the request and standing behind the result, or not.
The third step separates products. A router that cannot say what happens when the cheap model fails has not finished the design. The complete answer includes escalation, meaning a refusal or a failed machine-checkable validation re-serves on the requested model, and a record of what ran, so the decision can be audited afterward.
Selection is also the narrowest form of the idea. The Request Compiler, the published method behind Finest, treats a router as a compiler for requests and admits five execution strategies: route, cascade, ensemble, plan, and passthrough, where nothing is proven, the requested model serves, and the fee is zero. Every strategy shares one floor: a loud failure escalates to the requested model, and the customer is debited no more than that model alone would have cost.
The four ways a substitution gets authorized
The first three trade safety for coverage in different proportions. The fourth trades coverage for safety: routing exists only where evidence exists, and grows at the speed of measurement. Which trade is right depends on what a wrong substitution costs you, and in production that cost is rarely small.
- Heuristics
- Price and availability presets. Cheap to run, blind to your quality bar.
- Learned prediction
- A trained router guesses per prompt which model suffices. Adaptive everywhere, auditable nowhere in advance.
- Rules you author
- Explicit configs your team writes and maintains. Full control, and the evaluation burden stays with you.
- Sealed evidence
- A configuration serves a task class only after clearing a pre-registered bar on a sealed corpus, with the record published. Finest routes this way, and serves the requested model verbatim outside it.
Questions to ask any router
What authorizes a substitution, and can I read that authorization before trusting it? What happens on refusal or validation failure? What does traffic outside proven coverage do? What does a routing decision look like after the fact, per request? And what do I pay when routing saves me nothing? Finest's answers: published evidence bars; escalation to your requested model; verbatim serving; a receipt naming model, evidence, and saving; and nothing.
Questions people ask
- Do LLM routers reduce quality?
- A router without pre-registered quality bars and escalation can. One that routes only inside measured evidence and re-serves failures on the requested model bounds the risk structurally.
- What is the difference between an LLM router and an AI gateway?
- A gateway is the traffic layer: one endpoint, keys, caps, visibility. A router is a decision layer that picks the model. Some gateways include a router; Finest is a gateway whose router only acts on published evidence.
- Does routing help if I always need the best model?
- On uniformly frontier-hard workloads there is little differential to capture, and honest routing serves your requested model verbatim. The bill then equals list price, which on Finest also means no fee.
In 2 minutes, start cutting your API spend without sacrificing quality. Free if you don’t save money.
No model markup. You pay the host’s rate. 25% of what it proves it saved on a request. No saving, no fee.