The Request Compiler

⤢ fullscreen

1The overpayment problem

The model wars produced a strange market: quality went flat while prices stayed vertical. Mid-tier models now sit within a few points of the frontier on most workloads at a fraction of the price, and frontier models ship with reasoning modes on by default, so a classification call pays output-token prices to wait while the model thinks about a task that needed none of it. Claude Opus 5 costs $25 per million output tokens and Claude Fable 5 costs $50; Gemini 3.6 Flash costs $7.50, GPT-5.6 Luna costs $1.20, and DeepSeek V4 Flash costs $0.28. The last increment of quality is an order-of-magnitude step.

Engineering teams respond rationally: they pick a frontier model once, set max settings, and never revisit it, because the alternative is re-running evaluations they don't have against models that change monthly. The result is a market-wide overpayment that no one can see, because evidence about models on your workload is exactly what doesn't exist. Everyone sells access to models. Nobody sells proof about them.

2Workloads have anatomy

The compiler's first move is not a model call. It is a deterministic observation: countable structure, computed in microseconds, never embeddings or a serve-time classifier. Every request carries an anatomy, and the anatomy decides what may be tried. This is the same lesson document pipelines learned the hard way: born-digital versus scanned versus genomic-dense are knowable before any expensive pass, and the safe direction for a classification error is toward the expensive model.

Anatomy is stamped on every receipt as content-free facts, never prompt text, which produces a census of real traffic before any routing is built for it. Each axis earns its build: below a funding bar of measured demand, the observation stands alone as a trust fix and the routing waits.

3Execution: four strategies, one floor

"Best of best" is not always model selection. The compiler admits four execution strategies per request, plus a fifth, passthrough, which is a first-class success state. Every strategy shares one property: any loud failure escalates to the requested model, and the customer is debited no more than that model alone would have cost. The detour is our cost, by law.

Why not simply send everything to the cheapest modern model? Because quality cliffs are input-dependent and invisible on easy inputs. The measurements below, from a real extraction benchmark run with production scoring rather than vibes, are the reason evidence gates every routing decision:

4Evidence before authority

A plan may influence a response only after it has climbed a ladder that is deliberately hard to climb: observation, then shadow comparison against the requested model at our cost, then a frozen plan confirmed once on a sealed, group-disjoint set, then a sticky canary with a circuit breaker, then, and only then, active service within a declared eligibility envelope. Quality-affecting changes (a model version bump, a prompt edit, a price epoch) invalidate authority until re-confirmed. The model-wars churn that today forces every team to re-evaluate quarterly becomes our scheduled maintenance.

The verifier hierarchy is honest about its limits. Deterministic checks (schema, conservation, provenance) authorize only the property they actually decide. Calibrated learned checks carry measured false-pass behavior. Blinded cross-family judges are supporting evidence, never promotion authority. And the requested model itself is the measured baseline: never ground truth merely because it was requested. Where nothing machine-checkable applies, the system abstains, the receipt says so, and no fee is earned.

5Case study: decomposing a document pipeline

A health-records pipeline began, as most do, by sending every document to the strongest available model. Its operators then decomposed the job into typed capabilities (OCR, extraction, verification, classification), measured twelve models against four document shapes with the production scorers, and re-assigned each leg to the cheapest sufficient executor while keeping the verification gate on the strongest model. The routing engine shipped dark, behind flags, with residency asserted at construction: a configuration that would route data to an ineligible endpoint refuses to build.

Two details matter more than the headline numbers. First, the failures were found by measurement, not prevented by intuition: an 86-page genomic report lost 37% of its facts on the cheap extractor, so a deterministic doc-shape classifier now routes those documents upward, a classifier that can raise cost but never weaken the gate. Second, the one error class a verifier cannot catch, an OCR misread that grounds against its own transcription, was closed not with a stronger single model but with independence: a second OCR read whose disagreements surface to review. Consensus is disagreement evidence, and it is the only mechanism that works there.

6The economics of the attempt

Two questions kill any system like this if they aren't answered in numbers: who pays for failed cheap attempts, and what happens to latency. Our answers are structural. On money: escalations are capped per request (retries are zero by design), every loss is recorded as signed negative savings on an auditable ledger, and routed cells auto-suspend when their measured escalation rate crosses a ceiling, long before the portfolio math turns negative. On latency: the median improves, because cheap models answer faster than frontier; the tail is escalations, bounded at roughly champion + cheap, controllable per request with one header.

This is our money. We pay for every failed cheap attempt, which is why we cannot afford to be wrong.

7The two-minute experience

None of this should be the customer's job. The integration is a key swap; the observation and shadow evidence are free and automatic; the only question is a one-tap activation (provider boundary, posture, latency), asked after the measured report has already shown what overpayment looks like on their own requests.

Everything else is harvested, not asked: schemas from the codebase, shape from traffic, constraints from a one-time boundary declaration. New model releases are re-measured against the customer's workloads automatically. Companies should focus on their core business, not on tracking the model wars. That is the product.

8Limitations

Coverage is deliberately incomplete. Tightly coupled reasoning, holistic creative work, and requests with no verifiable joint are not decomposable by this method; the system abstains, and abstention rates are published per workload, not averaged away.

Correlated errors escape comparison. Two systems can share an omission. The requested model is a baseline, not truth; consensus across provider families and deterministic conservation checks narrow but do not eliminate this class.

Synthetic corpora are not production. Minted corpora give free, exact labels at the decision boundary; they are guarded by real-distribution holdouts, and arms that regress there are re-minted before anything publishes.

The counterfactual can lie if estimated. Pricing the requested model at a cheap arm’s token usage can invent savings (verbosity differs per model). Graph savings stay estimated and fee-free until an observed or reconciled baseline exists.

Benchmark scores here are directional. Prices are provider-official; quality scores are public-benchmark orientation. Finest admits nothing on benchmarks; admissions happen on your workload’s evidence, and your workloads will differ.

9Related work

Cost-aware routing is established: FrugalGPT [1], RouteLLM [2], and AutoMix [3] built cascades and learned routers; compound-AI framing [4] and Anthropic's workflow taxonomy [5] describe the design space; decomposed prompting [6] shows modular subtask routing while newer work warns that incorrect decomposition and synthesis can degrade end-to-end reasoning [7]; Mixture-of-Agents [8] demonstrates portfolio gains at portfolio cost; DSPy and successors [9] optimize prompts when a metric and examples exist; semantic uncertainty [10] supports calibrated abstention; and judge-evaluation work [11] is why no uncalibrated judge sits on our serving path. Agent-security work [12] informs the trusted-control/untrusted-data separation in the plan runtime.

What these leave open, and what we claim as the product, is the contract around the machinery: an immutable passthrough baseline, evidence-bound activation, per-request receipts, a database-enforced billing law, and abstention as a reported first-class outcome.

References

1. FrugalGPT: how to use large language models while reducing cost and improving performance
2. RouteLLM: learning to route LLMs with preference models
3. AutoMix: automatically mixing language models
4. Berkeley AI Research: Compound AI Systems, the shift from models to compound systems
5. Anthropic: Building effective agents, workflows vs. agents; routing, chaining, parallelism, evaluator loops
6. Decomposed Prompting: a modular approach for solving complex tasks
7. Reasoning can hurt: decomposition, subtask, and synthesis failures as distinct error classes
8. Mixture-of-Agents enhances large language model capabilities
9. DSPy: compiling declarative language model calls into self-improving pipelines
10. Semantic entropy: detecting confabulations with semantic uncertainty
11. JudgeBench: a benchmark for evaluating LLM-based judges
12. AgentDojo: a dynamic environment to evaluate prompt-injection attacks and defenses for LLM agents

Cite this work