Model Routing for Enterprise Agents: Cheap First, Escalate on Evidence

Prerequisites

Before changing production routing, assemble the evidence the router will use and the constraints it must respect. You need a representative sample of completed tasks, including successful requests, recoverable failures, ambiguous inputs, and cases that should stop for human review. Remove or protect sensitive fields before using examples in evaluation, and keep the evaluation set separate from the examples used to tune prompts or rules.

You also need a defined task boundary, a current model inventory, and an owner for the routing policy. Record the provider’s exact model identifiers from your approved deployment configuration, not from a blog post or a remembered alias. The illustrative artifact below uses internal aliases only; it does not assert that a particular vendor model is available or suitable.

Agree on the operational limits before testing: maximum end-to-end latency, permitted escalation rate, acceptable error categories, and which actions may change external state. Specify who can pause a rollout and who investigates a rising failure rate. If the agent can send a message, issue a refund, modify a record, or trigger another consequential action, define how the system verifies the resulting business state independently of the model’s response.

Finally, make the current path observable. A request should have a trace or correlation identifier connecting its routing decision, model call, tool calls, retries, and outcome. You do not need a sophisticated dashboard to begin, but you do need records that let an operator distinguish a slow request from an incorrect classification and a provider error from a failed business action.

Warning: A retry is not a recovery policy. Repeating the same request against the same conditions can repeat cost, delay, and side effects without producing new evidence. Decide which failures are transient, which require a different route, and which must stop.

The routing principle: spend more only when evidence justifies it

A cheap-first policy sends work to a lower-cost or lower-latency candidate when the task is within its demonstrated operating envelope. It does not assume that the cheapest candidate is always adequate, nor that a more capable candidate will repair every failure. The router needs observable criteria for continuing, escalating, asking for clarification, or stopping.

Treat the first model as a candidate worker, not the authority on whether it succeeded. Model-generated confidence can be logged as one signal, but it is not proof of correctness. Stronger evidence comes from checks tied to the task: a required field is missing, a cited record does not exist, a tool returned an error, a structured response fails validation, or an independent business-state check shows the requested change did not occur.

The basic control loop is: classify the task, select a permitted route, validate the result, and choose the next action from the observed outcome. Escalation means a deliberate change to the route or handling policy, not an automatic second attempt. The alternative route should have a reason to perform better for this specific failure. If there is no such reason, stop safely or send the task for review.

This article focuses on routing policy and failure detection. Model selection across a larger production stack is a separate architectural decision; see Multi-Model Production Stacks in 2026: Routing Claude Opus 4.7 and GPT-5.5. Likewise, broad model-release evaluation is distinct from the route-level acceptance test described here; Benchmarks That Actually Matter for New Model Releases covers that wider question.

Step 1: Define one decision boundary

Start with a narrow workflow, not a global rule such as use the cheap model unless it fails. Write down the input, the permitted outputs, the available tools, and what counts as a completed task. A useful first target is a workflow where the initial model can produce a draft or classification, while a validator can determine whether the result meets a finite set of requirements.

For example, a hypothetical operations team might route incoming supplier inquiries. The first route extracts supplier name, invoice reference, request type, and requested action. It may classify an inquiry as informational, a correction request, or a payment-status question. The system can check that required fields are present and that referenced records exist before any action is considered. An ambiguous request can be held for clarification rather than forced into one of the categories.

Draw the boundary between interpretation and execution explicitly. Producing a proposed payment-status response is not the same as sending it. Extracting a requested correction is not the same as editing a financial record. If an action changes external state, require approval before execution, bind that approval to the exact payload and destination, and make it expire. A later model rewrite must not inherit approval for the earlier payload.

Document cases that should never be handled by an automatic escalation. Examples might include conflicting account identifiers, missing authorization data, a request outside the workflow’s declared scope, or an attempted action whose result cannot be independently verified. The point is not to claim these are universal categories; each team should define them from its process and risk tolerance.

Step 2: Separate failure signals from model opinions

Create a failure taxonomy before choosing thresholds. Keep categories that imply different next actions separate. A malformed structured response may justify one controlled repair attempt or a different model route. A timeout may justify a bounded transport retry if the operation is safe to repeat. A contradictory source record may require a human decision. A tool permission failure should not be sent to a more expensive model as though reasoning power could grant access.

A practical initial taxonomy can include:

• Transport failure: timeout, connection interruption, or a provider response that indicates the request did not complete. Determine whether the request may already have been processed before retrying.
• Format failure: output cannot be parsed or lacks required fields. Record the validator and field-level error, not just a generic failure label.
• Task failure: the result is syntactically valid but does not answer the requested task, conflicts with supplied evidence, or fails a workflow-specific check.
• Tool failure: a dependency rejects the call, returns an error, or yields data that cannot support the requested operation.
• Policy stop: the task is out of scope, ambiguous in a consequential way, or requires a human decision under the team’s rules.
• Verified completion: independent checks confirm the task’s required outcome, not merely that the model returned a success message.

Pro tip: Store the raw signal and the normalized category. If every failure is recorded as agent_error, your escalation policy will be unable to distinguish a recoverable output-format problem from an unsafe action or unavailable dependency.

Do not use a model’s self-assigned confidence score as a universal escalation threshold. If you keep such a score, evaluate whether it predicts correctness for this workflow and whether its behavior remains stable across task types. A useful policy can instead rely on deterministic checks, clear uncertainty classes, and evidence from the downstream system.

Step 3: Build the route table around evidence

Make routing decisions legible enough for an operator to explain after an incident. Each row should define the task conditions, initial route, validation, escalation trigger, next route, stop condition, and owner. A generic fallback row is not sufficient: it should say what happens, what information is preserved, and whether the system may take an external action.

The following decision table is a starting point for a hypothetical supplier-inquiry workflow. Its model names are internal aliases for a proposed design, not vendor model IDs. No model capability is implied by the names. Replace them with exact, pinned provider identifiers in the deployment record before a measured test or rollout.

Treat version numbers as policy versions, not model versions. Keep the route table in source control or another change-tracked location, and record changes to thresholds, validators, task scope, and model identifiers separately. A policy change can alter outcomes even when the provider model stays the same; a model change can alter outcomes even when the policy file is unchanged.

The table should answer a harder question than which model is cheaper: what evidence makes the next route preferable? If the initial route omits a field but the input itself lacks that field, escalation cannot recover it. If it returns malformed output, a distinct route may be appropriate if evaluation shows that it reduces that failure. If the underlying account service is unavailable, changing models is unlikely to repair the dependency. Route according to the cause, not the mere existence of an error.

Step 4: Establish a baseline and an acceptance set

Collect representative cases across normal and difficult conditions. Include straightforward examples, abbreviated requests, unfamiliar phrasing, missing identifiers, contradictory records, long context, and requests adjacent to the workflow boundary. Label the expected handling, not just the expected text: complete, clarify, human review, safe stop, or verified action.

Use a held-out set for the final comparison. Keep examples used to write prompts or refine rules separate from the cases used to decide whether the change is acceptable. For each case, capture the route chosen, validation result, escalation reason, final disposition, elapsed time, and relevant usage or cost fields available in your systems. Review the task outcome, not only the model response.

Compare the current route against the proposed cheap-first policy on the same cases and under consistent settings. Reasoning-effort controls are model-specific; compare quality, latency, and cost under consistent settings rather than assuming one shared setting has the same meaning everywhere (OpenAI reasoning guide). This is a narrow statement about controlled comparison; it does not prescribe a model or a particular effort level.

Set acceptance conditions before reviewing results. For example, the team could require no increase in unapproved external actions, a defined maximum rate of unresolved cases, and a measurable improvement in cost per successfully completed task without exceeding its latency limit. Choose thresholds appropriate to the business process and document them. Do not declare success because average cost per model call fell if the policy also increased human rework, abandoned tasks, or incorrect business changes.

A concise scorecard should separate quality and operations. Track task completion after independent verification, critical error count, clarification and human-review rates, escalation frequency by reason, p50 and p95 or p99 end-to-end latency as appropriate, and total cost per completed unit. Make sure the sample is large and varied enough to reveal meaningful failure modes; a small set of easy cases can make an unsafe route appear excellent.

Step 5: Run a measured migration test

Use a test date and a versioned record so results remain interpretable after policy changes. The artifact below is deliberately explicit about unknown provider identifiers. On the test date, the owner should replace each internal alias with the exact provider model ID from the approved deployment manifest, record the prompt or policy version, and note any relevant settings. Without that information, the comparison cannot be reproduced or safely promoted.

The date above identifies the illustrative artifact’s proposed record date, not a completed benchmark. Do not fill missing evidence with estimates presented as results. During an actual test, record the deployment’s exact model IDs, settings, policy commit, dataset version, test operator, and any exclusions. Preserve failed and stopped cases; dropping them from the report makes the cheap-first route look better by concealing its workload.

A safe migration can start in shadow mode: evaluate the proposed route on copies of eligible requests while the existing route remains responsible for the user-facing result. Shadow output must not trigger tools or side effects. Compare its classifications and validation signals with the actual outcome, then inspect disagreements. Shadow mode can reveal routing differences, but it does not prove that the proposed path performs acceptably under real latency, concurrency, or user behavior.

Next, consider a limited canary only if the system has an external mechanism to select traffic and a reliable rollback path. Do not assume a hosted model endpoint will split traffic for you. The router or an upstream service must implement any proposed canary, and operators must be able to identify which policy version handled a request. Start with a bounded, representative slice; expand only after the predeclared conditions hold and new failure signals are reviewed.

For interpreting latency across the whole workflow, rather than only the model call, see Benchmark Agent Tail Latency: Trace One Workflow End to End. This routing test should use end-to-end measurements where they affect acceptance, but it should not substitute a single timing number for the separate task-quality and failure checks.

Step 6: Implement escalation as a state transition

Represent the request lifecycle explicitly. A simple state machine might use received, classified, validated, needs_clarification, escalated, awaiting_approval, executed, verified, and stopped. Each transition should have a triggering event and a recorded reason. This makes it possible to tell whether a request was escalated because of a missing field, an invalid output, a failed dependency, or a business-rule stop.

The flow below illustrates the control sequence. The validator examines observable conditions; escalation is conditional, and unresolved or unsafe work terminates in a review or stop path rather than looping without a bound.

flowchart TD
    A["Receive task"] --> B["Choose initial route"]
    B --> C["Validate result and evidence"]
    C --> D["Accept if checks pass"]
    C --> E["Classify observed failure"]
    E --> F["Escalate once if justified"]
    E --> G["Stop or request human review"]
    F --> C

    accTitle: Evidence-based agent routing
    accDescr: A task uses an initial route, then a validator either accepts it, classifies a failure for one justified escalation, or stops for human review. The escalated result is validated again; the process does not loop indefinitely.

The diagram intentionally allows the escalated result to pass through validation again. It does not imply unlimited retries: set a maximum route depth, such as one escalation for this workflow, and enforce it in code. If the second route fails, preserve the evidence and stop or request review. A retry for a known transient transport failure is a separate mechanism with its own attempt limit and safety checks, not a new model escalation.

Keep the payload, route, and evidence connected. Record the input reference or protected content pointer, policy version, exact model ID, validator outcome, reason code, and any action approval identifier. Avoid logging sensitive content unnecessarily; logs should support diagnosis without becoming an uncontrolled copy of customer data. If a human reviewer changes the disposition, retain that decision as a labeled outcome for later evaluation rather than quietly overwriting the original route trace.

When the policy asks for human review, provide a useful handoff: the task summary, source references, what the system tried, which check failed, and what decision is needed. A queue that receives only an opaque error transfers work without context. Conversely, do not have the model invent a rationale after failure; present recorded signals and let the operator distinguish facts from generated interpretation.

Step 7: Make retries safe and bounded

Separate retryable transport behavior from task-level escalation. Before repeating a call, determine whether the previous attempt completed and whether it could have caused an external effect. Use the correlation ID to look up status where possible. If the status is unknown, reconcile the business record or pause for a human decision instead of assuming the operation is safe to repeat.

For external actions, bind approval to the exact payload, destination, and relevant record, and give it an expiry. Any material change to the proposed action requires new approval. Where the external system supports an idempotency mechanism, use it according to that system’s contract; otherwise, design reconciliation and duplicate detection around the business record. Do not promise exactly-once effects across a network boundary.

Define separate limits for transport retries and model-route escalations. A transport retry might be permitted only for a known transient condition and only when the request is safe to repeat. A model escalation should occur only when a recorded task-level signal indicates that a different route is a reasonable next step. Neither limit should reset simply because a new component in the chain was called.

A useful operational rule is to stop when the evidence does not distinguish recoverable failure from possible completion. For example, if a write request times out after submission and the response is missing, a second write could duplicate the action. First query the authoritative record or use a reconciliation process. If that check cannot determine what happened, place the request in a review state with enough information for an operator to resolve it.

Warning: A higher-cost route is not a substitute for permission, missing source data, an unavailable tool, or an unknown action status. Escalate interpretation problems; resolve system and authorization problems at their source.

Step 8: Calculate cost per verified completion

A route that lowers the cost of its initial call may raise total operating cost if it escalates frequently or creates more review work. Compare complete workflow cost, not just the first model request. Count model calls on the initial and escalation paths, transport retries, tool usage, human review, and follow-up work that the organization can reasonably attribute to the task.

An illustrative calculation makes the tradeoff visible. Assume, solely for planning, that a current route costs $0.12 in model charges per attempt and completes 90 of 100 tasks after verification, with 10 requiring human review at an assumed $4 each. Its modeled total is $12 for model calls plus $40 for review, or $52 for 90 verified completions: about $0.58 per verified completion. These are invented assumptions, not market prices, observed results, or a forecast.

Now assume a proposed first route costs $0.04 per task, 20 of 100 tasks escalate to a second route costing $0.16 each, and 12 tasks require the same assumed $4 review. Model charges total $4 plus $3.20, while review adds $48: $55.20 for the same 90 verified completions, or about $0.61 each. Under these assumptions, the cheaper first call does not lower the unit cost. If the proposed policy instead reduces review to six cases while maintaining the same verified completion count, its modeled total becomes $31.20, or about $0.35 per verified completion.

The calculation is useful only if its assumptions are visible and replaced with measured values. Include indirect costs that materially change the decision, such as operator time, exception handling, or the cost of delayed completion. Unit economics connects costs to business units of value, and its assumptions and relevant indirect costs should be shown (FinOps Foundation: Unit Economics). For this workflow, a defensible unit might be a supplier inquiry resolved and independently verified, not a model call or a generated response.

Do not convert these illustrative numbers into a promised saving or universal threshold. Measure the actual workflow with the finance and operations owners, define what counts as a completed unit, and report both the denominator and the included cost categories. A route with a lower average can still be unacceptable if it produces rare high-impact failures, so keep critical error counts and safe-stop rates visible alongside unit cost.

Step 9: Monitor drift and respond to failure timelines

Monitor routing distributions as well as outcomes. A sudden change in the share of requests sent to escalation may indicate a changed input mix, a degraded first route, a validator change, or a dependency failure. A decline in the apparent escalation rate is not necessarily good; it could mean the validator stopped detecting errors. Alert on meaningful changes by failure reason and inspect example traces before adjusting the threshold.

Consider a hypothetical failure timeline. At 09:00, a supplier system begins returning an additional reference format. The extraction step still returns valid structured output, but the record lookup finds no match. At 09:15, escalation volume rises. If the router treats every unmatched reference as model uncertainty, it may spend more on a second route without fixing the new format. The right response is to identify the common lookup failure, pause any action dependent on that record, and determine whether input parsing or the supplier system changed.

At 09:30, suppose an operator changes a parser rule and the lookup begins returning a match, but a delayed write response leaves the outcome uncertain. That is now a separate failure: possible action completion with an unknown response. The system should reconcile the record before retrying. At 10:00, compare the corrected traces with the original cases and verify that the business result is right. A model’s claim that it completed the correction is not the verification.

This sequence shows why incident handling should preserve timestamps, policy versions, route IDs, dependency results, and action status. It also shows why changing several thresholds during an incident is risky: the team would lose the ability to tell whether the parser fix, route change, or retry behavior caused recovery. Apply the smallest reversible change, record it, and evaluate the affected cases before restoring automated execution.

Step 10: Promote, pause, or roll back deliberately

Promote a policy only when its predeclared acceptance conditions are met on the held-out evaluation and the operational path can detect the failures that matter. Include an owner, an effective time, the policy version, exact provider model IDs, and a rollback target in the release record. For a canary, define the traffic selection mechanism and the signal that pauses expansion; the hosted endpoint should not be assumed to provide that mechanism.

Pause or roll back when a critical error appears, a required validator fails open, the escalation or review queue grows beyond its operating capacity, or trace data is missing. Set thresholds from the workflow’s actual service constraints rather than importing a generic number. If the failure is localized to one route or dependency, disable that path where possible and preserve safe handling for unaffected tasks.

After a change, compare like with like. Separate new input patterns from model or policy changes, inspect changes in completion and stop categories, and review whether operators are doing more correction work. A weekly average can hide a severe change concentrated in one task type. Report results by meaningful category and retain enough trace information to reproduce the decision without exposing unnecessary customer data.

Do not let the router silently learn from every outcome by rewriting its own thresholds or route table. A change to policy should be reviewed, versioned, evaluated, and promoted through the same controlled process as the original. Feedback can inform a future update, but the decision artifact should preserve which policy produced each result.

A printable routing worksheet

Use this compact worksheet in a design review or migration ticket. Complete it for one workflow at a time; blank fields are unresolved decisions, not permission to use defaults.

• Task boundary: State the input, permitted output, tools, and definition of verified completion.
• Initial route: Record the internal route alias and the exact provider model ID from the deployment manifest.
• Escalation route: Record its exact provider model ID and the specific failure evidence that justifies using it.
• Validation: List required fields, source checks, schema checks, and independent business-state checks.
• Stop conditions: Specify ambiguity, conflicting records, missing authority, unknown action status, and out-of-scope handling relevant to this task.
• Side effects: Identify every external action, approval binding, expiry, duplicate-detection method, and reconciliation path.
• Test record: Record policy version, test date, model IDs, settings, dataset version, sample exclusions, and comparison route.
• Acceptance: Predeclare quality, critical-error, latency, escalation, human-review, and unit-cost criteria.
• Operations: Name the owner, trace fields, alert conditions, rollback target, and person authorized to pause expansion.
• Decision: Mark the policy as proposed, tested, approved, paused, or rolled back; attach measured results rather than estimates.

For a practical implementation review, AI strategy and model selection can help teams frame the candidate routes, evaluation boundary, and operating constraints. The most useful next step is usually a small, well-instrumented migration test with explicit stop rules—not a broad model swap.

Key takeaways

Begin with one workflow and define what completion means independently of the model’s assertion. Route the first attempt cheaply only within a demonstrated task boundary; validate its result using observable evidence and make escalation conditional on the failure cause. Treat missing data, permissions, dependency outages, and uncertain external-action status as problems to resolve or stop, not as invitations to spend more on another model call.

Version the policy, record exact model IDs and test settings, and compare the current path with the proposed path on the same representative cases. Measure cost per verified business unit, including relevant review and indirect work, and keep latency and critical errors visible beside cost. Bound retries, bind approvals to exact payloads, and verify side effects through the business system. When evidence is ambiguous, stop safely and preserve a useful trace for human review.

WANT TO TALK THROUGH YOUR SITUATION?

Talk it through with us. Thirty minutes, no slides.