Here’s the asymmetry that makes model routing worth the trouble: most of what an agent handles in production is easy — a well-formed tool call, a summary of a short document, a classification with an obvious answer — and you’re routing all of it through the same model you needed for the 10% of requests that are actually hard. A cascade fixes that by trying the cheap model first and escalating only when it’s warranted. Done right, this cuts the token bill 40-70% with no quality loss. Done wrong, it just moves your failures downstream where they’re harder to see.

This post covers the pattern itself, the one design decision that determines whether it works (the escalation trigger), the failure modes that show up when you get that decision wrong, and when the added architectural complexity isn’t worth it.

The pattern

A routing cascade is a pipeline, not an agent — the control flow is yours, not the model’s, which matters because it means you can reason about it and test it (see when not to build an agent for why that distinction is the whole ballgame). The shape:

def route(request, cheap_model, expensive_model, escalate_fn):
    cheap_response = cheap_model.complete(request)
    if escalate_fn(request, cheap_response):
        return expensive_model.complete(request)
    return cheap_response

That’s the entire pattern. Everything that makes it work or fail lives inside escalate_fn. A cascade with a bad escalation trigger is worse than not having one, because it adds latency (you paid for the cheap call and the expensive one) without saving money on the requests that needed escalating anyway, or worse — it fails to escalate the ones that did.

The trigger is the whole design problem

There are three broad strategies for escalate_fn, in order of how much I trust them:

1. Structural signals — cheapest to compute, hardest to game. Does the cheap model’s output pass a schema check? Did it call a tool with valid arguments? Did it produce a response of plausible length for the task? These are binary, deterministic, and don’t require another model call. If you’re doing structured extraction or tool-calling, this alone catches a large fraction of the cases that need escalation, because a model that’s out of its depth on a task usually fails structurally before it fails semantically — malformed JSON, a tool call with an argument that doesn’t type-check, an empty required field.

def escalate_on_structure(request, response):
    try:
        parsed = json.loads(response.text)
    except json.JSONDecodeError:
        return True
    if not schema.validate(parsed):
        return True
    return False

2. Self-reported confidence — cheap, but only trustworthy if you’ve measured it. Asking the cheap model to emit a confidence score alongside its answer costs nothing extra (same call, one more field), but the score is only meaningful if you’ve correlated it against ground truth for your task on your model. Confidence scores from LLMs are not calibrated out of the box — a model saying “0.9 confident” has no guaranteed relationship to a 90% chance of being right unless you’ve checked. Treat the raw score as a ranking signal, not a probability, and pick your threshold empirically:

def escalate_on_confidence(request, response, threshold):
    # threshold is not 0.5 by default — set it from a labeled
    # validation set for this task, this cheap model, this prompt.
    return response.confidence < threshold

The measurement step here is not optional. I’ve seen teams ship this with a threshold picked by feel, and the cascade quietly escalates 80% of requests (no savings) or 5% (all the savings, none of the safety).

3. A judge model — most expensive, most flexible. For tasks where structure and self-report both fall short (open-ended generation, nuanced classification), you can have a third, cheap-but-not-trivial model score the response before deciding whether to escalate. This is the LLM-as-judge pattern applied to a single response instead of a full eval, and it inherits the same biases — position bias, length bias, self-preference if the judge is a sibling of the model being judged. Use it only when the first two options genuinely don’t apply, and validate the judge against labeled examples before trusting it in the loop.

Where cascades break

Escalation latency compounds on the requests that most need to avoid it. The hard requests — the ones that escalate — now pay the cheap model’s latency plus the expensive model’s latency, serially. If your P99 latency budget is set assuming a single model call, a cascade blows through it precisely on the tail you cared most about. Measure P99 with escalation included, not average latency, or you’ll ship a cascade that looks fine in aggregate and pages someone at 2am on the hard cases.

Structural checks don’t catch confident wrongness. A cheap model can produce a perfectly valid, schema-conforming, plausible-length response that’s just wrong — hallucinated a field value, picked the wrong tool for a subtly different task, summarized the wrong section. Structural signals catch “this output is malformed,” not “this output is incorrect.” If your task has a failure mode that looks structurally fine but is semantically wrong, you need a confidence or judge signal, not just a schema check — and you need eval data showing your structural checks actually correlate with correctness for your task, not just an assumption that they do.

Threshold drift. The cheap model gets updated by the provider, your prompt changes, your input distribution shifts — any of these silently moves the relationship between your confidence threshold and actual accuracy. A threshold tuned once and never revisited degrades quietly: you don’t get an error, you get worse escalation decisions that show up as a slow quality decline nobody traces back to the cascade. Re-validate the threshold on a schedule or when you change the cheap model, not just at launch.

The cascade becomes the thing you’re debugging instead of your actual product. Every layer you add — structural check, confidence gate, judge call — is a component with its own failure modes, and now you’re maintaining a routing system alongside the thing it routes for. This is the real cost that doesn’t show up in the token bill: engineering time spent tuning thresholds and debugging misroutes instead of the product.

The arithmetic on whether it’s worth building

Say the expensive model costs 15x the cheap one per request (a reasonable ratio between a small and a frontier model), and 70% of your traffic is genuinely easy. Route everything through the expensive model: cost is N × 15. Route with a cascade at 70% cheap-only: cost is 0.7N × 1 + 0.3N × 16 (the 30% pays for both calls) = 0.7N + 4.8N = 5.5N. That’s a 63% reduction — real money at volume, and it’s the headline number that makes cascades attractive.

But that arithmetic assumes your escalation trigger correctly identifies the 70% that don’t need escalating. If it’s wrong 10% of the time in the direction of not escalating requests that needed it, you haven’t just lost some savings — you’ve shipped wrong answers to 7% of your total traffic, silently, at whatever quality bar the cheap model has for hard problems it doesn’t recognize as hard. That’s the trade a cascade actually makes: token savings you can calculate in advance, against an error rate you can only know by measuring, not assuming.

Build one when: your traffic has a genuine easy/hard split (check this — don’t assume it), you can build a structural or validated-confidence trigger for your specific task, and you can afford the eval work to validate the threshold before it’s live.

Skip it when: your traffic is uniformly hard (no savings available), you can’t validate the trigger against labeled data (you’re guessing at the threshold), or the engineering cost of building and maintaining the router exceeds what you’d save — which is common at low volume, where the token savings are real but small and the failure modes are exactly as expensive as they are at high volume.

What I’d do

Instrument your traffic first: log what fraction of requests the cheap model alone would get right, using whatever ground truth you have (even a small labeled sample). If that fraction isn’t large, a cascade is solving a problem you don’t have. If it is, start with the structural trigger — it’s free, deterministic, and catches more than you’d expect — and only add a confidence or judge layer if structural checks leave real failures uncaught. Measure P99 latency with escalation in the path, not around it, before you ship. And put the threshold re-validation on a calendar, because the cascade that was correctly tuned at launch is not the cascade you’re running six months later unless you checked.