Routing down is harder than routing

Model routing is a simple idea: score each request, send the easy ones to a cheaper model, keep the quality on the hard ones. Most turns don't need the most expensive model, so route them down.

We build a routing proxy, and last week it returned a 400 on one of our own sessions.

A request went to Claude's top model. The router scored the turn as medium complexity, rewrote the model field to a cheaper Sonnet, and forwarded it. Anthropic returned:

400 invalid_request_error
"role 'system' is not supported on this model"

The complexity call was fine. The problem was the request body: it was built for one model and sent to another. That is the part worth writing down. Routing down is not the same problem as routing. Routing is classification. Routing down also has to keep the request valid on the target and make the switch worth it. In our experience it fails in three ways, and only one of them is about which model you picked.

1. The body is model-shaped (correctness)

A Messages API request is not portable across models. Newer models accept shapes that older, cheaper ones reject:

Inline system turns. Newer models allow {"role":"system"} messages inline in the conversation. Older models accept system text only at the top level. Downroute a body with an inline system message and you get the 400 above.
Tool-type versions. Server tools carry dated type strings like web_fetch_20260209. A model that predates that date returns a 400.
Reasoning parameters. The newest models take an effort field and reject budget_tokens. Older models require budget_tokens and do not accept effort. Whichever one the client sent, the target might reject it.
Beta headers. A context-1m beta header is harmless on some models and triggers a "credits required" error on a model that already has 1M context.

None of these are complexity mistakes. The router picked a reasonable model; the body was wrong for it.

So a downrouting proxy has to check the outgoing body against the target and either fix it or skip the switch. For the inline-system case we now move the inline system content into the top-level system field, where the older model accepts it. If it can't be normalized cleanly, we don't downroute at all and forward the request on the original model. The rule we hold to is that a routing optimization should never turn a valid request into an error. A missed saving is invisible; a failed request is not.

2. Cheaper per token can still cost more (economics)

This is the one that decides whether routing down actually saves money.

Long agent sessions run on a warm prompt cache. The stable prefix (system prompt, tool definitions, early history) is cached, and later turns read it at about a tenth of the input price. That cache is per model. Route a mid-session turn to a different model and its cache is cold, so the cheaper model writes the whole prefix from scratch, and writes bill at 1.25x base input.

The comparison that matters is not "Sonnet is cheaper per token than the bigger model." It is the one-time cold write on the cheaper model against the warm-read discount you gave up on the expensive one. On a long session with a large cached prefix, downrouting a single turn can cost more than leaving it in place.

A router that only reads per-token prices will get this wrong. The switch has to be worth more than the cache it discards, which means the router needs the session's cache state, not just the prompt score. We gate downroutes on that: if the target is cold and the prefix is large, we keep the current model. When a switch is wanted consistently, we warm the target first and switch once it's a cache hit.

3. The wrong moment to be cheap (quality)

A complexity score reads the prompt, not the situation. Two turns can score the same and carry different risk:

The turn right after a tool call returned is_error: true. The agent is recovering. Route it down and a weaker model can compound the error, so the cheap turn buys a few expensive retries.
The turn in the middle of an active edit or build loop, where the model is holding a lot of state. Downrouting there tends to cost quality where it matters most.

So the router needs more than "is this prompt simple." It needs "is this a safe moment to be cheap." We keep the strong model on recent-tool-error turns and on active-editing turns regardless of the score, and take the quality over the saving when the two conflict.

Takeaway

Routing down in production is three checks, not one:

the body stays valid on the target,
the switch beats the cache it gives up,
the moment is a safe one to be cheap.

You can score complexity perfectly and still return a 400, spend more than you saved, or weaken a recovery. Changing the model is the last step, after those three check out.