Most teams route coding agents with a model router. We use a one-cent prompt instead, and it cut our cost per run by 68%. It runs once before each session and chooses the model, the reasoning effort, and the harness the run will use.
Nora is our coding agent at Abnormal. Requests reach it through Slack, GitHub, Jira, and automated feedback. Here’s how we route its requests.
Figure 1: The main request path runs across the top. Below it, exploration tests alternative routes, and calibration uses the results to inform future routing.
Why we route the harness
A model router can select a backend for each API call while the surrounding agent keeps the same tools and execution loop. Nora selects the coding setup once, before the session starts.
Nora’s router picks the model and reasoning effort, then launches the matching native coding CLI: Codex for GPT models, Claude Code for Anthropic models. That CLI is the harness: it manages the tools and conversation as the agent works.
Routing the whole harness gives us:
A native coding CLI for each model, with access to that model provider’s server-side features.
One tool and context manager for the full coding session.
Prompt-cache reuse from a stable setup across calls.
Consistent tool-use histories for evaluation and targeted prompt improvements.
A small routing policy we can explain and steer in prose.
Sizing the ask (not the PR)
The routing LLM, which we call the judge, reads the request and a little PR context. That includes the description and changed paths, plus the relevant diff for an inline review comment.
Nora supports GPT as well as Anthropic models. The routing objective is the best return on model spend for the job. We give the judge profiles of the models preferred by the current policy, with their strengths and relative costs.
The judge weighs the profiles against the request. A precise edit may suit a lightweight model; specialized work can favor a model whose profile describes the relevant strengths. It chooses reasoning effort separately from the model.
We optimize for return on model spend, so that we choose the model that can do the work well without buying capability the request won’t use. This edited excerpt shows how the judge reads the request and makes that choice:
Figure 2: An adapted routing-prompt excerpt showing the ROI objective and how the judge evaluates the request.
Here are three recent decisions, with our model aliases in parentheses:
Every choice comes with the judge’s one-line reason for it, which we feed back into calibration. The decision runs in the background while Nora prepares the sandbox, and engineers can always request a model explicitly.
Where the spending went
An attempt is one execution of a coding request. Comparing our earlier frontier-default sample with the recent routing sample, average model cost per attempt fell 68% and median cost fell 87%. Even the recent 90th percentile was below the earlier median. The recent sample includes exploration runs and requests where engineers explicitly chose a model, both of which nudge the average up.
Figure 3: Model cost per attempt (average, median, and 90th percentile) and median agent runtime, before and after routing..
Effort and model choice both affect how long a run takes. Nora reserves deeper reasoning for work that actually needs it. Across our before/after samples, median time inside the agent fell 69%, from 36.4 to 11.3 minutes.
A prompt we can improve
The policy is plain text, so tuning it can be as small as editing a sentence. A calibration job runs on a fixed schedule, comparing the judge’s choices and explanations with each run’s cost and duration. It looks for routine work that got more model than it needed and harder work that may have gotten too little.
It also reviews new-model exploration campaigns. A candidate might earn a place in the recommended fleet for specialized work, replace an existing model, or prove not worth continuing to test. Calibration proposes any prompt or campaign changes; we review them and replay prompt edits against past requests before rollout.
Giving new models a trial
New models keep arriving, and price-per-token only tells you so much about how much a task will cost. So we test each new model on real traffic; we route a forced random share of eligible requests to it, run it beside the judge’s usual choice, and feed the comparison into the next calibration.
We did this most recently with Astra, starting with 10% of eligible requests across the fleet. We did not see a return on simpler work, so we narrowed the trial to requests routed to Sol, eventually sampling 40%. That gave Astra a fair chance on work where extra capability might justify the cost.
In the Sol trial, Astra looked much faster on whole requests. It was also getting smaller jobs. Comparing cost and elapsed time per model turn — a step in the agent’s work — helped account for that difference.
Roughly twice the cost per turn bought 14% less elapsed time per turn. None of the tested effort levels reached cost parity.
We found no demonstrated capability gain over Sol to justify that premium, so we ended the campaign.
Engineers can still request Astra explicitly, and a future trial can focus on work where there is a specific reason to pay more.
Figure 5: Astra’s roughly doubled cost per turn bought 14% less elapsed time, without a demonstrated capability gain to justify the premium.
We also plan to evaluate TypeSafe’s Jev as a replacement for the routing judge. Our current judge costs about one cent per decision; we’ll compare routing quality, latency, and cost against that baseline.
In the meantime, a lint fix can stay a lint fix, even on a 6,000-line PR.