Token Engineering · ~7 min

Right Model — Cost-Aware Routing

Stop paying top-tier prices for file searches. Route each task to the cheapest model that clears the bar, and let a cheap validator decide when to escalate.

Model cost scales with tier and token volume. Top-tier models on every task waste compute; cheap models on complex tasks produce rework. The fix is not a single model — it's a route: match capability to complexity, and escalate only when a deterministic check says the cheap pass failed.

1 Route by complexity, not by habit

The default move — "use the most capable model so I never get burned" — is the anti-pattern. It is safe but wasteful at scale. Cost-aware design matches the model tier to what the task actually demands:

This is the big.LITTLE idea borrowed from CPU architecture: powerful cores for demanding work, efficient cores for background tasks. A community analysis of mixed coding workloads reports a 2–2.5x cost reduction at 85–95% quality when a fast model handles read-only exploration while the main model reasons.

2 Escalate on failure — the cascade

Routing by static task-type is a start, but the real lever is cascade routing: query the cheaper model first, and escalate to a capable model only when the cheap pass fails a check. FrugalGPT showed this can cut cost up to 98%, or improve accuracy 4% at equivalent cost.

The whole pattern hinges on the word deterministic. Escalation is only trustworthy when a cheap, mechanical check decides it — a passing test suite, a clean linter, a green type-checker. That gate is what stops a cheap-tier miss from shipping silently.

3 Don't reach for tier first — retrieval can beat it

Before you escalate to a bigger model, check whether better context closes the gap. A cheaper model with good retrieval can beat a bigger model used alone: on a large-codebase evaluation, a balanced model paired with a code-search MCP beat a top-tier model alone on 6 of 9 tasks at roughly half the cost per quality point (Sourcegraph).

Two more levers sit on the same trade-off. Role routing assigns models by capability, not just complexity — an Action model for tool-based execution, a Thinking model for extended reasoning, a Compact model for summarization during compaction — each with a fallback chain for graceful degradation. And agent weight matters: lightweight agents (<3k init tokens) get composed often; heavy agents (25k+) become bottlenecks. Prefer splitting into light agents over one heavy specialist.