Token Engineering · ~8 min
Eight lessons, one reflex: find where the tokens leak, then apply the one of the four rights that plugs that leak — model, token, cache, or time. Here's the whole course as a lookup table, plus a mixed review.
The through-line of the course: token engineering gets the same outcome for fewer, cheaper tokens — the right model, the right token, the right cache, at the right time — and every technique carries an implicit "without losing quality" clause. Reach for the lever that matches the leak, not the one nearest to hand.
| Symptom you're staring at | Move | From |
|---|---|---|
| The bill is high and you reflexively reach for the biggest model | Cost-aware routing — send each task to the cheapest model and tier that clears the bar, escalate only on a deterministic validation failure (linter, tests) | L2 |
| Every call drags bloated tool output, full-file rewrites, or messy code | Trim at the source — terse tool schemas, diff-style edits, clean idiomatic code (cut 7–8% with no pass-rate change) | L3 |
| Spend stays high on a stable, repeated prompt prefix | Caching discipline — order context stable-first so the cacheable prefix hits; read is ~0.1× input | L4 |
| Work runs synchronously at full price but could wait | Temporal routing — defer overnight evals, refactors, doc refreshes to the Batch API: 50% off, <24h SLA | L5 |
| Reasoning tokens are spent uniformly, on easy and hard tasks alike | Effort scaling — scale the reasoning budget to difficulty; the cheapest effort tier that holds quality is the default | L6 |
| The cost dashboard is green but users complain quality slipped | Plot the cost/quality Pareto frontier — a quality-losing downgrade shows as a vertical drop off the frontier, not a lower bill | L7 |
| You don't even know where the tokens go | Profile spend always-on — instrument → attribute → fix → verify before cutting anything | L7 |
| You're about to cut tokens hard to save money | Mind the guardrail — token preservation backfires when the cut degrades the output; the quality clause is the whole point | L1 |
The four rights are not a menu to stack blindly; they're a diagnosis. Right model attacks the per-task tier (route by complexity, escalate on failure). Right token attacks the size of each call (lean tool schemas, diff edits, clean code). Right cache attacks repeated prefixes (stable-first ordering). Right time attacks urgency (batch the deferrable). Effort scaling and always-on measurement are the two supporting levers that keep the four honest — you can't route or batch what you haven't profiled, and you can't trust a cut you haven't plotted against quality.
A Batch-API switch that holds quality and a model downgrade that loses 14 points of pass rate both register on a finance dashboard as "lower spend." Only the joint cost/quality plot tells them apart — one moves left along the frontier, the other drops below it. And watch the reflex: higher reasoning effort frequently lowers accuracy (21 of 36 settings in one study), and the most expensive models are rarely Pareto-optimal. Cheaper is not automatically worse; pricier is not automatically better. Measure, don't assume.
Mixed review — across all eight lessons
Question 1 · from L2Cost-aware routing sends each task to…
Question 2 · from L4Under flat per-token pricing, swapping the tokenizer…
Question 3 · from L5The Batch API trades latency for…
Question 4 · from L7On the cost/quality Pareto plot, a downgrade that loses quality shows as…
Question 5 · spaced recall from an earlier lessonScaling reasoning effort to task difficulty matters because higher effort…