The working vocabulary for this course. Once a term lives here, every lesson uses this word for it. Every cut named below is measured against the same rule: fewer, cheaper tokens, never at the cost of output quality.
The Premise
Token engineering
The discipline of spending the fewest, cheapest tokens that still clear a task's quality bar — treating tokens as a budget to allocate, not a number to minimize. It is design-time and runtime: where the work runs, which model, how much context, how long it may think. Distinct from prompt engineering, which optimizes the words; token engineering optimizes the bill.
Avoid: "use fewer tokens" as the goal — the goal is fewer tokens at the same quality; a cut that drops quality is a downgrade, not an optimization.
The four rights · right model, right token, right cache, right time
The course's diagnostic frame: every token decision is one of four questions — the right model (which tier), the right token (lean context and output), the right cache (reuse the stable prefix), at the right time (synchronous vs. deferred). A frame for locating where a leak lives, not a checklist to run top to bottom.
The one thing you never trade to cut tokens: output quality. It is the clause that makes every other term in this glossary safe — a cheaper route, a leaner prompt, a batch tier all count as wins only if quality holds. When a cut loses quality, the quality clause says it was a downgrade hiding behind a smaller bill.
Route each task to the cheapest model that meets its complexity — fast tiers (Haiku) for high-volume exploration, balanced (Sonnet) for implementation, powerful (Opus) for architecture — and escalate tier only when validation fails. Tier choice is the single biggest multiplier on spend, so this is the first lever to reach for.
Avoid: defaulting everything to the top tier — safe, but it pays a premium on every task a cheaper tier would have handled.
Query the cheap model first, gate the result on a deterministic check, and escalate to the capable model only on failure. FrugalGPT reports up to 98% cost reduction this way. No coding tool ships it natively, so it is a two-pass pattern you build: fast model → tests/linter/type-checker → escalate on fail.
Avoid: a cascade with a slow or absent validator — if the gate costs more than the tier gap, it adds latency without saving money.
The cheap, deterministic check that decides whether a cascade escalates — a test suite, linter, or type checker, not a model judgment. The gate is what makes escalate-on-failure trustworthy; its cost must stay well below the tier difference it guards, or the cascade is a net loss.
The policy of starting on the cheap tier and paying for the capable one only when the gate rejects — don't pre-pay for the biggest model on the chance you'll need it. The expected cost is the cheap pass plus the failure-rate-weighted retry, which beats defaulting high whenever the light tier clears the feasibility threshold.
The break-even condition for routing: the light tier's pass rate on the routed tasks must exceed the inter-tier cost ratio. Below it, cheap-tier failures generate rework that outweighs the savings, so every routing pattern is a net loss. Measure pass rates by tier on a representative sample before committing to a router.
Borrowed from CPU design: pair a powerful model for demanding reasoning with an efficient model for high-volume background work (Claude Code's Explore subagent runs Haiku for read-only exploration while the main model reasons). Community analysis reports 2–2.5× cost reduction at 85–95% quality on mixed workloads.
Assign models by the capability a step needs rather than its difficulty — action, thinking, critique, vision, compact — each independently configurable with a fallback chain for graceful degradation. Complexity routing picks the tier; role routing picks the role. The two compose.
Avoid: conflating role with tier — a thinking step and an action step can run the same tier yet want different models.
Shape every tool so each call injects only the tokens the agent needs for its next decision — return "3 checks passed, 1 failed: lint", not the 40-field API response. A tool that returns 10,000 tokens when 200 would do burns 10% of a 100k window on one call. Keep toolsets small; each extra tool is a per-call reasoning tax.
Avoid: full API-response passthrough — the agent reads one field and pays for the other 95% as noise.
The mechanism that makes oversized context a quality problem, not just a cost one: self-attention scores every token against every other, so irrelevant tokens compete with relevant ones for the model's focus. Accuracy drops sharply when the target sits in the middle of a long context rather than at its ends — so noise buried in a tool dump degrades the answer.
The two ways to cut generated tokens — and only one is reliable. Structural optimization rewrites code into AST-equivalent idiomatic forms (comprehensions, f-strings, compound operators) for an 18–38% reduction with correctness intact. Prompt-level "be concise" is fragile: it creates a competing objective and can make the agent do less work, not better work.
Avoid: "write concise code" in a system prompt — show idiomatic examples instead; agents pattern-match from examples more reliably than they obey abstract directives.
Cleaner code (fewer static-analysis violations, lower cognitive complexity) does not change an agent's pass rate but cuts token use 7–8% and file revisitations 34% — a controlled minimal-pair result. The mechanism is navigation: the agent finds the right file on the first try. A real lever, but small; it pays off only as a side effect of maintainability work you'd do anyway.
Avoid: a refactoring sprint for the agent — at 7–8% it rarely pays back inside a year, and agents re-introduce the mess without quality gates.
The SonarQube metric for intra-function branching depth — one of the two operationalized, CI-measurable proxies (with rule-violation count) for the cleanliness that lowers agent token use. A partial proxy: it captures branching, not cross-cutting concerns, hidden global state, or naming, so it under- or over-weights cleanliness depending on the codebase's actual pathology.
A provider mechanism that stores a prompt prefix so re-sends bill at the cache-read rate (~0.1× input on Claude, a 90% discount) instead of full price; the first write costs ~1.25×. A 10k-token static prefix reused ten times costs ~22,500 effective tokens cached vs. 110,000 uncached. The discipline is making the prefix reusable in the first place.
The ordering rule that makes the prompt cache hit: place stable content (system prompt, tool definitions, durable rules) first and volatile content (the live turn, fresh tool results) last. A single change near the top invalidates everything after it, so anything that varies per turn belongs at the end.
Avoid: a timestamp or per-turn ID in the system prefix — it busts the cache on every call.
The pre-migration audit confirming a prefix still clears the model's minimum cacheable length (typically 1,024–4,096 tokens) and that compaction thresholds reflect the current token count. A step in the tokenizer-swap checklist: a new tokenizer can quietly push a once-cacheable prefix under or over the boundary.
The hidden cost when a model upgrade ships a new tokenizer: the same text maps to a different token count (Opus 4.6→4.7 ran up to ~1.46× on system-prompt text), so effective cost, context-window headroom, and rate-limit envelopes all shift while per-token price and nominal context size hold flat. Forecast from your measured workload mix, never the vendor's worst-case range.
Avoid: trusting a client-side token estimator across a tokenizer change — re-test any fixed token-to-character ratio against the new model.
Temporal token routing · batch routing, right-time lever
Route any call whose result is not needed in the next second into an asynchronous tier. The decision is one question — does anything wall-clock-sensitive consume this output? If not, the synchronous API is the wrong price. Resolved by a single discount primitive: defer the result, halve the bill.
The asynchronous queue both Anthropic and OpenAI ship at 50% off synchronous, completing within a 24-hour SLA (most within an hour). The home for bulk evals, embeddings, moderation, dataset enrichment, and overnight refactors — work that is both deferrable and high-volume enough to amortize the submit-poll-retry pipeline.
Avoid: the batch tier for a one-off small job or a chained pipeline — queue-management overhead and serialized latency erase the discount.
OpenAI's interactive-but-pre-emptible tier billed at batch rates — the closest thing on the market to a latency-tiered discount for real-time traffic. Fits background evals, data enrichment, and low-priority agent steps that tolerate seconds-to-minutes of added latency; the caller must handle a 429 "resource unavailable" by retrying or falling back to standard.
The reframe that governs every batch-fit decision: the 24-hour SLA is not a generous deadline, it is the price ceiling. The provider promises to serve eventually, not promptly — expect long tails and expirations under demand-driven slowdown. A job with a hard sub-24h deadline must build for resubmission or stay synchronous.
Allocating compute by a task's difficulty curve — harder tasks get a larger budget, simple ones get less — rather than spending a flat maximum everywhere. A different cut than tier routing: routing picks which model, effort scaling picks how hard it works on a given call.
The cap on tokens a reasoning model may spend thinking before it answers, set by an effort (low/medium/high) or task-budget knob. The catch the lessons stress: more reasoning is not monotone — HAL found a larger reasoning budget lowered accuracy in 21 of 36 model×benchmark settings. The cheapest effort tier that holds quality is the default, never the highest.
Avoid: maxing the reasoning budget by reflex — sweep it like any other knob; high effort frequently costs more and scores worse.
The tokens a reasoning model spends in its internal reasoning trace before the visible answer — billed as output, so they hit the cost axis at the ~4× output weight. Worth paying only where they change the answer; on tasks that don't need multi-step reasoning they are pure overhead.
The four-stage loop that turns a recurring agentic workflow into a measurable cost surface: capture every call in a normalized log (instrument), rank spend by workflow and lever (attribute), apply a named lever — tool pruning, pre-agentic CLI substitution, relevance gating, cheaper-model routing (fix), then re-run and confirm cost trended down including output (verify). Pays back only on high-frequency, stable workflows with output-side measurement.
One scalar that ranks configurations across tier, output ratio, and cache hit rate: ET = m × (input + 0.1×cache_read + 4×output), with tier multiplier m = 0.25 / 1.0 / 5.0 for Haiku / Sonnet / Opus. The 4× weight matches output pricing; the 0.1× matches the cache-read discount. Convert ET to USD for the frontier plot. Prioritize work by ET/run × runs/day, not ET/run — frequency is the multiplier.
Plot each configuration as a (cost, quality) point and keep only the non-dominated set. The frame makes the quality clause visible: a cache or batch change holding quality moves a point left along the frontier; a model downgrade that loses quality drops it below it. The bill collapses both into "lower spend" and hides the regression until users complain.
Avoid: a frontier built on one run per cell — single-run sweeps inflate accuracy and cost estimates (up to 8.7% / 88%); average over runs or correct the bias.
A configuration that another beats on cost and quality at once — it sits below the frontier and should never ship. HAL found the most expensive models are rarely Pareto-optimal and only ~a third of configurations sit on the frontier on average, so choosing on intuition overspends by reflex. The plot exists to make the dominated points obvious.
The correct cost axis for the frontier: denominate in dollars (not raw tokens, which under-weight output-heavy runs) and divide by validated outputs, not attempts. If 100 requests need 112 tries to produce 100 valid results, cost is total spend over 100. Charging retries and failures to the cost axis stops a configuration that fails-and-retries from looking cheap.