Token Engineering · ~7 min

Right Cache — Caching Discipline

A prompt cache is the cheapest token there is — but it only pays out if your prefix is stable and your tokenizer never moves underneath it.

The first two Rights cut tokens you send. This one makes the tokens you re-send nearly free. A prompt cache discounts any leading run of tokens identical to a previous call — so the discount is a property of ordering, and it is silently fragile to anything that changes how text becomes tokens.

1 Stable-first, or no hit at all

Caching matches on a prefix. The model can reuse compute only up to the first token that differs from the previous request. One byte of drift near the front voids everything after it. So the discipline is brutally simple: put what never changes first, what changes every call last.

2 A tokenizer swap silently voids the cache

Caching matches on tokens, not characters. So the matching prefix only survives while the same text maps to the same token sequence. A model upgrade that ships a new tokenizer remaps that text — the cached prefix no longer exists in the new vocabulary, and every previously cached token is gone. The pricing page shows no change, which is exactly why this bites.

Anthropic documents the Claude Opus 4.7 tokenizer as producing roughly 1.0–1.35x as many tokens (up to ~35% more, varying by content) for the same text versus prior models — at unchanged $5 / $25 per MTok pricing and an unchanged 1M-token window. Simon Willison measured the real spread against the count_tokens endpoint, and it is workload-shaped:

3 Re-count before you cut over

The headline multiplier is a spread, not your number. Anthropic's migration guide names concrete pre-cutover work — and the cache-fit check is part of it. Re-run representative prompts through both tokenizers, then update the envelopes the new count shifts:

• Re-count by content type — text, images at target resolutions, PDFs, tool results all multiply differently; blend by your real mix, never the worst case.
• Audit client-side estimators — any code path assuming a fixed token-to-character ratio must be re-tested against the new model.
• Update max_tokens, compaction triggers, and prompt-cache fit checks — the same source now fills more of the window, so caches and compaction thresholds drift.

And remember the multiplier overcounts: a more capable model can finish tasks in fewer retries, and effort / task-budget controls can absorb part of the lift. Tokenizer arithmetic alone overestimates the migration cost — so forecast from your workload, then check the cache actually still hits after cutover.