Context Engineering · ~7 min

Measure Before You Optimize

Every token-saving trick looks free on a token counter. On a real task, most of them quietly cost you accuracy — measure that, not just the savings.

Three popular optimizations — swapping tool-call notation, minifying source code, and upgrading models — all save tokens. All three can regress accuracy or cost, and a token counter shows none of it. The fix is the same every time: A/B on a real task.

1 Token-saving formats lose accuracy

Token-optimized notations like TOON and TRON re-encode JSON to strip repeated property names and structural punctuation. Isolated single-turn benchmarks report 30–60% savings. Inside real agentic loops, the savings shrink and the accuracy bill arrives.

The cost has a clean mechanism: models were trained predominantly on JSON, so unfamiliar notation burns reasoning capacity on parsing instead of the task. Worse, TOON shows cascading parse failures across turns and collapses parallel tool-call output on most tested models — failure modes a single-turn test can never surface.

2 Minification cuts tokens, tanks solve rate

Stripping comments, whitespace, identifier length, and docstrings from source fed to a coding agent is the same trap with a sharper edge. The token savings are real; so is the lost capability.

The reason: LLMs use identifier names as a primary semantic channel, not redundant gloss on the AST. Strip the names and you remove load-bearing input. The lost capacity doesn't vanish — it resurfaces as failed tasks, or (in a log-compression study) as a 17% token cut that raised total session cost 67% because the model spent reasoning tokens reconstructing what was removed.

3 A new tokenizer moves your numbers

You don't have to change a line of code to get hit. A model upgrade can ship a new tokenizer, and the same prompt suddenly maps to more tokens — silently shifting cost, window headroom, and rate-limit consumption.

The multiplier is workload-shape dependent: a text-heavy system prompt measured 1.46×, a PDF 1.08×, matched-resolution images ~1.0×. Forecast from a measured per-content-type multiplier on your own traffic, not the vendor's worst case — and re-check max_tokens, compaction triggers, and any client-side token estimator before cutting production traffic.

4 Structural beats lossy compression

The escape hatch is to cut tokens without removing meaning. AST-preserving idiomatic transforms (comprehensions over loops, +=, f-strings) achieve 18–38% token reduction with no correctness loss on HumanEval — structural, not lossy. But even here, push too hard: reductions past 30% correlated with an 18.7% drop in unit-test pass rate. Validate against a test suite.