For any call whose result you don't need in the next second, the synchronous API is the wrong price. Defer it, and the same tokens cost half.
Why this, for you: you have a nightly eval, an embedding backfill, or a doc-refresh job
running at full sync list price — and a human looks at the output the next morning regardless.
You are paying premium responsiveness for work that nobody is waiting on. The "right time" lever recovers that.
The other three rights cut what you send and which model reads it. This one is about when:
the provider's GPU fleet is bursty, and work you let it defer into a capacity trough bills at marginal cost — a
~50% discount you pay for in latency, not quality.
1 Two cost primitives: batch and flex
Both major providers ship the same primary trade — defer the result, halve the bill — plus an interactive variant:
Tier
Discount
Turnaround
Anthropic Message Batches
50% off sync Messages
most complete <1h; 24h expiry
OpenAI Batch
50% off sync
24h completion SLA
OpenAI Flex
billed at batch rates
real-time but slower & pre-emptible
Batch is asynchronous: submit a job, poll, collect results within a day. Flex is the closest thing on the
market to an interactive latency-tiered discount — real-time, but slower and pre-emptible, returning a
429 "resource unavailable" (without charge) when capacity is short. There is no dynamic, spot-style per-token
pricing shipped by any major provider as of mid-2026; design for the tiers that exist.
The 24-hour SLA is a price ceiling, not a deadline. The provider promises to
serve eventually, not promptly — expect long tails and outright expirations under demand-driven slowdown.
2 One question picks the tier
Match workload to tier by asking exactly one thing: how long can this call wait without breaking
something downstream? It is not about volume — a single high-cost overnight eval belongs in batch the
same way 50,000 embedding requests do.
When one batched output feeds the next call's prompt, pipelining serializes the latency.
Two chained one-hour batches take two hours minimum — with a long tail if either job
slips toward the 24h expiry. Keep batched stages independent, or stay synchronous for the chain.
3 When the discount backfires
The 50% saving is not free — it buys you a submit-poll-retry-resubmit pipeline plus the operational overhead of
monitoring expirations and stuck "finalizing" states. Several conditions push the breakeven the other way:
Interactive loops — any path where a human or another agent waits; even Flex's extra seconds break a tool-calling loop.
Hard sub-24h deadlines — an eval that must land before a 9am standup can't absorb the expiration tail.
Volume below the crossover — for low-hundreds-of-dollars monthly spend, the discount saves less than a day of building the queue logic.
Caching-dominated work — when 80%+ of input tokens are cache hits on the sync path, tightening cache hit-rate is the cheaper lever.
Rule of thumb: route to batch only when the workload is both deferrable and
high-volume enough that the 50% saving covers the queue-management pipeline. Otherwise get sync cheaper
first — through caching, smaller models, or lean context.
# Nightly doc-refresh: 5,000 summaries, nobody waits → batch
batch = client.messages.batches.create(requests=[
{"custom_id": doc.id, "params": {... "messages": [...]}}
for doc in docs
])
# Poll batch.id until status == "ended"; re-batch any "expired" requests# 50% off; runs overnight in <1h typical; pipeline amortized across future jobs
↪ Your win: trade latency you have for a discount you want
Ask one question — does anything wall-clock-sensitive consume this output? If no, sync is overpriced.
Batch the deferrable — overnight evals, backfills, doc refreshes go to a 50%-off async tier.
Reach for Flex when work tolerates seconds-to-minutes but isn't a 24h job — and handle the 429.
Treat the 24h SLA as a ceiling — build resubmission for expirations, don't assume promptness.
Check the crossover — below it, tighter caching beats the queue-management overhead.
Retrieval practice — recall, don't peek
Question 1The single question that picks the temporal tier is…
Question 2The 24-hour batch SLA is best understood as a…
Question 3Chaining one batched output into the next call's prompt causes the latency to…
Question 4OpenAI Flex differs from Batch in that it is…
Question 5 · spaced recall from an earlier lessonFrom Right Cache: a tokenizer swap mid-deploy will…
Ask me anything. Want help deciding whether a specific job clears the batch crossover, or how to
structure resubmission for the expiration tail? Next in Part 2: Effort & Budget Scaling — spending
reasoning tokens only where they change the answer.