Cost and Latency

Prerequisites: The Agent Loop, Context Engineering — caching is context arrangement with money attached. Feeds problems: Cost and latency at scale, The context bottleneck.

Practitioner

Do the arithmetic once and you’ll never unsee it. An agent averaging 2,000 new tokens per turn re-sends everything prior on every call: by turn 20 it has processed roughly 400,000 cumulative input tokens — twenty times what a naive estimate suggests. That quadratic-ish growth is a property of the loop itself, which is why cost is an architecture decision. The levers, in order of leverage:

Prompt caching first. Providers cache the computation for context prefixes the model has already read; a turn whose transcript matches a cached prefix pays a small fraction — commonly around a tenth — of the input price, and starts faster. An agent transcript is naturally one growing prefix, so caching should be nearly free to exploit. In practice teams break it by accident, because the rule is exact: the cache matches byte-for-byte from the top, and the first divergent byte invalidates everything after it. The classic self-inflicted wounds:

• A timestamp or request ID interpolated into the system prompt — every call is a cache miss, at the very top, invalidating everything.
• Tool definitions that reorder nondeterministically (serialize them stably).
• Compaction that rewrites early history — every mid-transcript edit forfeits the cache from that point. Compaction still pays; just batch it into rare, deliberate events rather than continuous trimming, and know each one buys a one-turn cost spike.

Design rule: stable things first (system prompt, tools), append-only history after, dynamic values in the latest message only. Then verify — providers report cached vs. uncached counts per response, and a healthy agent shows cache hit rates above 80%. This one lever, done right, is the difference between an agent that costs $0.40 a task and the same agent at $2.50.

Send fewer tokens. Everything from Context Engineering doubles as cost work: tool results returning distillates instead of dumps, retrieval instead of pre-loading, subagents absorbing verbose exploration. The compounding is what makes it lucrative — a 3,000-token dump at turn 5 gets re-read (and re-billed, cached or not) every turn to the end.

Route by step difficulty. Inside one agent task, the steps differ wildly: “extract the date from this email” doesn’t need the model that plans the migration. Small models cost an order of magnitude (or two) less per token — route mechanical steps to them, reserve frontier models for judgment. But measure end-to-end: cost per completed task, not cost per token. A cheap model that fails 20% more forces retries and human cleanup that erase the per-token savings. Your eval suite is what makes routing decisions honest — run it per candidate model, divide spend by passes.

Attack latency’s real source. Users experience wall-clock, and in agents it’s dominated by serial round-trips: fifteen sequential tool calls at a few seconds each is a minute of waiting, whatever the model’s speed. Fixes in order: let the model issue independent tool calls in parallel (modern models do this when tool descriptions make independence obvious), collapse chatty tool sequences into one round trip, cache reads, and stream visible progress — an agent narrating “reading the failing test…” feels half as slow at identical latency, which is sometimes the cheapest fix that ships.

Meter from day one. Log tokens, cache hits, turns, and dollars per task, tagged by task type. The distribution will surprise you: typically a fat tail of doom-looping outliers burning a third of total spend — which turns budget caps from The Agent Loop into a line item you can price.

Expert pointers

Inference prices have fallen steeply and steadily for equivalent capability, which reshapes the build-vs-wait calculus: optimizations tied to today’s price sheet (elaborate routing cascades, aggressive quantization) can be obsolete in two quarters, while structural ones (caching- friendly context layout, distillate-returning tools) survive every price cut — the six-months-from-now model, denominated in dollars. Live debates: small-model cascades vs. one strong model as defaults shift, batch APIs for offline agent work at half price, and on-device models absorbing the mechanical tier entirely.

Misconceptions

• “Optimize cost after it works.” The expensive part is the loop’s shape — context layout, tool verbosity, turn counts — and retrofitting a shape is a rewrite. Meter from the first prototype; optimize when it matters, but design cache-friendly from day one.
• “Caching just works.” Caching works when the prefix is byte-stable, and a single interpolated timestamp quietly turns it off. Unverified caching should be assumed broken.
• “Latency is model speed.” It’s mostly your round-trips. The fastest model behind fifteen serial tool calls still feels slow; parallelism and perceived-progress beat model swaps.

Check yourself

1. An agent’s cache hit rate is 4%. Name the three most likely causes from this topic and the one-line fix for each.
2. Your per-token cost dropped 60% after switching extraction steps to a small model, but monthly spend rose. Reconstruct what happened and name the metric that would have caught it.
3. Why does a 3,000-token tool result at turn 5 cost far more than 3,000 tokens? Roughly how much more, over a 40-turn task?
4. A stakeholder wants the agent “twice as fast” and proposes paying for a faster model. Using this topic, where do you look first, and what do you propose if the trace shows 18 serial tool calls?

Apply it

Turn on prompt caching for your Agent Loop agent (cache breakpoints or your provider’s equivalent) and log cached vs. uncached input tokens per turn. Run a 15-turn task and compute the bill with and without caching from the provider’s price sheet. Then sabotage it — add a timestamp to the system prompt — and watch the hit rate go to zero. Few exercises in this hub buy more permanent intuition per hour.