Pay for an agent only where steps are unpredictable

This is for the person who signs off an AI project and has to defend what it costs a year later. The choice is usually presented as a product decision. It is an architecture decision, and there are two options.

One kind of system works out what to look up before the model writes a word. Everything it will fetch is settled in advance, so its cost and its response time are known before you ship. The other lets the model decide what to look up while it works, one step at a time, and neither number is known until the run ends. The first is usually called a retrieval pipeline. The second is usually called an agent.

This is the companion to our RAG architecture paper, one level up. That paper is about the quality of what gets retrieved. This one is about the code wrapped around the model that decides whether to retrieve at all.

A pipeline is predictable by design. An agent is not.

Two systems can answer the same customer question and behave nothing alike once they are live. The predictable one has a fixed shape: take the question, fetch documents, sort them, write an answer. One decision, made by you, before anything ships. The other hands that decision to the model on every step, and the number of steps is whatever the model chooses on the day.

Conditional lookups, rewording a search that came back empty, mixing several sources, a variable number of steps: none of these are separate features you buy one at a time. All of them are consequences of moving a single decision out of your code and into the model.

The common version of this argument is wrong in a way worth fixing, because vendors sell against it. A pipeline is not limited to one lookup. It is limited to one decision. Splitting a question into parts, or running a fixed two-stage fetch, are ordinary pipeline moves. What a pipeline cannot do is choose at run time how many more lookups to make based on what it has just read.

The part a buyer feels later is measurement. In a pipeline the same question always fetches the same documents, so you can score that step on its own, and when quality drops you know which part of the system to look at. In an agent the retrieval quality you observe depends on the queries the model wrote, which depend on the model, its instructions, the description of the tool, and every result that came before. Everything is entangled with everything. That is why the BrowseComp-Plus authors built their benchmark at all: they state that current evaluations compare a whole system at a moment in time and do not permit well-controlled experiments.

A tool is something the model can ask for but cannot do itself: run a search, read a customer record, send an email. The model only ever asks. Your code decides whether to allow it, runs it, and hands back the result. Anthropic puts the division of labour in one sentence: the model emits the request, your code handles it. Everything expensive in the rest of this paper sits on your side of that line.

Doubling the number of steps roughly triples the bill

Every step an agent takes re-sends everything that came before it, because the interface keeps no memory between calls. Step nine pays for steps one through eight all over again. So the bill grows faster than the number of steps, but slower than the square of it: going from eight steps to sixteen multiplies it by about 3.2, not by two.

The formula appears once, for the reader who wants it. Billed input across n steps is S(n+1) + (a+r)·n(n+1)/2, where S is the fixed part sent every time, r is a step’s tool result and a is the model’s output for that step. The most common mistake in this topic is reasoning about the final size of the conversation rather than the cumulative resend.

Worked at published list prices, and labelled as arithmetic rather than as anything measured. Models are billed by the token, which is roughly three quarters of a word, counted both on what you send and on what comes back. One question and answer through a pipeline is about 5,700 tokens in and 500 out, which costs about $0.0164 on Claude Sonnet 5, the model every price here is worked at. An eight-step loop with a 1,300-token fixed prefix, 1,500-token tool results and 250 tokens of output per step bills 74,700 input and 2,600 output: about $0.175, or 10.7 times the pipeline call. Sixteen steps is $0.566. Twenty-four steps is about $1.18, or 72 times. Put your own volume against that before you read on. At a thousand questions a day it is roughly $6,000 a year for the pipeline, $64,000 for the eight-step agent and $431,000 for the twenty-four-step one. The multiple is the point; the line item is what you sign.

Caching blunts that without removing it. Repeated text is billed at a tenth of the normal rate, which takes the eight-step figure from 10.7 times down to about 4.6, and the twenty-four-step figure from 72 to about 16.9. The section on request layout, further down, is about the ways that discount silently stops applying.

Treat the arithmetic as a sizing tool and nothing more: close enough to size an argument on a fixed model, nowhere near close enough to quote a price. BrowseComp-Plus, a public set of 830 hard research questions, publishes total cost per question for three different models: $0.108 for gpt-4.1 at 8.67 searches, $0.435 for GPT-5 at 21.74 searches, and $2.22 for Opus 4 at 10.24 searches. The arithmetic above runs about 1.6 times high against the gpt-4.1 row at a similar number of searches, and roughly nine times low against Opus 4, because it is priced on one model’s rates throughout.

Waiting time behaves the same way. The steps are serial by construction, since each one’s query depends on the last one’s result. Tools inside a single step can run together; the steps themselves cannot, so streaming does not hide the wait the way it does for a pipeline. Nothing the user can act on arrives until the run stops. Set that against the per-stage budget in the RAG paper and it becomes a product decision rather than an infrastructure one.

Most questions do not need an agent

The extra steps buy something only where nobody can say in advance how many are needed. Where the answer sits in one document they buy almost nothing, and a badly aimed lookup can score worse than no lookup at all.

Search-R1 put the same questions through three set-ups, using one model and one search index throughout, so the only thing changing across the columns is who decides what to look up. Two cautions before you read it. The third column was also specially trained to search as it works, so it prices that training and the extra steps together rather than the steps alone. And two of the question sets, NQ and HotpotQA, were used in that training, so the model has effectively seen the exam. The other five it had not, and those are the rows that carry the argument.

Every figure is the share of questions answered exactly right, from 0 to 1. 0.408 means 41 in every 100. Looking up one popular fact gains 9%. Chaining several facts together gains 238%. Then read the 2Wiki row, which is the sharpest datum against a naive single lookup anywhere in the published record: one lookup scored 0.235 where the same model with no lookup at all scored 0.250. A single badly aimed fetch fills the page with confident, wrong material.

Agents also lose, and not narrowly. Ferrazzi et al., Is Agentic RAG worth it? (arXiv:2601.07711), found agent routing collapsing on fact verification: a user-intent F1 of 64.6 on FEVER against 87.9 for a well-built pipeline, and below the 66.7 you get from doing nothing clever at all. It also lost on putting results in the right order, scoring 43.9 against 49.5 on nDCG@10, which measures how well the best answers land near the top, while costing 3.3 times the input tokens.

An allocation rule falls out of both results. Give the agent the question and the choice of where to look. Keep the ordering of results in the pipeline.

Better search makes an agent cheaper, not only more accurate

The improvement teams reach for last is usually the one that pays. Not a bigger model or a longer leash: better search underneath. On BrowseComp-Plus, swapping a keyword index for a modern one under the same agent raised accuracy from 55.90% to 70.12% and cut the cost of the run from $400.36 to $360.71.

The bill fell because the agent searched less: 23.23 searches down to 21.74. Set that against what it had been doing before the swap. A keyword index used once reaches 13.7% Recall@1000 on that corpus, which is to say it mostly misses. GPT-5 driving the same weak tool over 23.23 searches recovers 61.70% evidence recall and 55.90% accuracy. The extra steps were repairing bad search at full price, every time.

The ceiling should end the architecture argument. Handed all the right documents directly, a non-reasoning model scored 93.49% against 14.58% with keyword search. Neither architecture is the constraint when retrieval is right.

So there is one question to ask before agreeing to pay for steps: what does the search underneath score on your own documents? The RAG paper’s corpus measurement and evaluation sections are where that work is written down.

A run that gave up looks like one that finished

An agent can stop for four reasons and only two of them produce an answer. If the code downstream treats all four alike, a run that hit its spending limit reaches the customer looking exactly like a finished result.

The four: the model decided it was done, it called an explicit finishing tool, it ran out of budget, or the system stopped it through a denied permission, a policy block or a crash.

Anthropic’s own toolkit is worth copying here. A run that finished carries an answer. A run that stopped early carries no answer at all, rather than an empty one, so the code physically cannot confuse the two. Most systems built on top throw that distinction away at their own edge, which is where it was doing the work.

One fact belongs to whoever owns the monitoring dashboard. When the model refuses a request, the refusal arrives as a completely successful response with the refusal written inside it. Standard monitoring only checks whether the call succeeded, so the dashboard stays green straight through a refusal.

The design opinion worth having at a review is about what running out of budget should do. Where one agent calls another, a sub-agent that runs out of budget should be able to file a partial finding rather than kill the whole run. AISI Inspect does this, and almost nobody copies it.

A vague error message costs you the answer

When a tool fails, the text it returns is not a log line. It is the next thing the model reads, and its only evidence about what to do next. “Failed” ends the run. “Rate limit exceeded. Retry after 60 seconds.” does not.

The intuition about retries is wrong in a way that costs money. Anthropic’s tool-use documentation states that Claude retries an invalid call two or three times with corrections before apologising to the user. That budget is spent repairing the shape of the call, never recovering from a failed execution. A malformed call the model can fix, it fixes. A well-formed call whose execution failed is read once and reported straight to the user. A vague error does not burn three attempts; it skips to the apology on the first one.

Two rules follow, and both are checkable at a code review. Return the error type and message rather than the whole stack trace, because a traceback is thousands of tokens that teach the model nothing about what to do differently, which is exactly what Anthropic’s SDK tool runner does deliberately. And every request the model made gets a reply, including the ones a batch never reached, where “Not executed: the preceding write failed.” is a sentence the model can act on and a missing reply is a protocol error.

type ToolOutcome = {
  ok: boolean;
  kind: "result" | "error" | "budget_exceeded" | "denied" | "skipped";
  retryAfterMs: number | null;
  truncated: { of: number; kept: number } | null;
  body: string;
};

// One envelope, one failure grammar, rendered in one place.
function render(o: ToolOutcome, toolUseId: string) {
  const lines: string[] = [];
  if (!o.ok) lines.push(o.kind.toUpperCase() + ".");
  if (o.retryAfterMs !== null) {
    lines.push("Retry after " + Math.ceil(o.retryAfterMs / 1000) + " seconds.");
  }
  if (o.truncated !== null) {
    lines.push(
      "Showing " + o.truncated.kept + " of " + o.truncated.of +
        " results. Narrow the query rather than paging through this one.",
    );
  }
  lines.push(o.body);

  return {
    type: "tool_result",
    tool_use_id: toolUseId,
    is_error: !o.ok,
    content: lines.join("\n"),
  };
}

The protocol underneath has two channels for failure, and one sent down the wrong one never reaches the model at all. An unknown tool name is a transport error the model never sees. An execution failure has to come back as an ordinary result flagged as an error. Without one envelope like the above, the prompt is made of whatever each library’s exception happens to stringify to.

Every tool you add makes the others harder to choose

Somewhere between twenty and fifty available tools the model starts picking the wrong one, and the two major vendors put the line in different places. Anthropic states that tool selection degrades once you exceed 30 to 50 available tools. OpenAI recommends fewer than 20 available at the start of a turn. Deleting tools is the cheapest improvement in this paper.

The cost lands before any work happens. Anthropic’s worked example is a GitHub, Slack, Sentry, Grafana and Splunk setup spending roughly 55k tokens on tool definitions before the model does anything, cut by over 85% by putting a search in front of them instead. The accuracy gain from that reduction alone is published: on their MCP evaluation, Opus 4 moved from 49% to 74% and Opus 4.5 from 79.5% to 88.1%.

Their switch-over criteria are usable as written: ten or more tools, definitions over 10k tokens, selection accuracy dropping, or several aggregated servers. So is the negative. Under ten tools, or every tool used on every request, and a search over them is pure overhead.

One line from Anthropic’s guidance belongs on a review checklist, and it needs no technical vocabulary at all. If a human engineer cannot say definitively which tool should be used in a given situation, an agent will not do better.

Tool replies are the largest uncontrolled input in the system. Claude Code caps them at 25,000 tokens by default, and Anthropic’s worked Slack example measures a detailed reply at 206 tokens against 72 for a concise one, chosen by the model per call. A tool that returns everything it knows is an attack on your own context window.

Whether a tool is safe to run alongside others is a property of that tool, not a switch on the orchestrator. Reads run together; anything that writes runs one at a time. That is also the argument for promoting an action out of a general shell into a tool of its own, in Anthropic’s framing: a system that sees only shell commands cannot tell a parallel-safe search from a parallel-unsafe push, so it must serialise everything.

The same request can cost ten times more without warning

Anthropic bills repeated text at a tenth of the normal rate, but only when the front of the request is byte-for-byte what it was last time. Change one byte near the front and you pay full price for everything after it. No error is raised and nothing appears in the logs.

The mental model is a single long strip, matched from the front, so the order of what you send decides what it costs. Tool definitions first, standing instructions next, the conversation last, and anything that changes often belongs at the end. Reads cost a tenth of the base input rate, a five-minute save costs 1.25 times and a one-hour save twice, so the short option pays back on the second request and the long one on the third.

Two failure modes cost real money and report nothing. The five minutes runs from when the request starts rather than when it finishes, so a four-minute step leaves about one minute of the saved copy still valid.

The second one catches teams. Anthropic’s prompt-caching documentation states that the bookmark which finds the saved copy searches back only twenty blocks. The conversation is stored as a stack of blocks, one for each thing said or returned, so a single step that fires eleven tools at once adds twenty-two in one go. The next request looks back twenty, finds nothing, pays full price on the entire conversation and raises no error. The only evidence is the amount being written to the cache sitting near the full conversation size on every request, which is why the ratio of cache reads to cache writes belongs on a dashboard. The fix is an extra bookmark roughly every fifteen blocks.

Timing surprises people too. An entry becomes readable only once the first reply starts arriving, so launching ten sub-agents at once means ten full-price requests, none of them reading what the others are still writing. Start one, wait for its first token, then launch the rest.

A bigger window is not the way out of any of this. Chroma’s eighteen-model study found a single distracting document already lowering accuracy and, on LongMemEval, a focused 300-token prompt beating the same task at 113k tokens across every model family. The commercial reason for restraint has gone, since Claude 4.6 and later carry the full 1M window at standard per-token pricing with no multiplier. That is exactly why the restraint now has to come from somewhere else.

Assume it gets fooled, then decide what it can reach

Whether an agent can be tricked by text it reads is settled: it can. The question your security review will actually ask is what a successful trick can reach. Which data, which systems, and whether anything can be sent out of the building.

The published record is one-sided. Nasr et al., The Attacker Moves Second(arXiv:2510.09023), with authors at OpenAI, Anthropic and Google DeepMind, took twelve published defences whose own papers reported near-zero attack success and broke most of them above 90%; a 500-participant human red team reached 100% on every system tested. Zhan et al. (arXiv:2503.00061, NAACL 2025 Findings) broke eight indirect-injection defences above 50%. OpenAI’s December 2025 position compares prompt injection to scams and social engineering and says it is unlikely ever to be fully solved, which costs the vendor something to say and is therefore worth repeating.

What works is structural rather than persuasive. Beurer-Kellner et al., Design Patterns for Securing LLM Agents against Prompt Injections (arXiv:2506.08837), state that once an agent has read untrusted input it must be constrained so that input cannot trigger a consequential action. CaMeL prices that guarantee at roughly seven points of task completion on AgentDojo: 77% solved with provable security against 84% undefended. That is the trade a security review is really asking you to approve.

Four properties follow, in order of how much they buy. Never let private data, untrusted text and a route out of the building coexist in one agent, which is the combination Simon Willison named the lethal trifecta. Sandbox files and network together, because either alone leaves a path open. Enforce permissions in the system being called rather than in the instructions, per OWASP’s excessive-agency guidance. And keep untrusted text inside tool replies rather than promoting it into the instructions.

A permission model enforced by the code is real. The same rules written into a prompt are a preference.

Reliable once is not reliable four times running

A demo succeeds once. A production system has to succeed on the same task every time a customer asks. Those are different numbers, and the second is the one you are committing to.

What changes is the unit of measurement: a distribution rather than a score. On the tau-bench airline leaderboard the same model scored 0.460 when a task was attempted once and 0.225 when the same task had to succeed four times in a row. Half the headline number disappears by the fourth identical attempt. Those figures come from a 2024-era model and are useless as a capability claim; they are exactly right as a structural one. An agent making dozens of calls that each fail independently at 2% is useless end to end.

The escape hatch teams reach for first does not exist. Temperature 0 is not deterministic, and Thinking Machines Lab’s Defeating Nondeterminism in LLM Inference found 1,000 identical temperature-0 requests to Qwen3-235B-Instruct producing 80 distinct completions, diverging at token 103. You cannot pin your way out of variance, so measure it.

Four numbers a pipeline gate does not have: how many steps a task takes at the median and at the worst 5%, the share of runs that stopped on a budget rather than on an answer, cost per completed task, and the ratio of cache reads to cache writes.

The order we build it in, which is the reverse of how it usually goes:

1. A scripted task set and a pass mark, before the loop
2. The wording every tool uses when it fails
3. The four exits, with a chosen behaviour for each
4. The request layout, with cache reads against cache writes as a standing signal
5. The tool surface: consolidate, then defer, then search over the rest
6. Budgets on turns, tokens, money and clock, alarmed on the worst 5% rather than the average, because published runs on the same corpus with the same tool span 8.67 to 21.74 searches
7. Sandbox and permissions, before the first tool that writes anything
8. Sub-agents last, and only for reading

A recommendation is only honest if it says what would overturn it. Route by question type instead of replacing the architecture, if the multi-step share of real questions turns out to be small. Keep the ordering of results in the pipeline unless the agent beats it on your own set. Move a budget breach from stop to report-back wherever a partial finding has value to the caller.

The first of those extends the RAG paper’s final section from the other end. That one abandons retrieval when the questions turn out to be exact-identifier lookups. This one abandons the extra steps when the questions turn out to need a single decision.