Agent Evals Cost: Per-Run and Monthly Pricing Breakdown

One run of a 12-step agent eval costs anywhere from $0.00 to $0.97, depending entirely on which model you point the eval harness at. That's the real spread across the 12 models we priced for a standard reference trace: 12 steps, a 4,000-token system/tool prefix, a 500-token task description, and each step adding a 300-token tool call plus a 1,500-token tool result. That trace totals 172,800 input tokens and 3,600 output tokens by the time step 12 finishes, because every step re-sends the growing transcript as input.

Uncached, GPT-5.5 runs $0.97 per trace, GPT-5.4 runs $0.49, Claude Opus 5 runs $0.95, Claude Sonnet 5 runs $0.38, Gemini 3.1 Pro Preview runs $0.39, Grok 4.6 runs $0.37, GPT-5.4 mini runs $0.15, Claude Haiku 4.5 runs $0.19, Gemini 3.8 Flash runs $0.14, GPT-5 Mini runs $0.05, DeepSeek V4 Pro runs $0.04, and DeepSeek V4 Flash rounds to $0.00. With prompt caching on the repeated prefix and transcript, those numbers fall to $0.30, $0.15, $0.29, $0.11, $0.12, $0.14, $0.05, $0.06, $0.04, $0.02, $0.01, and $0.00 respectively. If your eval suite runs thousands of these traces a month, the gap between $0.97 and $0.04 per run is the difference between a four-figure and a five-figure monthly bill.

The reference trace we're quoting isn't arbitrary — it mirrors what a typical agent eval actually does. Most automated eval harnesses (whether you're scoring tool-use accuracy, task completion, or regression against a golden dataset) replay an agent through a fixed number of steps, re-sending the growing conversation as context each time. We used 12 steps because that's a common depth for multi-step agent tasks: enough to exercise planning, tool calls, and retries without ballooning into a research-agent-scale trace.

The assumptions: a 4,000-token prefix (system prompt plus tool schemas), a 500-token task description, and at each step a 300-token tool call plus a 1,500-token tool result fed back in. Input at step k works out to 4,500 + 1,800 × (k − 1) tokens, so by step 12 you're sending a transcript well over 20,000 tokens as input for that single step. Summed across all 12 steps, total input is 172,800 tokens and total output is 3,600 tokens. If your real eval traces are shorter or your tool results are smaller, your actual cost will be lower than these numbers — this is a reference point, not a universal constant, so use the AI cost calculator to plug in your own step count and token sizes.

GPT-5.4 prices at $2.5 per 1M input tokens, $15 per 1M output tokens, and $0.25 per 1M cached input tokens. Running the 172,800 input tokens and 3,600 output tokens through those rates lands at $0.49 per uncached run and $0.15 per run once the repeated prefix and transcript are served from cache — that's the pre-computed figure in the pricing table, not something you need to re-derive, but it shows why caching matters: most of those 172,800 input tokens are the same prefix and prior steps repeated step after step, which is exactly what prompt caching is built to discount.

DeepSeek V4 Pro tells the same story at a different scale: $0.2088 per 1M input tokens, $0.4176 per 1M output tokens, $0.0174 per 1M cached input. Its 12-step run costs $0.04 uncached and $0.01 cached. The ratio of uncached to cached cost is similar between the two models, but the absolute gap between GPT-5.4's $0.49 and DeepSeek V4 Pro's $0.04 is what actually moves a monthly invoice. If you're running the same eval suite nightly, that per-run gap compounds fast — see the dataset and run-rate scenarios below.

Scale the per-run numbers to real eval volumes and the differences become budget line items. At 1,000 runs a month, GPT-5.5's $0.97 per run uncached and DeepSeek V4 Pro's $0.04 per run are both manageable in absolute terms, but the per-run gap is already large. At 10,000 runs, that same $0.97 rate and $0.04 rate diverge into materially different budget categories. At 100,000 runs — common for a dataset-driven suite running 10,000 examples across multiple agent seeds — the per-run figure stops being a rounding error and becomes the dominant cost of the eval program. Use the AI cost calculator to turn your exact run count into a dollar total rather than estimating by hand.

Caching changes the math further. GPT-5.5's cached rate of $0.30 per run versus its uncached $0.97 is a clear reduction, and the same pattern holds across the table: Claude Opus 5 goes from $0.95 to $0.29, Gemini 3.1 Pro Preview from $0.39 to $0.12, Grok 4.6 from $0.37 to $0.14. If your dataset re-runs the same prefix and task description across many seeds or many examples that share a system prompt, caching isn't optional — it's the difference between a sustainable eval cadence and one you have to ration.

Running evals in parallel doesn't change the per-run cost quoted in the table — each run still costs what the table says — but it changes how fast you hit rate limits and how predictable your monthly bill is. A nightly CI job running 100 eval traces a night at Claude Sonnet 5's $0.38 uncached per-run rate accumulates that rate 100 times a night in direct model spend, before any orchestration overhead. A pre-release sweep running 10,000 traces at that same $0.38 per-run rate accumulates the same rate 10,000 times, concentrated into whatever window your pipeline allows.

The cost lever here isn't really parallelism itself — it's which model you point the sweep at, and whether you cache. Running that same 10,000-trace pre-release sweep on Claude Sonnet 5 cached at $0.11 per run instead of $0.38 uncached multiplies out to a much smaller total, using the exact per-run figures in the table — plug your own run count into the AI cost calculator to see the exact monthly total rather than estimating. If your CI pipeline reruns evals on every merge rather than nightly or pre-release, multiply accordingly — the nightly-vs-sweep split is about frequency, and frequency is the multiplier that turns a cheap per-run cost into a real monthly number.

Four levers move an eval bill: model choice, caching, sample rate, and step-level model routing. Model choice is the biggest lever by far — the table shows a wide spread between GPT-5.5's $0.97 uncached run and DeepSeek V4 Pro's $0.04 run, and if your eval only needs a pass/fail signal rather than frontier reasoning quality, a cheaper model like GPT-5 Mini at $0.05 per run or Gemini 3.8 Flash at $0.14 per run may score your agent just as reliably.

Caching is the second lever and it's nearly free to implement if your harness replays the same prefix across runs: every model in the table shows a meaningful reduction from uncached to cached, for example GPT-5.4 mini's $0.15 uncached to $0.05 cached. Sample-rate reduction — running your full dataset against a cheap model and only re-running a stratified sample against an expensive model for validation — lets you combine DeepSeek V4 Pro's $0.04 run for the bulk of a 100,000-run sweep with Claude Opus 5's $0.95 run on a held-out slice. Step-level routing (cheap model does early steps, expensive model only validates the final answer) isn't captured directly in these reference-run numbers since they assume one model for all 12 steps, but it's the same principle: pay the premium rate only where it matters.

The reference trace assumes a fixed 4,000-token prefix and 1,500-token tool results, but real tool outputs vary — a web search result or a file read can be far larger, and every extra 1,000 tokens of tool output gets multiplied by however many steps follow it, since each later step resends the growing transcript. Long-context edge cases in your eval dataset (a task that needs 20 steps instead of 12, or a tool result that's 5,000 tokens instead of 1,500) will push your real per-run cost above anything in this table, so treat these numbers as a floor for your own trace shape, not a ceiling.

Orchestration overhead is the other hidden cost: if your eval harness runs through a multi-agent orchestrator rather than a single model call per step, you're paying the orchestrator's token overhead on top of the model's per-token rate — see the multi-agent system cost guide for how much orchestrators typically add. Dataset storage, result logging, and CI runner time aren't priced in this table at all; they're real but separate from the per-token model cost this guide addresses. Budget for them separately rather than assuming the per-run figures above are your total eval bill.

Reference trace: 12 steps, 4,000-token prefix, 500-token task, each step adds a 300-token tool call and a 1,500-token result. Input at step k is 4,500 + 1,800 × (k − 1); total 172,800 input tokens and 3,600 output tokens.

Use the AI cost calculator to plug in your own step count, prefix size, and run volume rather than relying on the reference trace assumptions if your eval shape differs materially from 12 steps.

A research lab iterating daily on agent behavior wants fast feedback over raw accuracy, and typically reruns the same small eval set many times a day. DeepSeek V4 Pro at $0.04 uncached or $0.01 cached per run keeps that loop cheap enough to run constantly without a budget conversation. A mid-size team running nightly CI with 100 eval traces a night is better served by a mid-tier model like Claude Sonnet 5 ($0.38 uncached, $0.11 cached) or Gemini 3.1 Pro Preview ($0.39 uncached, $0.12 cached) — enough quality to catch real regressions, cached to keep nightly spend predictable.

An enterprise running a pre-release sweep of 10,000 traces needs the highest-fidelity signal it can afford, which points toward GPT-5.5 ($0.97 uncached, $0.30 cached) or Claude Opus 5 ($0.95 uncached, $0.29 cached) for the traces that gate a release, while routing the bulk of exploratory runs to a cheaper model. In every archetype, caching the repeated prefix and transcript cuts the bill substantially regardless of which model you pick — it's the one lever that doesn't force a quality tradeoff. Run your own run counts through the AI cost calculator to size each archetype's exact monthly total.

Pay for DeepSeek V4 Pro ($0.04 per run) or DeepSeek V4 Flash (effectively $0.00 per run) when you're iterating many times a day and need a cheap, fast signal rather than a definitive one — these are the right defaults for inner-loop development evals. Pay for GPT-5 Mini ($0.05 per run) or Gemini 3.8 Flash ($0.14 per run) when you want a slightly stronger signal than the cheapest tier without approaching the cost of frontier models. Reserve GPT-5.5 ($0.97 uncached, $0.30 cached) or Claude Opus 5 ($0.95 uncached, $0.29 cached) for gating decisions — pre-release sweeps, final regression checks — where the cost of a false pass is higher than the cost of the eval itself.

A simple budget rule: multiply your expected monthly run count by the per-run cost in the table (cached, if your harness supports it) to get your floor estimate, then add a margin for dataset-driven suites that re-run across multiple agent seeds. If that number surprises you, it's usually cheaper to switch the eval model than to run fewer evals — evals you skip don't catch regressions. Use the AI cost calculator to model your own step count, run volume, and model choice before committing to a monthly eval budget, and note that SERP and search-demand data for this specific query weren't available when writing this guide — track real traffic after publish to see which scenarios readers care about most.