Summarise · 3 checks · max 400 tokens · benched 2026-09-16

Claude Opus 5 vs Claude Sonnet 5 vs Claude Sonnet 4.6 vs Claude Haiku 4.5 on one-sentence changelog summary

Anthropic models side by side on "One-sentence changelog summary": Claude Haiku 4.5 scores 9/10; Claude Haiku 4.5 is the cheapest answer scoring 8+ at $0.33 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

User

Summarise this release note in one sentence of at most 30 words. Output only the sentence.

Release 4.7.0: The public API now enforces a rate limit of 600 requests per minute per key (previously unlimited). Requests over the limit return HTTP 429 with a Retry-After header. Webhook deliveries are now signed with HMAC-SHA256 and the old unsigned mode will be removed in 4.9. We also fixed a bug where exporting more than 10,000 rows produced an empty file.

Rubric for the judge: A single accurate sentence that prioritises the rate limit and signed webhooks; no bullet list, no preamble.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Claude Opus 5

anthropic/claude-opus-5

9/10

Latency: 1.5s
Cost: $0.00302
Per 1,000: $3.02

169 in · 87 out · 27 words · checks 2/3

max_words
contains
regex

Release 4.7.0 adds a 600-requests-per-minute API rate limit returning HTTP 429, signs webhooks with HMAC-SHA256 (unsigned mode removed in 4.9), and fixes empty exports above 10,000 rows.

Judge: Accurate single sentence under 30 words (28), prioritizes rate limit and signed webhooks as required, no preamble or bullet list, minor: could emphasize rate limit enforcement more prominently.

Claude Sonnet 5

anthropic/claude-sonnet-5

9/10

Latency: 1.1s
Cost: $0.00110
Per 1,000: $1.10

169 in · 76 out · 25 words · checks 2/3

max_words
contains
regex

Release 4.7.0 adds a 600 req/min API rate limit with HTTP 429 responses, HMAC-SHA256 webhook signing (unsigned mode deprecated), and fixes large-export empty file bug.

Judge: Accurate single sentence under 30 words (28), prioritizes rate limit and webhook signing as required, no preamble or bullets, though 'req/min' abbreviation is slightly informal.

Claude Sonnet 4.6

anthropic/claude-sonnet-4.6

9/10

Latency: 795ms
Cost: $0.00097
Per 1,000: $0.97

128 in · 39 out · 21 words · checks 3/3

max_words
contains
regex

API rate limiting (600 req/min) and HMAC-SHA256 webhook signing are introduced, unsigned webhooks deprecated, and a large export bug is fixed.

Judge: Accurate single sentence under 30 words (24 words) that prioritises rate limiting and signed webhooks as required, with no preamble or bullet list.

Claude Haiku 4.5

anthropic/claude-haiku-4.5

9/10

top judge scorecheapest scoring 8+

Latency: 746ms
Cost: $0.00033
Per 1,000: $0.33

128 in · 40 out · 17 words · checks 2/3

max_words
contains
regex

Public API now rate-limited to 600 requests/minute; webhooks use HMAC-SHA256 signing; export bug for 10,000+ rows fixed.

Judge: Accurate single sentence under 30 words (21 words), prioritizes rate limit and signed webhooks as required, no preamble or bullet list, minor: could emphasize the enforcement aspect more explicitly.

Frequently asked

What does this prompt test?

Summarise: A single accurate sentence that prioritises the rate limit and signed webhooks; no bullet list, no preamble. The deterministic checks are max_words, contains, regex.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Claude Haiku 4.5 at $0.33 per 1,000 runs. If you need the top score, Claude Haiku 4.5 at $0.33 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Anthropic with your own OpenRouter key, or tweak the wording and see what changes.

Side by side

Claude Opus 5

Claude Sonnet 5

Claude Sonnet 4.6

Claude Haiku 4.5

Frequently asked

Same prompt, other families

Stop overpaying for AI tools