Summarise · 3 checks · max 400 tokens · benched 2026-09-16

Claude Opus 5 vs Claude Sonnet 5 vs Claude Sonnet 4.6 vs Claude Haiku 4.5 on summarise a support thread in 60 words

Anthropic models side by side on "Summarise a support thread in 60 words": Claude Haiku 4.5 scores 9/10; Claude Haiku 4.5 is the cheapest answer scoring 8+ at $0.61 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a concise assistant. Follow length limits exactly and output plain text only.

User

Summarise the following customer support thread in no more than 60 words. Mention what the customer wanted and how it was resolved.

---
Customer (Mon 09:12): Hi, I was charged twice for my Team plan on 1 September. Order references are 88213 and 88214. I only have one workspace. Can you fix this?
Agent (Mon 10:40): Sorry about that. I can see the duplicate charge. Could you confirm the last four digits of the card so I can raise the refund?
Customer (Mon 11:02): 4471.
Agent (Mon 11:30): Thanks. I have refunded order 88214 in full (USD 96.00). It should appear on your statement within 5 business days. I have also added a note so it cannot recur next cycle.
Customer (Mon 11:35): Great, thanks for the quick turnaround.
---

Rubric for the judge: Accurate, complete summary of the double charge and its refund, within 60 words, no leaked card digits.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Claude Opus 5

anthropic/claude-opus-5

2/10

Latency: 1.2s
Cost: $0.00436
Per 1,000: $4.36

307 in · 113 out · 57 words · checks 2/3

max_words
contains
not_contains

The customer was charged twice for their Team plan on 1 September (orders 88213 and 88214) despite having one workspace. After confirming the card's last four digits (4471), the agent refunded order 88214 in full (USD 96.00), due within five business days, and flagged the account to prevent a repeat charge next cycle. The customer was satisfied.

Judge: Output violates rubric by including leaked card digits (4471) which should never appear in a summary, despite explicit instruction against this.

Claude Sonnet 5

anthropic/claude-sonnet-5

9/10

Latency: 2.6s
Cost: $0.00160
Per 1,000: $1.60

307 in · 99 out · 43 words · checks 3/3

max_words
contains
not_contains

Customer was double-charged for their Team plan (orders 88213 and 88214). After verifying card details, the agent refunded the duplicate charge ($96.00, order 88214), expected within 5 business days, and flagged the account to prevent recurrence. Customer was satisfied with the quick resolution.

Judge: Accurate, complete summary covering the double charge, refund amount, timeline, and resolution within 60 words (exactly 54), with no card digits leaked; only minor: could have explicitly stated 'Team plan on 1 September' for full context.

Claude Sonnet 4.6

anthropic/claude-sonnet-4.6

9/10

Latency: 1.2s
Cost: $0.00185
Per 1,000: $1.85

232 in · 77 out · 52 words · checks 3/3

max_words
contains
not_contains

The customer was charged twice for their Team plan on 1 September (orders 88213 and 88214) despite having only one workspace. The agent confirmed the duplicate, refunded the second charge of USD 96.00 to the customer's card, and added a note to prevent recurrence. The refund should appear within five business days.

Judge: Accurate, complete summary covering the issue and resolution within 60 words (exactly 59), with no card digits leaked; minor deduction only for not explicitly stating customer satisfaction.

Claude Haiku 4.5

anthropic/claude-haiku-4.5

9/10

top judge scorecheapest scoring 8+

Latency: 974ms
Cost: $0.00061
Per 1,000: $0.61

231 in · 76 out · 46 words · checks 3/3

max_words
contains
not_contains

Customer was charged twice for their Team plan (orders 88213 and 88214). Agent identified the duplicate charge, verified the card details, and refunded order 88214 ($96.00). The refund should appear within 5 business days, and a note was added to prevent future recurrence. Issue resolved satisfactorily.

Judge: Accurate summary covering the double charge, refund amount, timeline, and prevention measure within 60 words (exactly 57), with no card digits leaked; only minor: could have explicitly stated 'Team plan on 1 September' for full completeness.

Frequently asked

What does this prompt test?

Summarise: Accurate, complete summary of the double charge and its refund, within 60 words, no leaked card digits. The deterministic checks are max_words, contains, not_contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Claude Haiku 4.5 at $0.61 per 1,000 runs. If you need the top score, Claude Haiku 4.5 at $0.61 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Anthropic with your own OpenRouter key, or tweak the wording and see what changes.

Side by side

Claude Opus 5

Claude Sonnet 5

Claude Sonnet 4.6

Claude Haiku 4.5

Frequently asked

Same prompt, other families

Stop overpaying for AI tools