Summarise · 3 checks · max 400 tokens · benched 2026-09-16

Gemini 3.1 Pro Preview vs Gemini 3.8 Flash vs Gemini 3.5 Flash Lite on summarise a support thread in 60 words

Google models side by side on "Summarise a support thread in 60 words": Gemini 3.8 Flash scores 9/10; Gemini 3.8 Flash is the cheapest answer scoring 8+ at $0.40 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a concise assistant. Follow length limits exactly and output plain text only.

User

Summarise the following customer support thread in no more than 60 words. Mention what the customer wanted and how it was resolved.

---
Customer (Mon 09:12): Hi, I was charged twice for my Team plan on 1 September. Order references are 88213 and 88214. I only have one workspace. Can you fix this?
Agent (Mon 10:40): Sorry about that. I can see the duplicate charge. Could you confirm the last four digits of the card so I can raise the refund?
Customer (Mon 11:02): 4471.
Agent (Mon 11:30): Thanks. I have refunded order 88214 in full (USD 96.00). It should appear on your statement within 5 business days. I have also added a note so it cannot recur next cycle.
Customer (Mon 11:35): Great, thanks for the quick turnaround.
---

Rubric for the judge: Accurate, complete summary of the double charge and its refund, within 60 words, no leaked card digits.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Gemini 3.1 Pro Preview

google/gemini-3.1-pro-preview

9/10

Latency: 2.5s
Cost: $0.00493
Per 1,000: $4.93

235 in · 372 out (327 reasoning) · 41 words · checks 3/3

max_words
contains
not_contains

The customer requested a fix after being charged twice for their Team plan subscription. The agent resolved the issue by issuing a full refund of $96 for the duplicate charge and updating the account to prevent the error from happening again.

Judge: Accurate summary covering what the customer wanted (fix for double charge) and resolution (refund + prevention), well under 60 words (31 words), no card digits leaked, though could have mentioned the specific timeframe for refund appearance.

Gemini 3.8 Flash

google/gemini-3.8-flash

9/10

top judge scorecheapest scoring 8+

Latency: 1.2s
Cost: $0.00040
Per 1,000: $0.40

235 in · 61 out · 50 words · checks 3/3

max_words
contains
not_contains

The customer reported being charged twice for their Team plan on 1 September and requested a fix. After the customer verified the last four digits of their card, the agent resolved the issue by issuing a full refund for the duplicate charge (order 88214) and ensuring it would not recur.

Judge: Accurate summary covering what the customer wanted (fix for duplicate charge) and resolution (refund issued, prevention measures taken), within 60 words, no card digits leaked, though could have mentioned the specific refund amount for completeness.

Gemini 3.5 Flash Lite

google/gemini-3.5-flash-lite

2/10

Latency: 1.0s
Cost: $0.00106
Per 1,000: $1.06

235 in · 396 out (362 reasoning) · 30 words · checks 3/3

max_words
contains
not_contains

The customer requested a refund after being charged twice for their Team plan. The agent resolved the issue by verifying the customer's card details, issuing a full refund for the

Judge: Output is incomplete (cuts off mid-sentence), fails to mention the refund amount or timeline, and doesn't fully explain the resolution despite being under 60 words.

Frequently asked

What does this prompt test?

Summarise: Accurate, complete summary of the double charge and its refund, within 60 words, no leaked card digits. The deterministic checks are max_words, contains, not_contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Gemini 3.8 Flash at $0.40 per 1,000 runs. If you need the top score, Gemini 3.8 Flash at $0.40 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Google with your own OpenRouter key, or tweak the wording and see what changes.