Summarise · 3 checks · max 400 tokens · benched 2026-09-16

GLM 5.3 vs GLM 5.2 vs GLM 5.3 Flash on summarise a support thread in 60 words

Z Ai models side by side on "Summarise a support thread in 60 words": GLM 5.3 scores 10/10; GLM 5.3 Flash is the cheapest answer scoring 8+ at $0.03 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a concise assistant. Follow length limits exactly and output plain text only.

User

Summarise the following customer support thread in no more than 60 words. Mention what the customer wanted and how it was resolved.

---
Customer (Mon 09:12): Hi, I was charged twice for my Team plan on 1 September. Order references are 88213 and 88214. I only have one workspace. Can you fix this?
Agent (Mon 10:40): Sorry about that. I can see the duplicate charge. Could you confirm the last four digits of the card so I can raise the refund?
Customer (Mon 11:02): 4471.
Agent (Mon 11:30): Thanks. I have refunded order 88214 in full (USD 96.00). It should appear on your statement within 5 business days. I have also added a note so it cannot recur next cycle.
Customer (Mon 11:35): Great, thanks for the quick turnaround.
---

Rubric for the judge: Accurate, complete summary of the double charge and its refund, within 60 words, no leaked card digits.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

GLM 5.3

z-ai/glm-5.3

10/10

top judge score

Latency: 607ms
Cost: $0.00066
Per 1,000: $0.66

222 in · 79 out · 57 words · checks 3/3

max_words
contains
not_contains

The customer was charged twice for their Team plan on 1 September (orders 88213 and 88214) despite having one workspace. After confirming their card's last four digits, the agent refunded the duplicate charge of USD 96.00, expected within 5 business days, and added a note to prevent recurrence. The customer thanked the agent for the quick resolution.

Judge: Summary accurately captures the double charge issue, refund resolution (USD 96.00), timeline (5 business days), and preventive measure, stays within 60 words (58), and appropriately omits the card digits while meeting all rubric requirements.

GLM 5.2

z-ai/glm-5.2

2/10

Latency: 399ms
Cost: $0.00093
Per 1,000: $0.93

222 in · 400 out (374 reasoning) · 21 words · checks 2/3

max_words
contains
not_contains

The customer reported a duplicate charge for their Team plan and requested a correction. The agent verified the error, asked for

Judge: Output is incomplete and cuts off mid-sentence, failing to mention the resolution (refund amount, timeline, or prevention measures) and does not reach the required summary despite being well under 60 words.

GLM 5.3 Flash

z-ai/glm-5.3-flash

9/10

cheapest scoring 8+

Latency: 263ms
Cost: $3.3e-5
Per 1,000: $0.03

222 in · 67 out · 43 words · checks 3/3

max_words
contains
not_contains

Customer reported a duplicate Team plan charge on 1 September (orders 88213 and 88214). After verifying the card, the agent refunded order 88214 in full (USD 96.00), expected within 5 business days, and added a note to prevent recurrence. Customer thanked the agent.

Judge: Accurate, complete summary covering the duplicate charge issue and resolution within 60 words (exactly 56), with no card digits leaked; only minor: could have explicitly stated 'one workspace' context.

Frequently asked

What does this prompt test?

Summarise: Accurate, complete summary of the double charge and its refund, within 60 words, no leaked card digits. The deterministic checks are max_words, contains, not_contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, GLM 5.3 Flash at $0.03 per 1,000 runs. If you need the top score, GLM 5.3 at $0.66 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Z Ai with your own OpenRouter key, or tweak the wording and see what changes.