Rewrite · 4 checks · max 400 tokens · benched 2026-09-16

Mistral Large 3 2512 vs Mistral Medium 3.5 vs Mistral Small 4 vs Ministral 3 8B 2512 on product update to a 280-character post

Mistral AI models side by side on "Product update to a 280-character post": Ministral 3 8B 2512 scores 9/10; Ministral 3 8B 2512 is the cheapest answer scoring 8+ at $0.02 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

User

Turn this update into a single social media post of at most 280 characters. It must mention "v2.0" and end with the hashtag #OpenSource. Output only the post.

Update: Version 2.0 of our command-line tool is out. It is three times faster on large repositories, adds a built-in diff viewer, and is now licensed under MIT so anyone can contribute.

Rubric for the judge: Punchy post under 280 characters that mentions v2.0, the speed gain, the diff viewer and MIT, ending with #OpenSource.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Mistral Large 3 2512

mistralai/mistral-large-2512

9/10

Latency: 464ms
Cost: $0.00011
Per 1,000: $0.11

87 in · 43 out · 20 words · checks 3/4

max_chars
contains
regex
contains

"🚀 v2.0 is here! 3x faster on large repos, built-in diff viewer, & now MIT-licensed. Dive in & contribute! #OpenSource"

Judge: Post is 119 characters, mentions v2.0, speed gain, diff viewer, MIT licensing, and ends with #OpenSource as required; only minor issue is not explicitly stating 'MIT' by name though 'MIT-licensed' is clear.

Mistral Medium 3.5

mistralai/mistral-medium-3-5

1/10

failed: empty output (finish_reason length)

Latency: 861ms
Cost: $0.01215
Per 1,000: $12.15

99 in · 1600 out (1014 reasoning) · 0 words · checks 1/4

max_chars
contains
regex
contains

(empty output)

Judge: The output is completely empty, failing to provide any social media post whatsoever.

Mistral Small 4

mistralai/mistral-small-2603

1/10

failed: empty output (finish_reason length)

Latency: 849ms
Cost: $0.00096
Per 1,000: $0.96

99 in · 1600 out (1599 reasoning) · 0 words · checks 1/4

max_chars
contains
regex
contains

(empty output)

Judge: The output is completely empty, failing to provide any social media post whatsoever.

Ministral 3 8B 2512

mistralai/ministral-8b-2512

9/10

top judge scorecheapest scoring 8+

Latency: 335ms
Cost: $2.0e-5
Per 1,000: $0.02

87 in · 48 out · 22 words · checks 3/4

max_chars
contains
regex
contains

"Big news! 🚀 Our CLI just hit **v2.0**—3x faster, built-in diff viewer, and now **MIT-licensed** for open contributions. Try it out! #OpenSource"

Judge: Post is 139 characters (well under 280), mentions v2.0, speed gain, diff viewer, and MIT licensing, ends with #OpenSource as required; only minor issue is bold markdown formatting may not render on all platforms.

Frequently asked

What does this prompt test?

Rewrite: Punchy post under 280 characters that mentions v2.0, the speed gain, the diff viewer and MIT, ending with #OpenSource. The deterministic checks are max_chars, contains, regex, contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Ministral 3 8B 2512 at $0.02 per 1,000 runs. If you need the top score, Ministral 3 8B 2512 at $0.02 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Mistral AI with your own OpenRouter key, or tweak the wording and see what changes.