Extract JSON · 5 checks · max 400 tokens · benched 2026-09-16

Mistral Large 3 2512 vs Mistral Medium 3.5 vs Mistral Small 4 vs Ministral 3 8B 2512 on event announcement to structured json

Mistral AI models side by side on "Event announcement to structured JSON": Ministral 3 8B 2512 scores 10/10; Ministral 3 8B 2512 is the cheapest answer scoring 8+ at $0.03 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a data extraction engine. Reply with JSON only, no prose, no code fences.

User

Return one JSON object with keys title (string), date (YYYY-MM-DD), start_time (24-hour HH:MM), location (string), attendees (integer). Use only facts stated in the announcement.

Announcement: Join us for the Autumn Data Meetup on Tuesday 3 November 2026. Doors open at 2 pm and talks start at 2:30 pm at The Loft, 14 Harbour Street. We have confirmed 48 attendees so far and can take up to 60.

Rubric for the judge: Correctly picks the talk start (14:30, not doors), ISO date, integer attendees 48, and a sensible title and location.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Mistral Large 3 2512

mistralai/mistral-large-2512

10/10

Latency: 428ms
Cost: $0.00017
Per 1,000: $0.17

134 in · 71 out · 20 words · checks 5/5

json_valid
json_keys
contains
contains
regex

```json
{
  "title": "Autumn Data Meetup",
  "date": "2026-11-03",
  "start_time": "14:30",
  "location": "The Loft, 14 Harbour Street",
  "attendees": 48
}
```

Judge: All fields correctly extracted: title matches announcement, date in ISO format (2026-11-03), start_time correctly uses talk start time (14:30 not doors open 14:00), location fully specified, and attendees set to confirmed count (48).

Mistral Medium 3.5

mistralai/mistral-medium-3-5

4/10

Latency: 443ms
Cost: $0.00254
Per 1,000: $2.54

146 in · 309 out (203 reasoning) · 16 words · checks 4/5

json_valid
json_keys
contains
contains
regex

{"title": "Autumn Data Meetup", "date": "2026-11-03", "start_time": "14:00", "location": "The Loft, 14 Harbour Street", "attendees": 48}

Judge: Correctly extracted title, date, location, and attendees, but start_time is 14:00 (doors open) instead of 14:30 (talks start) as specified in the rubric.

Mistral Small 4

mistralai/mistral-small-2603

10/10

Latency: 453ms
Cost: $0.00015
Per 1,000: $0.15

146 in · 220 out (127 reasoning) · 16 words · checks 5/5

json_valid
json_keys
contains
contains
regex

{"title": "Autumn Data Meetup", "date": "2026-11-03", "start_time": "14:30", "location": "The Loft, 14 Harbour Street", "attendees": 48}

Judge: All fields correctly extracted: title matches announcement, date is proper ISO format, start_time correctly uses talk start time (14:30) not door opening, location is complete and accurate, and attendees uses confirmed count (48) as instructed.

Ministral 3 8B 2512

mistralai/ministral-8b-2512

10/10

top judge scorecheapest scoring 8+

Latency: 383ms
Cost: $3.1e-5
Per 1,000: $0.03

134 in · 71 out · 20 words · checks 5/5

json_valid
json_keys
contains
contains
regex

```json
{
  "title": "Autumn Data Meetup",
  "date": "2026-11-03",
  "start_time": "14:30",
  "location": "The Loft, 14 Harbour Street",
  "attendees": 48
}
```

Frequently asked

What does this prompt test?

Extract JSON: Correctly picks the talk start (14:30, not doors), ISO date, integer attendees 48, and a sensible title and location. The deterministic checks are json_valid, json_keys, contains, contains, regex.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Ministral 3 8B 2512 at $0.03 per 1,000 runs. If you need the top score, Ministral 3 8B 2512 at $0.03 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Mistral AI with your own OpenRouter key, or tweak the wording and see what changes.

Side by side

Mistral Large 3 2512

Mistral Medium 3.5

Mistral Small 4

Ministral 3 8B 2512

Frequently asked

Same prompt, other families

Stop overpaying for AI tools