Tool call · 7 checks · max 400 tokens · benched 2026-09-16

Mistral Large 3 2512 vs Mistral Medium 3.5 vs Mistral Small 4 vs Ministral 3 8B 2512 on emit a calendar tool call

Mistral AI models side by side on "Emit a calendar tool call": Ministral 3 8B 2512 scores 10/10; Ministral 3 8B 2512 is the cheapest answer scoring 8+ at $0.04 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a function-calling engine. Reply with JSON only, no prose, no code fences.

User

You have one tool:
{"name": "create_event", "parameters": {"title": "string", "start": "ISO 8601 date-time without timezone, e.g. 2026-01-31T09:00", "duration_minutes": "integer", "attendees": "array of email strings"}}

Produce the tool call for this user message as a JSON object of the shape {"name": string, "arguments": object}.

User: Book a 45-minute design review with [email protected] and [email protected] on 2 October 2026 at 3pm.

Rubric for the judge: Correct tool name, ISO start 2026-10-02T15:00, integer duration 45, both attendees in an array, a sensible title.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Mistral Large 3 2512

mistralai/mistral-large-2512

10/10

Latency: 516ms
Cost: $0.00020
Per 1,000: $0.20

161 in · 78 out · 19 words · checks 7/7

json_valid
json_keys
regex
contains
regex
contains
contains

```json
{
  "name": "create_event",
  "arguments": {
    "title": "Design Review",
    "start": "2026-10-02T15:00",
    "duration_minutes": 45,
    "attendees": ["[email protected]", "[email protected]"]
  }
}
```

Judge: All rubric requirements met: correct tool name, ISO start datetime 2026-10-02T15:00, integer duration 45, both attendees in array, and sensible title 'Design Review'.

Mistral Medium 3.5

mistralai/mistral-medium-3-5

10/10

Latency: 581ms
Cost: $0.00326
Per 1,000: $3.26

173 in · 400 out (292 reasoning) · 13 words · checks 7/7

json_valid
json_keys
regex
contains
regex
contains
contains

{"name": "create_event", "arguments": {"title": "Design Review", "start": "2026-10-02T15:00", "duration_minutes": 45, "attendees": ["[email protected]", "[email protected]"]}}

Judge: All rubric criteria met: correct tool name, proper ISO 8601 start datetime, integer duration of 45 minutes, both attendees in array format, and sensible title.

Mistral Small 4

mistralai/mistral-small-2603

10/10

Latency: 515ms
Cost: $0.00017
Per 1,000: $0.17

173 in · 245 out (158 reasoning) · 13 words · checks 7/7

json_valid
json_keys
regex
contains
regex
contains
contains

{"name": "create_event", "arguments": {"title": "Design Review", "start": "2026-10-02T15:00", "duration_minutes": 45, "attendees": ["[email protected]", "[email protected]"]}}

Judge: All rubric criteria met: correct tool name, proper ISO 8601 start datetime, integer duration of 45 minutes, both attendees in array format, and sensible title.

Ministral 3 8B 2512

mistralai/ministral-8b-2512

10/10

top judge scorecheapest scoring 8+

Latency: 336ms
Cost: $3.6e-5
Per 1,000: $0.04

161 in · 78 out · 19 words · checks 7/7

json_valid
json_keys
regex
contains
regex
contains
contains

```json
{
  "name": "create_event",
  "arguments": {
    "title": "Design Review",
    "start": "2026-10-02T15:00",
    "duration_minutes": 45,
    "attendees": ["[email protected]", "[email protected]"]
  }
}
```

Judge: All rubric requirements met: correct tool name, ISO start datetime 2026-10-02T15:00, integer duration 45, both attendees in array, and sensible title 'Design Review'.

Frequently asked

What does this prompt test?

Tool call: Correct tool name, ISO start 2026-10-02T15:00, integer duration 45, both attendees in an array, a sensible title. The deterministic checks are json_valid, json_keys, regex, contains, regex, contains, contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Ministral 3 8B 2512 at $0.04 per 1,000 runs. If you need the top score, Ministral 3 8B 2512 at $0.04 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Mistral AI with your own OpenRouter key, or tweak the wording and see what changes.

Side by side

Mistral Large 3 2512

Mistral Medium 3.5

Mistral Small 4

Ministral 3 8B 2512

Frequently asked

Same prompt, other families

Stop overpaying for AI tools