SQL · 5 checks · max 500 tokens · benched 2026-09-16

Mistral Large 3 2512 vs Mistral Medium 3.5 vs Mistral Small 4 vs Ministral 3 8B 2512 on monthly active users

Mistral AI models side by side on "Monthly active users": Mistral Large 3 2512 scores 9/10; Mistral Large 3 2512 is the cheapest answer scoring 8+ at $0.22 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a database engineer. Reply with a single SQL statement and nothing else.

User

Table: events(user_id, event_type, occurred_at). Write one PostgreSQL query returning, for each calendar month of 2025, the month (as the first day of that month) and the number of distinct users who had at least one event in that month, ordered by month ascending.

Rubric for the judge: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Mistral Large 3 2512

mistralai/mistral-large-2512

9/10

top judge scorecheapest scoring 8+

Latency: 522ms
Cost: $0.00022
Per 1,000: $0.22

83 in · 118 out · 33 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT
    DATE_TRUNC('month', d.month)::date AS month,
    COUNT(DISTINCT e.user_id) AS distinct_users
FROM
    generate_series('2025-01-01'::date, '2025-12-01'::date, '1 month'::interval) d(month)
LEFT JOIN
    events e ON DATE_TRUNC('month', e.occurred_at) = d.month
GROUP BY
    d.month
ORDER BY
    d.month ASC;
```

Judge: Query correctly uses date_trunc('month'), COUNT(DISTINCT user_id), filters to 2025, groups and orders by month, and returns first day of each month; minor deduction for using generate_series with LEFT JOIN (over-engineered but functionally correct) rather than simpler filtering approach.

Mistral Medium 3.5

mistralai/mistral-medium-3-5

1/10

failed: empty output (finish_reason length)

Latency: 973ms
Cost: $0.01289
Per 1,000: $12.89

95 in · 1700 out (1586 reasoning) · 0 words · checks 0/5

regex
regex
regex
regex
contains

(empty output)

Judge: No query was provided; the output is completely empty when a specific PostgreSQL query was explicitly required.

Mistral Small 4

mistralai/mistral-small-2603

9/10

Latency: 974ms
Cost: $0.00088
Per 1,000: $0.88

95 in · 1454 out (1238 reasoning) · 32 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT s.month AS month, COUNT(DISTINCT e.user_id) AS distinct_user_count FROM generate_series('2025-01-01'::timestamp, '2025-12-01'::timestamp, '1 month'::interval) AS s(month) LEFT JOIN events e ON date_trunc('month', e.occurred_at) = s.month GROUP BY s.month ORDER BY s.month;
```

Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters to 2025, groups and orders by month, and returns the first day of each month; minor deduction for using LEFT JOIN with generate_series (which works but is slightly unconventional compared to filtering the events

Ministral 3 8B 2512

mistralai/ministral-8b-2512

6/10

Latency: 359ms
Cost: $3.8e-5
Per 1,000: $0.04

83 in · 172 out · 62 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT
    DATE_TRUNC('month', DATE '2025-01-01') + (n * INTERVAL '1 month') AS month_start,
    COUNT(DISTINCT user_id) AS distinct_users
FROM
    generate_series(0, 11, 1) AS n
JOIN
    events ON DATE_TRUNC('month', occurred_at) BETWEEN DATE_TRUNC('month', DATE '2025-01-01') + (n * INTERVAL '1 month')
                                               AND DATE_TRUNC('month', DATE '2025-01-01') + ((n + 1) * INTERVAL '1 month') - INTERVAL '1 day'
GROUP BY
    n
ORDER BY
    month_start ASC;
```

Judge: Uses date_trunc and COUNT(DISTINCT user_id) with 2025 filter and proper grouping/ordering, but the approach is unnecessarily complex with generate_series and manual interval arithmetic instead of the simpler direct filtering pattern specified in the rubric.

Frequently asked

What does this prompt test?

SQL: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month. The deterministic checks are regex, regex, regex, regex, contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Mistral Large 3 2512 at $0.22 per 1,000 runs. If you need the top score, Mistral Large 3 2512 at $0.22 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Mistral AI with your own OpenRouter key, or tweak the wording and see what changes.