SQL · 5 checks · max 500 tokens · benched 2026-09-16

Claude Opus 5 vs Claude Sonnet 5 vs Claude Sonnet 4.6 vs Claude Haiku 4.5 on monthly active users

Anthropic models side by side on "Monthly active users": Claude Haiku 4.5 scores 10/10; Claude Haiku 4.5 is the cheapest answer scoring 8+ at $0.51 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a database engineer. Reply with a single SQL statement and nothing else.

User

Table: events(user_id, event_type, occurred_at). Write one PostgreSQL query returning, for each calendar month of 2025, the month (as the first day of that month) and the number of distinct users who had at least one event in that month, ordered by month ascending.

Rubric for the judge: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Claude Opus 5

anthropic/claude-opus-5

10/10

Latency: 1.1s
Cost: $0.00367
Per 1,000: $3.67

114 in · 124 out · 29 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT date_trunc('month', occurred_at)::date AS month,
       COUNT(DISTINCT user_id) AS distinct_users
FROM events
WHERE occurred_at >= DATE '2025-01-01'
  AND occurred_at < DATE '2026-01-01'
GROUP BY 1
ORDER BY 1;
```

Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups by month, orders ascending, and casts to date to show first day of month.

Claude Sonnet 5

anthropic/claude-sonnet-5

10/10

Latency: 1.4s
Cost: $0.00155
Per 1,000: $1.55

114 in · 132 out · 29 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT date_trunc('month', occurred_at)::date AS month,
       COUNT(DISTINCT user_id) AS distinct_users
FROM events
WHERE occurred_at >= '2025-01-01' AND occurred_at < '2026-01-01'
GROUP BY date_trunc('month', occurred_at)::date
ORDER BY month ASC;
```

Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups and orders by month, and returns the first day of each month as required.

Claude Sonnet 4.6

anthropic/claude-sonnet-4.6

10/10

Latency: 856ms
Cost: $0.00134
Per 1,000: $1.34

88 in · 72 out · 25 words · checks 5/5

regex
regex
regex
regex
contains

SELECT DATE_TRUNC('month', occurred_at) AS month, COUNT(DISTINCT user_id) AS distinct_users
FROM events
WHERE occurred_at >= '2025-01-01' AND occurred_at < '2026-01-01'
GROUP BY 1
ORDER BY 1;

Judge: Query correctly uses DATE_TRUNC('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups by month, and orders ascending; all rubric requirements met.

Claude Haiku 4.5

anthropic/claude-haiku-4.5

10/10

top judge scorecheapest scoring 8+

Latency: 719ms
Cost: $0.00051
Per 1,000: $0.51

87 in · 84 out · 27 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT 
  DATE_TRUNC('month', occurred_at)::DATE AS month,
  COUNT(DISTINCT user_id) AS distinct_users
FROM events
WHERE EXTRACT(YEAR FROM occurred_at) = 2025
GROUP BY DATE_TRUNC('month', occurred_at)
ORDER BY month ASC;
```

Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups and orders by month, and returns the first day of each month as required.

Frequently asked

What does this prompt test?

SQL: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month. The deterministic checks are regex, regex, regex, regex, contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Claude Haiku 4.5 at $0.51 per 1,000 runs. If you need the top score, Claude Haiku 4.5 at $0.51 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Anthropic with your own OpenRouter key, or tweak the wording and see what changes.

Side by side

Claude Opus 5

Claude Sonnet 5

Claude Sonnet 4.6

Claude Haiku 4.5

Frequently asked

Same prompt, other families

Stop overpaying for AI tools