SQL · 5 checks · max 500 tokens · benched 2026-09-16
Meta models side by side on "Monthly active users": Llama 4 Scout scores 10/10; Llama 4 Scout is the cheapest answer scoring 8+ at $0.03 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.
System
You are a database engineer. Reply with a single SQL statement and nothing else.
User
Table: events(user_id, event_type, occurred_at). Write one PostgreSQL query returning, for each calendar month of 2025, the month (as the first day of that month) and the number of distinct users who had at least one event in that month, ordered by month ascending.
Rubric for the judge: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month.
Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.
meta/muse-spark-1.3
0 in · 0 out · 0 words · checks 0/5
(empty output)
Judge: —
meta/muse-glimmer-30b
104 in · 464 out (365 reasoning) · 33 words · checks 5/5
```sql
SELECT
m.month AS month,
COUNT(DISTINCT e.user_id) AS distinct_users
FROM generate_series('2025-01-01'::date, '2025-12-01'::date, interval '1 month') AS m(month)
LEFT JOIN events e
ON date_trunc('month', e.occurred_at)::date = m.month
GROUP BY m.month
ORDER BY m.month;
```
Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups and orders by month, but uses generate_series approach instead of filtering the events table directly, which is less efficient and doesn't explicitly filter WHERE occurred_at is in 2025.
meta-llama/llama-4-maverick
91 in · 74 out · 29 words · checks 5/5
```sql
SELECT DATE_TRUNC('month', occurred_at) AS month, COUNT(DISTINCT user_id) AS distinct_users
FROM events
WHERE occurred_at >= '2025-01-01' AND occurred_at < '2026-01-01'
GROUP BY DATE_TRUNC('month', occurred_at)
ORDER BY month ASC;
```Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups and orders by month as required by the rubric.
meta-llama/llama-4-scout
91 in · 83 out · 29 words · checks 5/5
```sql
SELECT
DATE_TRUNC('month', occurred_at) AS month,
COUNT(DISTINCT user_id) AS num_users
FROM
events
WHERE
occurred_at >= '2025-01-01' AND occurred_at < '2026-01-01'
GROUP BY
DATE_TRUNC('month', occurred_at)
ORDER BY
month ASC;
```Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups and orders by month as required by the rubric.
SQL: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month. The deterministic checks are regex, regex, regex, regex, contains.
If the judge's bar of 8/10 is good enough for you, Llama 4 Scout at $0.03 per 1,000 runs. If you need the top score, Llama 4 Scout at $0.03 per 1,000 runs.
Run it yourself
Re-run this prompt on Meta with your own OpenRouter key, or tweak the wording and see what changes.
More Meta prompts
If our calculators helped you cut down on hidden AI wallet leaks, thanks for using them. A tiny fraction of your savings is what keeps our pricing indexes updated daily.
Not sure which AI is cheapest for your use case? Find out in 30 seconds — no signup required.
AI cost intelligence
Join the OverpayingForAI list for pricing updates, cheaper alternatives, and practical buying guidance.
Now tracking 50+ AI tools, models, platforms, subscriptions, coding tools, and automation products.