SQL · 5 checks · max 500 tokens · benched 2026-11-05

Muse Spark 1.3 vs Muse Glimmer 30B vs Llama 4 Maverick vs Llama 4 Scout on monthly active users

Meta models side by side on "Monthly active users": Llama 4 Scout scores 10/10; Llama 4 Scout is the cheapest answer scoring 8+ at $0.03 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a database engineer. Reply with a single SQL statement and nothing else.

User

Table: events(user_id, event_type, occurred_at). Write one PostgreSQL query returning, for each calendar month of 2025, the month (as the first day of that month) and the number of distinct users who had at least one event in that month, ordered by month ascending.

Rubric for the judge: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Muse Spark 1.3

meta/muse-spark-1.3

4/10

Latency: 1.1s
Cost: $0.00400
Per 1,000: $4.00

101 in · 912 out (769 reasoning) · 41 words · checks 4/5

regex
regex
regex
regex
contains

SELECT m.month AS month, COUNT(DISTINCT e.user_id) AS distinct_users FROM generate_series(DATE '2025-01-01', DATE '2025-12-01', INTERVAL '1 month') AS m(month) LEFT JOIN events e ON e.occurred_at >= m.month AND e.occurred_at < m.month + INTERVAL '1 month' GROUP BY m.month ORDER BY m.month ASC;

Judge: Query produces correct results but fails to use date_trunc('month', occurred_at) as specified in rubric; instead uses generate_series with manual interval logic, which is a valid alternative approach but doesn't follow the required pattern.

Muse Glimmer 30B

meta/muse-glimmer-30b

7/10

Latency: 208ms
Cost: $0.00054
Per 1,000: $0.54

104 in · 464 out (365 reasoning) · 33 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT 
    m.month AS month,
    COUNT(DISTINCT e.user_id) AS distinct_users
FROM generate_series('2025-01-01'::date, '2025-12-01'::date, interval '1 month') AS m(month)
LEFT JOIN events e
  ON date_trunc('month', e.occurred_at)::date = m.month
GROUP BY m.month
ORDER BY m.month;
```

Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups and orders by month, but uses generate_series approach instead of filtering the events table directly, which is less efficient and doesn't explicitly filter WHERE occurred_at is in 2025.

Llama 4 Maverick

meta-llama/llama-4-maverick

10/10

Latency: 287ms
Cost: $7.0e-5
Per 1,000: $0.07

91 in · 74 out · 29 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT DATE_TRUNC('month', occurred_at) AS month, COUNT(DISTINCT user_id) AS distinct_users 
FROM events 
WHERE occurred_at >= '2025-01-01' AND occurred_at < '2026-01-01' 
GROUP BY DATE_TRUNC('month', occurred_at) 
ORDER BY month ASC;
```

Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups and orders by month as required by the rubric.

Llama 4 Scout

meta-llama/llama-4-scout

10/10

top judge scorecheapest scoring 8+

Latency: 443ms
Cost: $3.4e-5
Per 1,000: $0.03

91 in · 83 out · 29 words · checks 5/5

regex
regex
regex
regex
contains

```sql
SELECT 
    DATE_TRUNC('month', occurred_at) AS month,
    COUNT(DISTINCT user_id) AS num_users
FROM 
    events
WHERE 
    occurred_at >= '2025-01-01' AND occurred_at < '2026-01-01'
GROUP BY 
    DATE_TRUNC('month', occurred_at)
ORDER BY 
    month ASC;
```

Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups and orders by month as required by the rubric.

Frequently asked

What does this prompt test?

SQL: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month. The deterministic checks are regex, regex, regex, regex, contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Llama 4 Scout at $0.03 per 1,000 runs. If you need the top score, Llama 4 Scout at $0.03 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on Meta with your own OpenRouter key, or tweak the wording and see what changes.

Partner picks

Claude (credits)Replit Make Taskade Fireflies.ai Pressmaster.ai Murf AI

Affiliate/referral links: commissions or credits