SQL · 5 checks · max 500 tokens · benched 2026-09-16

Grok 4.6 vs Grok 4.5 vs Grok 4.20 vs Grok 4.3 on monthly active users

xAI models side by side on "Monthly active users": Grok 4.5 scores 8/10; Grok 4.5 is the cheapest answer scoring 8+ at $3.19 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a database engineer. Reply with a single SQL statement and nothing else.

User

Table: events(user_id, event_type, occurred_at). Write one PostgreSQL query returning, for each calendar month of 2025, the month (as the first day of that month) and the number of distinct users who had at least one event in that month, ordered by month ascending.

Rubric for the judge: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

Grok 4.6

x-ai/grok-4.6

7/10

Latency: 705ms
Cost: $0.00363
Per 1,000: $3.63

289 in · 540 out (447 reasoning) · 36 words · checks 4/5

regex
regex
regex
regex
contains

SELECT gs.month::date, COUNT(DISTINCT e.user_id)
FROM generate_series(DATE '2025-01-01', DATE '2025-12-01', INTERVAL '1 month') AS gs(month)
LEFT JOIN events e ON e.occurred_at >= gs.month AND e.occurred_at < gs.month + INTERVAL '1 month'
GROUP BY gs.month
ORDER BY gs.month;

Judge: Query correctly returns distinct users per month in 2025 ordered by month, but uses generate_series with manual interval logic instead of the rubric-specified date_trunc('month', occurred_at) approach, which is less idiomatic for this use case.

Grok 4.5

x-ai/grok-4.5

8/10

top judge scorecheapest scoring 8+

Latency: 287ms
Cost: $0.00319
Per 1,000: $3.19

289 in · 472 out (372 reasoning) · 37 words · checks 4/5

regex
regex
regex
regex
contains

SELECT gs.month::date AS month, COUNT(DISTINCT e.user_id) AS distinct_users FROM generate_series('2025-01-01'::date, '2025-12-01'::date, '1 month'::interval) AS gs(month) LEFT JOIN events e ON e.occurred_at >= gs.month AND e.occurred_at < gs.month + INTERVAL '1 month' GROUP BY gs.month ORDER BY gs.month;

Judge: Query correctly returns distinct users per month in 2025 ordered by month, but uses generate_series with interval logic instead of the rubric's specified date_trunc('month', occurred_at) approach, which is a valid alternative but doesn't match the required method.

Grok 4.20

x-ai/grok-4.20

3/10

Latency: 538ms
Cost: $0.00517
Per 1,000: $5.17

265 in · 1991 out (1921 reasoning) · 33 words · checks 4/5

regex
regex
regex
regex
contains

SELECT d::date AS month, COUNT(DISTINCT user_id) FROM generate_series('2025-01-01'::date, '2025-12-01'::date, '1 month'::interval) d LEFT JOIN events ON occurred_at >= d AND occurred_at < d + INTERVAL '1 month' GROUP BY d ORDER BY d;

Judge: The query uses generate_series instead of date_trunc as specified in the rubric, and the LEFT JOIN approach is unnecessarily complex; while it may produce correct results, it violates the explicit rubric requirement to use date_trunc('month', occurred_at) and doesn't filter events by 2025 in the WHE

Grok 4.3

x-ai/grok-4.3

7/10

Latency: 383ms
Cost: $0.00200
Per 1,000: $2.00

271 in · 746 out (654 reasoning) · 32 words · checks 4/5

regex
regex
regex
regex
contains

SELECT m.month_start AS month, COUNT(DISTINCT e.user_id) AS distinct_users FROM (SELECT generate_series('2025-01-01'::date,'2025-12-01'::date,interval '1 month') AS month_start) m LEFT JOIN events e ON e.occurred_at>=m.month_start AND e.occurred_at<m.month_start+interval '1 month' GROUP BY m.month_start ORDER BY m.month_start;

Judge: Query correctly returns distinct users per month in 2025 ordered ascending, but uses generate_series with manual date range instead of date_trunc('month', occurred_at) as specified in rubric, and lacks explicit WHERE clause filtering for 2025.

Frequently asked

What does this prompt test?

SQL: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month. The deterministic checks are regex, regex, regex, regex, contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, Grok 4.5 at $3.19 per 1,000 runs. If you need the top score, Grok 4.5 at $3.19 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on xAI with your own OpenRouter key, or tweak the wording and see what changes.