SQL · 5 checks · max 500 tokens · benched 2026-09-16

GPT-5.5 vs GPT-5.4 vs GPT-5.4 mini vs GPT-5.4 Nano vs GPT-5.3-Codex on monthly active users

OpenAI models side by side on "Monthly active users": GPT-5.4 Nano scores 10/10; GPT-5.4 Nano is the cheapest answer scoring 8+ at $0.11 per 1,000 runs. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a database engineer. Reply with a single SQL statement and nothing else.

User

Table: events(user_id, event_type, occurred_at). Write one PostgreSQL query returning, for each calendar month of 2025, the month (as the first day of that month) and the number of distinct users who had at least one event in that month, ordered by month ascending.

Rubric for the judge: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

GPT-5.5

openai/gpt-5.5

6/10

Latency: 980ms
Cost: $0.00526
Per 1,000: $5.26

86 in · 161 out (56 reasoning) · 41 words · checks 4/5

regex
regex
regex
regex
contains

SELECT
  m.month::date AS month,
  COUNT(DISTINCT e.user_id) AS distinct_users
FROM generate_series(DATE '2025-01-01', DATE '2025-12-01', INTERVAL '1 month') AS m(month)
LEFT JOIN events e
  ON e.occurred_at >= m.month
 AND e.occurred_at <  m.month + INTERVAL '1 month'
GROUP BY m.month
ORDER BY m.month ASC;

Judge: Query produces correct output but uses generate_series with manual interval logic instead of the rubric-specified date_trunc('month', occurred_at) approach, and lacks explicit 2025 filter in WHERE clause despite achieving correct results.

GPT-5.4

openai/gpt-5.4

7/10

Latency: 406ms
Cost: $0.00186
Per 1,000: $1.86

86 in · 110 out (9 reasoning) · 39 words · checks 4/5

regex
regex
regex
regex
contains

SELECT month_start, COUNT(DISTINCT e.user_id) AS distinct_user_count
FROM generate_series(DATE '2025-01-01', DATE '2025-12-01', INTERVAL '1 month') AS gs(month_start)
LEFT JOIN events e
  ON e.occurred_at >= gs.month_start
 AND e.occurred_at < gs.month_start + INTERVAL '1 month'
GROUP BY month_start
ORDER BY month_start ASC;

Judge: Query correctly returns distinct users per month in 2025 ordered ascending, but uses generate_series with manual date range logic instead of the rubric-specified date_trunc('month', occurred_at) approach, which is less elegant though functionally equivalent.

GPT-5.4 mini

openai/gpt-5.4-mini

6/10

Latency: 429ms
Cost: $0.00071
Per 1,000: $0.71

86 in · 144 out (21 reasoning) · 49 words · checks 4/5

regex
regex
regex
regex
contains

WITH months AS (
  SELECT generate_series(
           DATE '2025-01-01',
           DATE '2025-12-01',
           INTERVAL '1 month'
         )::date AS month_start
)
SELECT
  m.month_start,
  COUNT(DISTINCT e.user_id) AS distinct_users
FROM months m
LEFT JOIN events e
  ON e.occurred_at >= m.month_start
 AND e.occurred_at < (m.month_start + INTERVAL '1 month')
GROUP BY m.month_start
ORDER BY m.month_start ASC;

Judge: The query produces correct results but doesn't use date_trunc('month', occurred_at) as specified in the rubric; instead it uses generate_series and manual interval logic, which is a valid alternative approach but fails to meet the explicit rubric requirement.

GPT-5.4 Nano

openai/gpt-5.4-nano

10/10

top judge scorecheapest scoring 8+

Latency: 256ms
Cost: $0.00011
Per 1,000: $0.11

86 in · 74 out · 27 words · checks 5/5

regex
regex
regex
regex
contains

SELECT
  date_trunc('month', occurred_at)::date AS month_start,
  COUNT(DISTINCT user_id) AS distinct_users
FROM events
WHERE occurred_at >= DATE '2025-01-01'
  AND occurred_at <  DATE '2026-01-01'
GROUP BY 1
ORDER BY 1;

Judge: Query correctly uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), filters for 2025, groups by month, orders ascending, and returns the first day of each month as requested.

GPT-5.3-Codex

openai/gpt-5.3-codex

6/10

Latency: 427ms
Cost: $0.00407
Per 1,000: $4.07

86 in · 280 out (173 reasoning) · 39 words · checks 4/5

regex
regex
regex
regex
contains

SELECT m.month_start AS month, COUNT(DISTINCT e.user_id) AS distinct_user_count
FROM generate_series('2025-01-01'::date, '2025-12-01'::date, interval '1 month') AS m(month_start)
LEFT JOIN events e
  ON e.occurred_at >= m.month_start
 AND e.occurred_at < m.month_start + interval '1 month'
GROUP BY m.month_start
ORDER BY m.month_start ASC;

Judge: Query produces correct output but uses generate_series with manual date range logic instead of the rubric-specified date_trunc('month', occurred_at) approach, and lacks explicit 2025 filtering in the WHERE clause (though it's implicitly constrained by the series range).

Frequently asked

What does this prompt test?

SQL: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month. The deterministic checks are regex, regex, regex, regex, contains.

Which model should I pick for this task?

If the judge's bar of 8/10 is good enough for you, GPT-5.4 Nano at $0.11 per 1,000 runs. If you need the top score, GPT-5.4 Nano at $0.11 per 1,000 runs.

Same prompt, other families

Run it yourself

Re-run this prompt on OpenAI with your own OpenRouter key, or tweak the wording and see what changes.

Side by side

GPT-5.5

GPT-5.4

GPT-5.4 mini

GPT-5.4 Nano

GPT-5.3-Codex

Frequently asked

Same prompt, other families

Stop overpaying for AI tools