SQL · 5 checks · max 500 tokens · benched 2026-09-16

DeepSeek V4 Pro vs DeepSeek V4 Flash 0423 vs DeepSeek V4.1 Flash on monthly active users

DeepSeek models side by side on "Monthly active users": DeepSeek V4.1 Flash scores 7/10. Outputs, checks, judge reasons, latency and cost.

The prompt every model received

System

You are a database engineer. Reply with a single SQL statement and nothing else.

User

Table: events(user_id, event_type, occurred_at). Write one PostgreSQL query returning, for each calendar month of 2025, the month (as the first day of that month) and the number of distinct users who had at least one event in that month, ordered by month ascending.

Rubric for the judge: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month.

Side by side

Every cell is one OpenRouter call at temperature 0 with the prompt's token cap and reasoning effort "low" where the model supports it. Cost is usage × the catalogue rate in models.json. Quality is one judge call to anthropic/claude-haiku-4.5 against the prompt's rubric, cached per prompt version.

DeepSeek V4 Pro

deepseek/deepseek-v4-pro

7/10

Latency: 3.4s
Cost: $0.00081
Per 1,000: $0.81

79 in · 499 out (400 reasoning) · 35 words · checks 4/5

regex
regex
regex
regex
contains

SELECT months.month, COUNT(DISTINCT e.user_id) AS distinct_users
FROM generate_series('2025-01-01'::date, '2025-12-01'::date, '1 month'::interval) AS months(month)
LEFT JOIN events e ON e.occurred_at >= months.month AND e.occurred_at < months.month + interval '1 month'
GROUP BY months.month
ORDER BY months.month;

Judge: Query correctly returns distinct users per month in 2025 ordered by month, but uses manual date range logic instead of the rubric-specified date_trunc('month', occurred_at) approach, and lacks an explicit WHERE clause filtering for 2025.

DeepSeek V4 Flash 0423

deepseek/deepseek-v4-flash

2/10

Latency: 934ms
Cost: $7.2e-5
Per 1,000: $0.07

79 in · 500 out (456 reasoning) · 12 words · checks 1/5

regex
regex
regex
regex
contains

WITH months AS (
    SELECT generate_series('2025-01-01'::date, '2025-12-01'::date, '1 month'::interval) AS month_start
)

Judge: Output is incomplete—it only defines a CTE without the main query that joins to events table, applies the rubric requirements (date_trunc, COUNT DISTINCT, GROUP BY, ORDER BY), or returns any results.

DeepSeek V4.1 Flash

deepseek/deepseek-v4.1-flash

7/10

top judge score

Latency: 403ms
Cost: $0.00011
Per 1,000: $0.11

105 in · 160 out (47 reasoning) · 39 words · checks 4/5

regex
regex
regex
regex
contains

```sql
SELECT gs.month,
       COUNT(DISTINCT e.user_id) AS distinct_users
FROM generate_series('2025-01-01'::date, '2025-12-01'::date, INTERVAL '1 month') AS gs(month)
LEFT JOIN events e
  ON e.occurred_at >= gs.month
 AND e.occurred_at < gs.month + INTERVAL '1 month'
GROUP BY gs.month
ORDER BY gs.month ASC;
```

Judge: Query correctly returns distinct users per month in 2025 ordered ascending, but uses generate_series with interval logic instead of the rubric-specified date_trunc('month', occurred_at) approach, which is less direct and doesn't follow the exact required pattern.

Frequently asked

What does this prompt test?

SQL: Uses date_trunc('month', occurred_at), COUNT(DISTINCT user_id), a 2025 filter, grouped and ordered by month. The deterministic checks are regex, regex, regex, regex, contains.

Which model should I pick for this task?

Bench results are not in yet for this family and prompt.

Same prompt, other families

Run it yourself

Re-run this prompt on DeepSeek with your own OpenRouter key, or tweak the wording and see what changes.