Open-source / open-weight only: No. Include both open and closed models.
These example filters shortlist candidates. Evaluate them on your own task to measure actual quality, latency and cost.
4 Create ranking
Compare quality, latency and cost on your task. Here is a rounded example from one evaluation run.
Model
Score
Latency
Cost / 1K
GPT-5.6 Sol
66%
2.6s
$1.19
Gemini 3.6 Flash
52%
3.1s
$1.49
GLM 5.1
48%
12.2s
$1.33
Claude Opus 5
46%
5.8s
$5.93
MiniMax M3
42%
5.9s
$0.31
Each run snapshot records the dataset, model versions and evaluator. Keep an immutable record of that run.
Example run, not a general model benchmark. Costs are USD per 1,000 evaluation examples, calculated from average cost per example. Scores, latency and costs are rounded; your task and request sizes will change the results.
5 Optimal Mix of Models
Split traffic between a higher-scoring model and a lower-cost model. Each request goes to just one model, not both.
In this example, Gero-0 routes approximately half of requests to GPT-5.6 Sol and half to MiniMax M3 over time.
Model
Score
Latency
Cost / 1K
GPT-5.6 Sol
66%
2.6s
$1.19
Gero-0 (50/50 mix)
54%
4.3s
$0.75
MiniMax M3
42%
5.9s
$0.31
The trade-off: about 37% lower average cost than GPT-5.6 Sol alone, with a 54% score instead of 66% in this example. The mix gives a higher score than using MiniMax M3 alone, at a higher cost.
Gero-0 is a routing mix, not a new model or a combination of two answers. Weights depend on the evaluation results; 50/50 is this example, not a fixed setting. Available in automatic evaluation. Without an eligible pair, use the ranking winner.
6 Serve · One endpoint
One stable URL. Automation optional.
POST /v1/leaderboard/{id}/inference
Your application calls the Dr. Gero endpoint. Your routing choice determines which selected model responds: the ranking winner, the Gero-0 model mix, or a pinned model.
Human Eval: reviewed common-case winner.
7 Continuous learning · Re-evaluate
Optional, in automatic evaluation. Trigger new runs when your data or models change.
Cron schedule
Daily, weekly, monthly, or custom cron.
Accepted PUSH rows
New accepted PUSH rows reach the configured threshold N.
Linked model version
A new linked Dr. Gero model version is available.
Re-evaluate to create a new ranking. Control the triggers; measure the changes.
8 Auto fine-tuning
Train on your data. Evaluate every new version.
Schedule fine-tuning on suitable training examples. Evaluate each new model version on your task before using it in production.
Same endpoint. New evidence.
Real runs.Test your task.
Examples
Real agents. Real rankings.
Real examples from our users’ agents. Even SOTA models can score poorly on specific tasks.
Your task. Your benchmark.
01
Real user-agent run
Agent run 01
Dataset v1 / 12 ranking entries
Recorded results
Gero-0 score per $1.30×vs. GPT-5.6 Sol
−37%inference cost
54%task scorevs. 66%
User-agent run 01: task-specific score, latency, recorded inference cost per 1,000 examples, and score per dollar indexed to the first-ranked model.
Model
Score
Latency
$ / 1Kexamples
Score / $vs. #1
#1GPT-5.6 Sol
66%
2.63 s
$1.19
1.00×
#2Gero-050/50 MIX
54%
4.26 s
$0.75
1.30×
#3Gemini 3.6 Flash
52%
3.14 s
$1.49
0.63×
#4GLM 5.1
48%
12.22 s
$1.33
0.65×
#5Qwen3.8 2.4T A95B
48%
6.53 s
$2.08
0.42×
#6Claude Opus 5
46%
5.80 s
$5.93
0.14×
#7MiniMax M3
42%
5.89 s
$0.31
2.45×
#8DeepSeek V4 Flash 0731
40%
14.25 s
$1.01
0.72×
#9MiMo-V2.5
32%
13.97 s
$1.09
0.53×
#10Hy-MT2-30B-A3B
20%
859 ms
$0.08
4.38×
#11Nemotron 3.5 Lightning
8%
15.89 s
$0.00
Free
#12Vinclat-qwen
6%
1.60 s
$1.17
0.09×
6 selected rows
The Gero-0 mix
50% GPT-5.6 Sol+50% MiniMax M3
One request → one model.
02
Real user-agent run
Agent run 02
Dataset v2 / 11 ranking entries
Recorded results
Gero-0 score per $1.90×vs. gpt-oss-120b
−50%inference cost
95%task scorevs. 100%
User-agent run 02: task-specific score, latency, recorded inference cost per 1,000 examples, and score per dollar indexed to the first-ranked model.
Model
Score
Latency
$ / 1Kexamples
Score / $vs. #1
#1gpt-oss-120b
100%
1.41 s
$0.59
1.00×
#2Claude Opus 5 (Fast)
100%
2.80 s
$249.82
<0.01×
#3MiMo-V2.5
95%
5.33 s
$1.32
0.42×
#4Gero-050/50 MIX
95%
6.84 s
$0.29
1.90×
#5Laguna S 2.1
90%
12.26 s
$0.00
Free
#6DeepSeek V4 Flash 0731
90%
5.35 s
$1.34
0.40×
#7Step 3.7 Flash
90%
3.87 s
$4.26
0.12×
#8Gemini 3.6 Flash
90%
1.63 s
$11.10
0.05×
#9MiniMax M3
80%
3.44 s
$4.23
0.11×
#10qwen/qwen3.6-flash
70%
1.77 s
$3.07
0.13×
#11MMMD
0%
—
—
—
6 selected rows
The Gero-0 mix
50% gpt-oss-120b+50% Laguna S 2.1 (free)
One request → one model.
Across the two real runs
Same model. Different results.
Task-specific results matter more than a model’s reputation.
The same models' observed scores in the two supplied user-agent runs.
Model
Run 01
Run 02
Difference
MiMo-V2.5
32%
95%
+63 pp
DeepSeek V4 Flash 0731
40%
90%
+50 pp
MiniMax M3
42%
80%
+38 pp
Gemini 3.6 Flash
52%
90%
+38 pp
Different evaluation snapshots; not a claim of model improvement. pp = percentage points.
Calculated from the same runs
Same volume. Lower spend.
Gero-0 trades some task score for lower inference cost.
10,000 examples
Projected inference spend for the selected number of evaluation examples, using each run's average recorded cost. This is a calculation, not a new measurement.
Run / score
#1 model
Gero-0
Saved
Run 0166% → 54% score
$11.88
$7.48
$4.40
Run 02100% → 95% score
$5.89
$2.94
$2.94
#1: GPT-5.6 Sol in run 01; gpt-oss-120b in run 02. Volume × recorded average cost. Platform fees excluded.