Drop your AI Agents inference cost

1 Create leaderboard

Define the task. Choose how to judge answers.

Task prompt

What is a good response to this customer question?

LLM as judge

Score quality against the expected answer and evaluation prompt.

Exact match

Compare deterministic outputs.

Human evaluation

Enter cases, compare model answers and mark every correct response.

2 Integrate your Agent

Connect agent traces, use a dataset, or start with manual cases.

Hugging Face

Load a JSON, JSONL, or CSV dataset.

Live agent traces · PUSH

Application input/output events.

Human Eval · manual cases

Create cases and review model responses. Review after choosing models.

3 Auto-select models

Set your cost, latency and model preferences. Example configuration:

Number of models

5 candidates.

Cost

Maximum input price: $5 per million tokens. Maximum output price: $25 per million tokens.

Latency

P95 preference: 15 seconds. P99 preference: 30 seconds.

Model availability

Open-source / open-weight only: No. Include both open and closed models.

These example filters shortlist candidates. Evaluate them on your own task to measure actual quality, latency and cost.

4 Create ranking

Compare quality, latency and cost on your task. Here is a rounded example from one evaluation run.

Each run snapshot records the dataset, model versions and evaluator. Keep an immutable record of that run.

Example run, not a general model benchmark. Costs are USD per 1,000 evaluation examples, calculated from average cost per example. Scores, latency and costs are rounded; your task and request sizes will change the results.

5 Optimal Mix of Models

Split traffic between a higher-scoring model and a lower-cost model. Each request goes to just one model, not both.

In this example, Gero-0 routes approximately half of requests to GPT-5.6 Sol and half to MiniMax M3 over time.

The trade-off: about 37% lower average cost than GPT-5.6 Sol alone, with a 54% score instead of 66% in this example. The mix gives a higher score than using MiniMax M3 alone, at a higher cost.

Gero-0 is a routing mix, not a new model or a combination of two answers. Weights depend on the evaluation results; 50/50 is this example, not a fixed setting. Available in automatic evaluation. Without an eligible pair, use the ranking winner.

6 Serve · One endpoint

One stable URL. Automation optional.

POST /v1/leaderboard/{id}/inference

Your application calls the Dr. Gero endpoint. Your routing choice determines which selected model responds: the ranking winner, the Gero-0 model mix, or a pinned model.

Human Eval: reviewed common-case winner.

7 Continuous learning · Re-evaluate

Optional, in automatic evaluation. Trigger new runs when your data or models change.

Cron schedule

Daily, weekly, monthly, or custom cron.

Accepted PUSH rows

New accepted PUSH rows reach the configured threshold N.

Linked model version

A new linked Dr. Gero model version is available.

Re-evaluate to create a new ranking. Control the triggers; measure the changes.

8 Auto fine-tuning

Train on your data. Evaluate every new version.

Schedule fine-tuning on suitable training examples. Evaluate each new model version on your task before using it in production.

Same endpoint. New evidence.