FROM EVIDENCE TO ENDPOINT
Your task. A living leaderboard.
Follow the whole process, from your first example to a ranking that keeps evolving.
01 / CREATE
Define what a good answer looks like.
Start with your task, a prompt and a way to judge the result.
- Name your leaderboard and describe the task.
- Choose exact match, an LLM judge or human evaluation.
- Use one evaluation setup to compare your candidates.
Leaderboard setup
Illustrative setup- YOUR TASK
- Support triage
- TASK PROMPT
- Classify each support request as Billing, Access, Technical or Other.
- SCORING
- Exact match · LLM judge · Human evaluation
Same task. Same criteria. A comparison you can inspect.
POST https://app.drgero.ai/api/leaderboardsRequest template.
02 / CONNECT
Bring the evidence from your agent.
Turn real inputs and reference answers into a dataset for evaluation.
- Send input/output examples through the API.
- Import a labeled Hugging Face dataset.
- Supply or review reference answers before evaluating.
Your evaluation dataset
Illustrative setup- INPUT
- I was charged twice this month.
- REFERENCE ANSWER
- Billing
- DATASET VERSION
- Examples checked and ready for evaluation
Capture agent I/O or import labeled examples.
POST https://app.drgero.ai/v1/leaderboard/{id}/dataset/push
POST https://app.drgero.ai/api/leaderboards/{id}/dataset/uploadRequest template · Replace the identifiers with your own.
03 / AUTO SELECT
Find candidates that fit your task.
Narrow the model pool using your quality, cost and latency preferences.
- Set the requirements that matter for your application.
- Filter the model catalog to a manageable shortlist.
- Evaluate the shortlist on your own examples.
Candidate selection
Illustrative setup- STARTING POOL
- 1,000 models
- FILTERS
- Task fit → Cost → Latency
- SHORTLIST
- 5 candidates · 995 discarded
Illustrative catalog and shortlist from the walkthrough.
POST https://app.drgero.ai/api/leaderboards/{id}/models/auto-selectRequest template · Replace the identifiers with your own.
04 / RANK
Let your data set the order.
Evaluate candidates on the same examples and compare the tradeoffs.
- Compare task score, response time and cost together.
- Inspect the evidence behind each ranking.
- Choose the model that meets your task requirements.
Leaderboard results
Illustrative setup- COMPARE
- Task score · Latency · Cost
- REFERENCE WINNER
- Ling 3.0 Flash
- EVIDENCE
- The same evaluation examples for every candidate
Reference metrics are task-specific; some costs may include estimates.
POST https://app.drgero.ai/api/leaderboards/{id}/runRequest template · Replace the identifiers with your own.
05 / DEPLOY
One endpoint. Your routing strategy.
Connect your application to a stable private API and choose how requests are served.
- Follow the completed, eligible ranking winner.
- Pin a model when you need a fixed choice.
- Use a balanced mix to route traffic across models.
Private inference endpoint
Illustrative setup- REQUEST
- Your application → Dr. Gero → Eligible model
- RESPONSE
- Billing
One request, one model’s answer. Your endpoint stays the same.
POST https://app.drgero.ai/v1/leaderboard/{id}/inferenceRequest template · Replace the identifiers with your own.
06 / FINE-TUNE
Build a specialist for your task.
Optionally train a new model version using curated supervised examples.
- Prepare training data and start a fine-tuning run.
- Evaluate the new version on held-out examples.
- Review and approve it before adding or serving it.
Training and review
Illustrative setup- TRAIN
- Curated examples + Ministral 3 3B 2512
- NEW VERSION
- Ministral tuned v1
- REVIEW GATES
- Evaluate on holdout → Review results → Approve as candidate
Fine-tuning is optional and user initiated.
POST https://app.drgero.ai/api/models/{model_id}/fine-tune/runRequest template · Replace the identifiers with your own.
07 / EVOLVE
Keep the ranking current.
Refresh your comparison as new evidence and approved candidates arrive.
- Schedule re-evaluation when your dataset changes.
- Approve tuned candidates and remove others from the comparison.
- Keep serving through the same application endpoint.
An evolving leaderboard
Illustrative setup- NEW EVIDENCE
- 1,000 new examples → Re-evaluate
- APPROVED CANDIDATES
- Tuned Ministral and tuned Nemo join after review
- APPLICATION
- Same URL · Follow the completed, eligible winner
Later ranking changes are illustrative. Candidate changes require your approval.
PATCH https://app.drgero.ai/api/leaderboards/{id}Request template · Replace the identifiers with your own.
Start with your task.
Build your first leaderboard and put your models to the test.
Start free ↗