Every week somebody sends me a leaderboard. One model is ahead on competition mathematics, another on a coding benchmark built from popular open-source repositories, a third on a reasoning test with a Greek letter in its name. None of that tells me which model writes a Delphi unit test that compiles, or which one finds the real fault in a log from a safe deposit system in a bank basement. So we built our own benchmark from our own work. This is how, and what it showed.
The tasks
A useful benchmark measures what you actually ask the model to do. We picked five tasks that we were already running agents on, each with a way to score the answer without a human in the loop wherever possible.
| Task | Cases | How it is scored |
|---|---|---|
| Write unit tests for a Delphi unit | 40 units | Tests compile and pass against the unit, and fail against a deliberately broken copy |
| Translate a Firebird query to Transact-SQL | 120 queries | Same rows returned on both engines |
| Find the root cause in a device log | 150 past incidents | Known cause appears in the model's top three, with the supporting log lines cited |
| Triage a support ticket | 300 tickets | Category and routing match the human label |
| Explain a legacy unit | 25 units | Blind rating by three developers who know the code |
The cases come from real work: units from the codebase, queries from the migration, logs and tickets from the support archive with customer details removed. The test sets grow every month, because every new incident with a known cause is a new benchmark case.
The harness
The harness is a small .NET console application that runs in Azure DevOps on a schedule. For each model it sends the same prompt for each case, with the same instructions and the same temperature, records the answer, the token counts, the cost and the latency, and runs the scorer. Each case runs three times and we keep the median, because the same model given the same input does not always give the same answer, and a benchmark that ignores that is measuring luck.
Models are pinned to dated versions. When a vendor releases a new version, it becomes a new row in the results, not a silent change to an old one. That is the same discipline we apply to any dependency, and it turned out to matter more than I expected.
The models
In this run we compared Claude Opus 4.5, Claude Sonnet 4.5 and Claude Haiku 4.5 from Anthropic with GPT-5.1 and GPT-5 mini from OpenAI, each at its default settings for the task. By the time you read this the names will have changed. The method is the point, not the row labels.
Results
Pass rates, median of three runs, with cost per hundred cases in the last column.
| Model | Delphi tests | SQL translation | Log root cause | Ticket triage | Explanation (1 to 5) | Cost / 100 cases |
|---|---|---|---|---|---|---|
| Claude Opus 4.5 | 78% | 94% | 71% | 93% | 4.4 | highest |
| GPT-5.1 | 74% | 95% | 69% | 94% | 4.2 | high |
| Claude Sonnet 4.5 | 69% | 93% | 66% | 93% | 4.1 | medium |
| GPT-5 mini | 52% | 88% | 55% | 91% | 3.4 | low |
| Claude Haiku 4.5 | 49% | 87% | 53% | 92% | 3.3 | low |
A few things stood out immediately.
There is no single winner. The two largest models trade places depending on the task, and the differences between them are within what the run-to-run variation can explain. Anyone who tells you one vendor is simply better is selling something.
The gap depends on the task, not the model. On ticket triage, the cheapest models are within a couple of points of the most expensive ones, at a fraction of the cost. On Delphi tests the gap is enormous. Small models know less Object Pascal, and it shows.
The compiler closes the gap. When we let each model see the compiler output and retry once, the Delphi pass rates rose for every model, and the small models gained the most. A model with a feedback loop beats a smarter model without one. That was the lesson of the agentic coding article, measured.
Root cause is the hard problem. Nothing scored above three out of four on the logs, and the failures were the dangerous kind: a confident, well-written narrative pointing at the wrong component. Requiring the model to cite the exact log lines for every claim lowered the raw score a little and raised the usefulness a lot, because an unsupported claim is now visible as unsupported.
Version changes are real changes. A point release from one vendor moved its SQL translation score by several points in the wrong direction, with no change on our side. Without pinned versions and a scheduled run we would have found out from a customer.
What we do with it
The benchmark decides routing. Ticket triage goes to a small model, because the accuracy is there and the volume is high. Delphi tests and log analysis go to a large model with a retry loop. Explanations go to whichever large model is cheaper that month, because the developers rate them about the same.
It also decides upgrades. A new model version has to beat the current one on our benchmark before it goes into the pipeline or the product, in the same way a new library version has to pass the test suite. We re-run monthly, and the results go on the same dashboard as build times and test coverage.
What the leaderboards cannot tell you
Our tasks are narrow, our cases are ours, and the numbers above will not transfer to your codebase or your logs. That is exactly why you should build your own. It took one developer about two weeks, most of it spent on the scorers, and it has saved more than that in arguments alone.
Questions for you
- What are the five tasks you would put in your own benchmark, and which of them can be scored without a human?
- Have you been surprised by a model update changing results on work that had not changed?
- Do you route different tasks to different models, or does one model do everything?
Email me at bjorn.nostdahl@nostdahl.com. If you have built a benchmark like this, I would like to compare notes on the scorers; that is where all the work is.