Model reliability by task

Aggregate cost and reliability for the models teams actually run on Daslab, grouped by the kind of work — not who ran what or how much, just how well each model does the job and what it costs.

How this is measured. Every run on Daslab over the last 60 days is placed into a fixed task set by the tools it called, then aggregated by the model its turns actually ran on — a run that switched models mid-way is evidence about routing rather than about any one model, so it is excluded rather than credited to one. Reliability = the share of runs that finished without error (running runs excluded); it does not yet judge whether the answer was correct. Tool errors = the share of a model's tool calls that the tool rejected. Cost / task = median spend per finished run. ★ marks the cost / reliability frontier: no other model is both cheaper and more reliable. ° marks a provisional cell.

Why the numbers are banded. Percentages are published in ten-point ranges on purpose. An exact percentage can be inverted — only certain values are reachable from a given number of runs, so “80%” would quietly disclose the sample it came from. Bands collapse many fractions onto one label. We publish how well the models do, never how much we run: no volumes, no counts, no per-customer data. Metrics that would need to know what the user wanted — how many turns a task took, whether depth was wanted — are deliberately absent, because fewer turns is not obviously better and a leaderboard that says so would reward models that give up early.