Benchmarks
Standardized evaluations of LLM reliability on real-world tasks. Open methodology, reproducible results, real tool calls against real APIs.
Thai Speech-to-Text
The models best at Thai drop every English word
PII Detection
Flagship-quality PII detection from a 3× smaller model
Sheets Benchmark
Tool architecture matters more than model choice
Email Benchmark
Draft quality, thread handling, attachment accuracy
Calendar Benchmark
Scheduling accuracy, conflict detection, timezone handling
GitHub Benchmark
PR reviews, issue triage, code search accuracy
Playwright Benchmark
Web task completion, UI resilience, action policy safety
Robotics VLA Benchmark
Vision-language-action task completion in simulated environments
Data Analysis Benchmark
SQL generation, data cleaning, visualization accuracy
OpenUSD Benchmark
Scene graph edits, material assignments, transform accuracy
STL Benchmark
Mesh manipulation, geometric accuracy, printability validation
Why we benchmark
Most LLM benchmarks test knowledge or reasoning in isolation. We test what matters for production: can a model reliably complete multi-step tasks using real tools against real APIs, without corrupting your data?
Every benchmark uses Daslab's tool-calling infrastructure with real Google, GitHub, and Slack APIs. Tasks are defined as expected input/output pairs with cell-level validation. Results are fully reproducible.
Real APIs
Not simulated. Every tool call hits the actual Google Sheets, Gmail, or GitHub API. Errors are real errors.
Cell-level validation
Success isn't "did it respond" — it's "does the spreadsheet contain exactly the right values in the right cells."
Open methodology
Task definitions, evaluation scripts, and raw results are all public. Run the same benchmark on your own infrastructure.
Cost-aware
We track tokens, retries, and dollar cost per task. The cheapest model that works is often the right choice.