Tag

#benchmarking

2 stories tagged benchmarking.

Bar chart comparing cold Claude Code run cost of $6.83 over 207 turns against AutoDev Studio's $1.70 for the same bug, showing the cost advantage of repo-aware multi-agent harnesses.

Open-source agent halves Claude Code costs on large repos

AutoDev Studio is an open-source multi-agent SDLC harness that learns a repo once and runs 7% to 75% cheaper than cold Claude Code on localized tasks. Its real benchmarks expose where repo-aware agents win and where they still lose.

Lars Cornelissen · Jul 25, 2026

Format Sensitivity Index showing benchmark score variance of up to 15 percentage points across JSON, XML, Markdown, and plain text prompt wrappers for four illustrative LLM models

Research

Format Sensitivity Index exposes LLM benchmark gaps

The Format Sensitivity Index measures how LLM benchmark scores shift when prompt wrappers and schema constraints change. The metric exposes a blind spot in model evaluation that developers ignore at their peril.

Lars Cornelissen · Jul 14, 2026