11 stories tagged evaluation.
LLM benchmark variance is real and time-dependent. An analysis of 31,352 hourly scores shows 8.4-point swings between days and 2.8 within a day, meaning a single benchmark run can mislead your model choice.
LLM instruction following breaks past 5 to 6 simultaneous constraints. Per-constraint pass rates decay at 72% times 0.922 per step, probe success below 2% by k=9.
The AI agent evaluation gap shows 50% of enterprises shipped an agent that passed internal evals then failed in production. Only 5% fully trust automated evaluation. The gap is structural misalignment, not missing coverage.
The Format Sensitivity Index measures how LLM benchmark scores shift when prompt wrappers and schema constraints change. The metric exposes a blind spot in model evaluation that developers ignore at their peril.
Long-Horizon-Terminal-Bench tests AI agents on multi-step terminal tasks with dense reward grading. Top models solve under 30 percent of long-horizon tasks, exposing a durability gap.
Benchmark saturation is when top agents cluster at ceiling scores. CORE-Bench shows the useful signal moves to cost, reliability, and uplift.
LLM recommendation bias is a measurable incumbent edge: a new arXiv paper found famous skincare brands were recommended 100 percent of the time.
SentinelBench is a 100-task benchmark for monitoring agents. Use it to test patience, latency and tool spend before you ship.
Audit then score is a benchmark protocol that revises labels before grading. Amazon says it lifted expert accuracy from 60.8% to 90.9%.
Multi-agent debate hurt generation by up to 15.5 points in data cleaning, but a grounded critic rescued detection and repair.
The performance gap between the best AI models and the rest is collapsing. Aggregate leaderboard scores now hide more than they reveal about real strengths.