25 stories tagged benchmarks.
LLM model routing promises cost savings by sending easy prompts to cheap models. A new study of four open-source routers finds a fixed mid-tier baseline matches the best router on every benchmark, with no statistically significant task-specific advantage detected.
Between-day variance reaches 8.4 points, three times the 2.8-point within-day noise: LLM performance drift silently regresses API models.
Frontier models tell eval transcripts from deployment at up to 0.89 AUROC, near the human 0.92: EvalDetectBench measures LLM evaluation awareness.
On the 550-query ESQ-Bench Oracle NL2SQL benchmark, GPT-4o falls from 79.8 to 57.2 percent as schemas grow, and 92 percent of passes diverge.
SOP-Bench tests AI agents on 2,000+ real business SOPs across 12 domains. The best models score 25% on the hardest procedures, and upgrades can lower success rates.
LLM instruction following breaks past 5 to 6 simultaneous constraints. Per-constraint pass rates decay at 72% times 0.922 per step, probe success below 2% by k=9.
Reasoning models follow deceptive power-seeking instructions 58 percent of the time on the SysAdmin benchmark, three times the base model rate.
Long-Horizon-Terminal-Bench tests AI agents on multi-step terminal tasks with dense reward grading. Top models solve under 30 percent of long-horizon tasks, exposing a durability gap.
Under AgentLens trajectory review, coding agent success rates drop 30 to 60 percent, so production readiness is roughly half the benchmark headline.
Models tend to converge on similar answers. In Springboards tests, Flint scored 7.47 distinct replies out of 10 against LLM groupthink.
When top agents cluster at ceiling scores, benchmark saturation moves the useful signal to cost, reliability and uplift, CORE-Bench shows.
Attacks activated in 78.9 to 89.3 percent of tested agent runs on RIFT-Bench, a dynamic agentic red-teaming benchmark.
Diffusion language models generate by denoising full sequences, but an 8 model, 8 benchmark study shows deployment depends on inference choices.
On the MosaicLeaks privacy benchmark for research agents, PA-DR training cut answer or full-information leakage from 34.0 to 9.9 percent.
A 56.2 times speed win at 1M tokens makes subquadratic sparse attention promising, but treat it as gated infrastructure for now.
Diversifying the first query adds 5 to 7 pass@4 points: DivInit seeds agentic search without any training.
WorkBench agents now solve 89 percent of workplace tasks with 2.5 percent harmful actions, changing the risk math for builders.
The first AgentPerf results for concurrent AI agents put NVIDIA GB300 NVL72 at up to 20 times Hopper's efficiency per watt.
Old benchmarks maxed out and the AGI goalposts keep moving, so the AI plateau is mostly an illusion while frontier capability keeps climbing.
A new ACL paper finds LLM judges stable on reruns but reversible after challenge, so evals must test interaction, not just scores.
With 100 tasks for monitoring agents, SentinelBench tests patience, latency and tool spend before you ship an agent that has to wait.
EVA-Bench Data 2.0 is a 213 scenario dataset for testing enterprise voice agents across airline, IT, and healthcare HR workflows.
Amazon says revising labels before grading lifted expert accuracy from 60.8 to 90.9 percent. Audit then score turns AI ground truth into a process.
Generalist agents can run data curation loops, but one benchmark shows they need scaffolds to beat baselines at 10 percent data budget.
The gap between the best AI models and the rest is collapsing, and aggregate benchmark leaderboard scores now hide more than they reveal.