Research
LLM instruction following collapses past 6 constraints
LLM instruction following breaks past 5 to 6 simultaneous constraints. Per-constraint pass rates decay at 72% times 0.922 per step, probe success below 2% by k=9.
#evaluation
9 stories tagged evaluation.
Research
Format Sensitivity Index exposes LLM benchmark gaps
The Format Sensitivity Index measures how LLM benchmark scores shift when prompt wrappers and schema constraints change. The metric exposes a blind spot in model evaluation that developers ignore at their peril.
Research
Long-Horizon-Terminal-Bench finds agents hit a long-task wall
Long-Horizon-Terminal-Bench tests AI agents on multi-step terminal tasks with dense reward grading. Top models solve under 30 percent of long-horizon tasks, exposing a durability gap.
Research
Benchmark saturation still leaves AI agents exposed
When top agents cluster at ceiling scores, benchmark saturation moves the useful signal to cost, reliability and uplift, CORE-Bench shows.
Research
LLM recommendation bias gives famous brands an edge
Famous skincare brands were recommended 100 percent of the time in a new arXiv study, a measurable LLM recommendation bias toward incumbents.
Research
SentinelBench tests whether AI agents can wait well
With 100 tasks for monitoring agents, SentinelBench tests patience, latency and tool spend before you ship an agent that has to wait.
Research
Audit then score turns AI ground truth into a process
Amazon says revising labels before grading lifted expert accuracy from 60.8 to 90.9 percent. Audit then score turns AI ground truth into a process.
Research
Multi-agent debate needs a boring data-cleaning cop
Multi-agent debate hurt generation by up to 15.5 points in data cleaning, but a grounded critic rescued detection and repair.
Research
The model leaderboard is tightening to a photo finish
The gap between the best AI models and the rest is collapsing, and aggregate benchmark leaderboard scores now hide more than they reveal.