Every production LLM prompt is a stack of constraints. Respond in JSON. Keep it under 200 tokens. Use this schema. Include a summary. Cite your sources. Do not mention X. Each line is one more constraint the model must satisfy simultaneously, and each one multiplies the odds of failure.
A new benchmark called Constraint Saturation Evaluation (CSE) puts a number on that tax. The study, posted to arXiv on August 15, 2026, finds that all 15 models tested break down beyond 5 to 6 simultaneous constraints. The strongest model drops below 50% probe-level success at 7 constraints. Twelve of fifteen models cross that line at 3 or fewer. The benchmark covers 36 constraint types across 8 processing dimensions, with 369,753 deterministic checks at k=1 to 12, and the pattern holds across every model family tested.
One author, Mariya Vasileva, notes affiliation with Meta Superintelligence Labs, making this one of the more rigorous constraint-following studies to come out of a frontier lab.
How many constraints can an LLM actually follow at once?
Fewer than you think. The CSE benchmark, introduced by Vasileva and colleagues, procedurally generates probes that stack verifiable constraints one at a time. Each constraint is checked by a deterministic function, not an LLM judge. That design choice matters: the same compositional degradation that makes models fail instructions would also corrupt an LLM-based evaluator. The benchmark sidesteps that confound entirely.
The evaluation covers 15 models across 8 families at k=1 through k=12 simultaneous constraints. At k=1, models start strong: 70.7% aggregate probe-level success. By k=4 to 7, performance steepens into a cliff. By k=10 or above, probe-level success is near zero across the board. The authors call this a phase transition: a threshold region where the system flips from mostly working to mostly failing, and the transition is consistent across all 8 model families despite different architectures, scales, and training approaches.
The per-constraint pass rate tells a gentler story. It follows a clean multiplicative model: 72.0% times 0.922 to the power of k minus 1, with a held-out mean absolute error of just 0.2 percentage points. Each additional constraint shaves about 7.8% off the per-constraint success rate. The decline is real but gradual.
The probe-level success rate, which requires all constraints to pass simultaneously, is where the cliff lives. At k=8, a model passing individual constraints at roughly 41% succeeds on all eight just 5.7% of the time. By k=9, probe-level success drops below 2%.
The chart above shows the per-constraint pass rate declining along the multiplicative model, from 72.0% at one constraint to 29.4% at twelve. That gradual slope looks manageable on its own. The collapse happens when you multiply k of those rates together to get the probability that every constraint holds at once.
Why does per-constraint success stay high while joint success collapses?
Because the failures compound. The paper's central finding is that constraint failures are nearly independent, with a mean correlation coefficient of just phi = +0.067 across shared output features. When failures are roughly independent, the probability that all k constraints pass is approximately the product of k individual pass rates.
That multiplicative structure is what makes the collapse predictable and brutal. If each constraint passes at 70%, two together succeed 49% of the time. Three succeed 34%. Five succeed 17%. Seven succeed 8%. The math is simple. The implications for production systems are not.
The paper also tested whether constraint failures interfere with each other through shared output features, like a wrong sentence count that cascades into every constraint reading it. The residual coupling does exist, but it is weak. The dominant force is accumulation, not interference. You cannot fix the problem by reordering constraints or finding better pairings. The only lever is per-constraint reliability, full stop.
A separate study, Instruction Stacking Collapse, corroborates the finding with production-tier models. It tested Claude Sonnet 4.6, GPT-5-mini, and Gemini 2.5 Flash on 24 verifier-checked instructions, stacked one to twenty at a time. The follow rate fell from roughly 96% at low constraint counts to as low as 20% under heavy stacking. Different benchmark, different models, same phase transition.
Which constraints break first under load?
Not all constraints degrade equally. The paper identifies a depth-of-processing hierarchy that governs which constraints fail first. Structural constraints degrade 2 times faster than lexical ones, with retention-normalized slopes of negative 0.073 for structural and negative 0.036 for lexical, and a 95% confidence interval on the ratio of 1.9 to 2.3.
Structural constraints include things like sentence count, output ordering, and formatting schemas. They require the model to sustain a representation of its output structure across the entire generation. Lexical constraints include word inclusion and length limits, which are binary decisions the model can make once and move on.
The paper calls this the comprehension-maintenance gap. Structural constraints demand sustained tracking, not just comprehension. Under load, the maintenance fails first.
The chart above shows the projected per-constraint pass rates for structural and lexical constraints, derived from the reported retention slopes. Structural constraints start at the same 72% baseline but fall to roughly 32.3% by k=12, while lexical constraints hold above 48.4%. That 2 times gap is consistent across all 36 constraint types and 15 models tested.
A related finding from a companion paper, LLMs Know the Constraint But Do Not Use It, traces this to an activation bottleneck. Models encode constraints in their internal representations but do not always route that knowledge into generation. The model knows it should produce exactly three sentences. It produces four anyway.
What does this mean for your production LLM stack?
If your prompts stack more than 5 or 6 hard constraints, you are operating past the phase transition. Here is what that costs you:
- Reliability drops multiplicatively, not linearly. Every constraint you add taxes the success rate of every other constraint. Going from 3 to 6 constraints can drop your pass rate from 35% to 6%, even if each individual constraint passes 70% of the time.
- Structural constraints are your weakest link. JSON schema compliance, output ordering, and format requirements are the first to break under load. These are often the constraints you care about most for downstream parsing.
- Safety boundaries count as constraints. If you are stacking content rules, format requirements, and safety guidelines in one prompt, you are already deep into the failure zone.
- Agents compound the problem. If you are building agentic coding pipelines that run autonomously, each step in the chain adds its own constraints. A 5-step pipeline with 3 constraints per step is effectively a 15-constraint problem, and the failure modes multiply.
- The multiplicative model is predictable. You can estimate your failure rate before shipping: take your per-constraint pass rate, raise it to the power of your constraint count, and that is roughly your probe-level success rate.
The practical takeaway: split your calls. Move structural constraints to deterministic post-processing. A JSON schema validator is cheaper and more reliable than asking the model to produce valid JSON on the first try. Move safety checks to a separate moderation pass. Keep the generation prompt focused on content.
Can self-correction or retries push the threshold higher?
Barely. The paper tested three inference-time interventions, and all three moved the threshold by one to two constraints at most.
Post-hoc self-correction, where the model checks its own output and fixes violations, delays the phase transition by about one to two constraints. Best-of-5 retries, where the model generates five responses and you keep the best, buys you roughly the same. Pre-generation planning, where the model outlines its approach before generating, does not move the threshold at all.
The reason is structural. The collapse is driven by per-constraint reliability, not constraint ordering or pairing. If your per-constraint pass rate is 50% and you have 8 constraints, no amount of retrying changes the fact that the joint probability is roughly 0.5 to the 8th power. You need a better base model, not a cleverer prompt.
The MOSAIC benchmark, presented at EACL 2026, reaches a similar conclusion with up to 20 constraints across a modular synthetic framework. The more constraints you stack, the worse it gets, and the degradation is non-linear in every model tested.
The constraint tax you cannot prompt away
The CSE benchmark gives you something rare in LLM evaluation: a precise, deterministic, reproducible measurement of a failure mode you have probably been hitting in production without understanding why. Your constraints multiplied past the point the model could sustain them.
The number to remember is 0.922. That is the per-constraint retention factor. Every constraint you add retains 92.2% of the previous per-constraint success rate, but the joint success rate compounds downward. Six constraints at 72% per-constraint gives you roughly 14% joint success. Eight gives you 5%. The math is unforgiving and it is the same across every model family tested.
Count your constraints. Split your calls. Move structure to code. Stop expecting the model to hold ten things in its head at once.
Sources
- arxiv.org - Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction
- arxiv.org - Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation
- arxiv.org - LLMs Know the Constraint But Do Not Use It: Activation Bottlenecks in Pragmatic Constraint Reasoning
- aclanthology.org - Deconstructing Instruction-Following: A New Benchmark for Granular Evaluation of LLM Instruction Compliance Abilities
