Search papers, labs, and topics across Lattice.
This study investigates how large language models (LLMs) perform under the pressure of multiple simultaneous constraints using a novel benchmark called Constraint Saturation Evaluation (CSE). The results reveal that while individual constraints are managed effectively, the ability to satisfy multiple constraints simultaneously collapses dramatically, with a notable performance drop occurring after just five or six constraints. Additionally, the research highlights that structural constraints are significantly more susceptible to degradation than lexical ones, indicating a critical distinction in how LLMs handle different types of compositional requirements.
LLMs can handle individual constraints well, but their ability to satisfy multiple constraints simultaneously collapses dramatically, with performance dropping below 50% at just seven constraints.
Large language models are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints - reasoning structure, safety boundaries, output schemas. Individual constraints are handled proficiently, but the compositional regime, where many must hold jointly, remains poorly characterized: how rapidly does performance degrade, what governs the degradation, and can the collapse be mitigated? We introduce Constraint Saturation Evaluation (CSE), a procedurally generated benchmark that systematically varies the number of simultaneous constraints (k), with every constraint scored by a deterministic, rule-based verifier and zero LLM-judge involvement: 15 models, 36 constraint types, 369,753 checks at k=1-12. Three findings emerge. First, per-constraint pass rate decays gradually and predictably, while the chance of satisfying all k constraints collapses - a model passing individual constraints at ~41% at k=8 succeeds on all eight just 5.7% of the time. Second, constraints do not degrade equally: structural constraints lose 2x more baseline capability per added constraint than lexical ones, ordered by a comprehension-maintenance gap that separates constraints requiring sustained tracking from binary decisions immune to composition. Third, failures are nearly independent, which is what makes the accumulation multiplicative; the residual coupling that does exist tracks shared output features rather than pairwise interference - a wrong sentence count fails every constraint that reads it. Reliable instruction following breaks down beyond 5-6 simultaneous constraints: probe-level success falls below 50% at 7 constraints for the strongest model, and at 3 or fewer for 12 of 15.
