Search papers, labs, and topics across Lattice.
The paper introduces "composition collapse," where models fail to assemble stably-known facts into chains, despite high aggregate benchmark scores. They propose a "double-gate protocol" to decompose post-training gains into atomic stability, residual composition, and critical depth, revealing that aggregate metrics mask shifts in composition capability. Their diagnostic probes indicate that generation-time computation constraints, rather than permanent inability, contribute to composition failure.
Aggregate benchmark scores can be misleading: models with statistically indistinguishable atomic knowledge can exhibit composition behavior differences exceeding 40 percentage points.
Post-training is routinely evaluated through aggregate benchmark scores that treat multi-hop reasoning as a single capability -- as if a model that answers more questions correctly must be better at assembling facts. We show that this assumption can be misleading: recipes with statistically indistinguishable atomic knowledge produce composition behaviour separated by over 40 percentage points, a phenomenon we call composition collapse: the systematic failure to assemble stably-known facts into chains, invisible to aggregate metrics. We introduce a double-gate protocol that changes the estimand from an aggregate compositionality gap to residual composition failure conditioned on stable atomic access, decomposing post-training gains into three independent channels: atomic stability, residual composition, and critical depth. On a benchmark of temporal factual chains spanning depths 2--11 across four post-training recipes, this decomposition reveals that post-training objectives shift composition capability in directions that aggregate metrics mask, and suggests that claims about multi-hop reasoning improvement should be accompanied by atomic-gate-controlled composition metrics. Diagnostic probes further show that a substantial share of measured composition failure reflects generation-time computation constraints rather than permanent inability to compose.
