Search papers, labs, and topics across Lattice.
This study investigates the relationship between what large language models (LLMs) encode regarding geometric reasoning and how that information influences model behavior, using parametric CAD constraints as a testbed. By probing the hidden states of six frozen decoder-only LLMs, the authors reveal that while pretraining enhances the decoding of local geometric relations, sketch-level degrees of freedom (DOF) status is largely decodable even from randomly initialized representations. Notably, the findings indicate a divergence between decodability and actionable generation, as the models often fail to express the encoded information effectively despite its presence.
Decodable information in LLMs doesn't guarantee actionable outputs, revealing a critical gap in how these models handle geometric constraints.
Large language models (LLMs) have demonstrated strong performance on structured reasoning tasks, but what they encode and whether it informs model behavior remain unclear. We investigate this question through geometric reasoning, using parametric CAD constraints as a controlled testbed for separating local pairwise relations from sketch-level constraint status. By probing the hidden states of six frozen decoder-only LLMs, we examine four properties: linear decodability, forced-choice generation, activation-level influence, and behavioral steerability. Pretraining substantially improves the decoding of local geometric relations, and this advantage persists after accounting for positional cues with shuffled-order controls. In contrast, sketch-level DOF status is already highly decodable from randomly initialized representations and improves only modestly with pretraining, indicating that much of its probe performance is available without learned weights. Further analyses show that decodable information is not always actionable. Generation often fails to express this information, and on the two intervention-tested backbones, activation-restoration effects at the patched entity position vanish while decodability persists across depth. Mean-difference steering also does not reliably control outputs. These results show that decodability, generation, activation-level influence, and steerability can diverge in the tested setting. The audit provides a controlled way to distinguish failures to encode geometric structure from failures to express or control encoded information.
