Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the impact of sampling position on the geometric regularization of LLM hidden-state trajectories during multi-step reasoning, building on the Semantic Tube Prediction (STP) framework. By applying STP at semantic reasoning step boundaries, the authors achieved a remarkable 168x improvement in multi-step latent prediction accuracy on the ProcessBench dataset compared to traditional random-token STP methods. Additionally, the study reveals that STP-shaped trajectories exhibit smooth curves rather than straight lines, highlighting a tradeoff between generation quality and geometric purity in LLM training.
Sampling position is the key to unlocking a 168x boost in multi-step latent prediction accuracy in LLMs, reshaping our understanding of geometric regularization in AI reasoning.
Semantic Tube Prediction (STP) leverages representation geometric to regularize LLM hidden-state trajectories toward locally linear geodesics during fine-tuning, thereby greatly improving data efficiency. The original STP recipe samples random token sub-spans, which is compatible with the base large language model (LLM) training architecture. Inspired by STP, we are interested to investigate whether the sampling position can further enhance the semantic structure of multi-step reasoning, and hence affect its geometric impact. We applied STP at consecutive semantic reasoning step boundaries and achieved 168x more accurate multi-step latent prediction than frozen baselines on ProcessBench (3,400 samples), compared to only 4x for the random-token STP. Probing the latent manifold with a learned non-linear predictor reveals that STP-shaped trajectories are smooth curves, not straight lines: a 3-layer MLP reduces prediction error by a further 3-12x over linear extrapolation on step-boundary models. Removing the language modeling loss yields trajectories that are 2x more MLP-predictable than the combined loss, revealing a tradeoff between generation quality and geometric purity. Our results identify sampling position as the critical variable in geometric regularization and establish multi-step latent prediction MSE as a new evaluation metric for this class of methods.