Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
The authors introduce Step-TP, a dataset for LLM-guided tensor program optimization featuring grounded, atomic, step-level supervision with structured chain-of-thought (CoT) reasoning. This dataset addresses the limitations of existing datasets by providing verifiable intermediate representations and composable optimization strategies, enabling reliable multi-step optimization. Experiments using Step-TP demonstrate improved LLM performance in making single-step optimization decisions within large combinatorial spaces.
LLMs can now learn to reliably optimize tensor programs step-by-step, thanks to a new dataset that provides grounded, verifiable supervision at each transformation.
Despite the strong reasoning capabilities of large language models (LLMs), optimizing the execution efficiency of tensor programs remains challenging due to the need for precise, composable transformation decisions. Recent LLM-guided approaches frame tensor program optimization as an iterative decision process, but existing datasets provide only end-to-end optimized program pairs using token-inefficient representations, lacking verifiable step-level supervision and interpretability. As a result, LLMs struggle to make reliable single-step decisions in large combinatorial optimization spaces. We introduce Step-TP, a post-training dataset for tensor program optimization that provides grounded, atomic, step-level supervision with structured chain-of-thought (CoT) reasoning. Step-TP forms a closed reasoning loop over intermediate program states, enabling reliable multi-step optimization rather than outcome imitation. Its design is guided by four principles: (i) a token-efficient, verifiable intermediate representation (IR) that deterministically lowers to TVM TIR; (ii) atomic and composable optimization strategies that decompose complex trajectories into interpretable single-step decisions; (iii) structured CoT supervision coupled with explicit IR-to-IR state transitions; and (iv) strategy filtering to balance coverage while preventing shortcut exploitation. The dataset and implementation are available at a GitHub link, https://github.com/LIUMENGFAN-gif/StepTP.