Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper constructs a fixed-weight Transformer that precisely executes a continuous causal discovery algorithm, demonstrating that Transformers can implement iterative optimization procedures. By explicitly encoding the algorithm's update rule and multiplier, the Transformer achieves exact reproduction of a reference solver's steps, highlighting the importance of retaining the multiplier for accurate execution. Experiments confirm the Transformer's fidelity to the reference update and its ability to inherit successes and failures on benchmark causal discovery tasks, distinguishing algorithm execution from causal recovery.
Fixed-weight Transformers can perfectly replicate iterative causal discovery algorithms, revealing a critical distinction between executing an algorithm and recovering the true causal structure.
Transformers can execute algorithms on data given in their input. We ask whether they can do the same for causal discovery. We study a standard continuous method that repeatedly updates a candidate causal graph while enforcing acyclicity. We explicitly construct a fixed-weight transformer whose forward pass exactly reproduces one update of this method, so repeated blocks reproduce its optimization trajectory. The transformer carries the current graph and the algorithm's multiplier between updates. We show that retaining the multiplier is essential for exact execution, since different multiplier values can lead to different next updates. We also give conditions under which, within a fixed stage, the number of updates needed to reach a target accuracy can be computed in advance and rounding errors stay bounded as depth grows. Experiments show that the constructed block agrees with a reference update to floating-point precision, while arithmetic replay on synthetic data and seven published benchmark network topologies inherits the reference solver's successes and failures. This separates accurate algorithm execution from accurate causal recovery. In contrast, the ordinary attention models tested under our training budgets do not reliably execute the update or transfer to larger graphs. Whether gradient training can learn an executor in the architecture class of the construction remains open.