Search papers, labs, and topics across Lattice.
FlowPipe introduces a novel framework for constructing data preparation pipelines using Conditional Generative Flow Networks (C-GFlowNets) to enhance the efficiency of operator sequence selection. By addressing limitations in existing Multi-DQN methods, such as weak credit assignment and inefficient exploration, FlowPipe connects early pipeline decisions to terminal validation rewards through a Trajectory Balance objective. The framework's integration of LLM-derived logical priors and failure awareness leads to significant improvements in accuracy and training convergence, achieving an average accuracy boost of 11.96% and 12.5x faster training times across multiple datasets.
FlowPipe achieves a remarkable 11.96% accuracy improvement and 12.5x faster training convergence for data preparation pipelines by leveraging LLMs and advanced flow generation techniques.
Data preparation pipelines improve data quality in machine learning by transforming raw tables into learning-ready data through sequential cleaning and feature transformation operators. However, automatically constructing such pipelines is computationally difficult because operator sequences are combinatorial and end-to-end evaluation is expensive. Existing state-of-the-art (SOTA) Multi-DQN methods still face three key limitations: decoupled value estimators weaken long-horizon credit assignment, dataset context is only weakly injected into the policy, and exploration is inefficient in a sparse search space with many invalid states. To address these issues, we propose FlowPipe, a unified framework that formulates pipeline synthesis as conditional probabilistic flow generation over a directed acyclic graph. FlowPipe uses Conditional Generative Flow Networks (C-GFlowNets) with a Trajectory Balance objective to connect terminal validation rewards with early pipeline decisions. It further introduces Deep Semantic Modulation through Feature-wise Linear Modulation (FiLM), allowing LLM-derived logical priors to condition the policy's internal activations according to dataset semantics. In addition, FlowPipe incorporates failure awareness into the flow objective to avoid invalid states and concentrate search on high-potential regions. Experiments on two benchmark suites with 74 real-world datasets show that FlowPipe outperforms SOTA baselines, improving accuracy by 11.96% on average and achieving 12.5x faster training convergence. Source code is available at https://github.com/KunyuNi/FlowPipe.
