Search papers, labs, and topics across Lattice.
The paper introduces NESTOR, a nested Mixture-of-Experts (MoE) neural operator, to improve the performance of large-scale PDE pre-training. NESTOR employs an image-level MoE to capture global dependencies and a token-level Sub-MoE to focus on local dependencies within PDE systems. Experiments demonstrate that pre-training NESTOR on twelve diverse PDE datasets enhances generalization and transferability to downstream tasks.
A nested Mixture-of-Experts architecture lets neural operators pre-trained on diverse PDEs transfer more effectively to downstream tasks.
Neural operators have emerged as an efficient paradigm for solving PDEs, overcoming the limitations of traditional numerical methods and significantly improving computational efficiency. However, due to the diversity and complexity of PDE systems, existing neural operators typically rely on a single network architecture, which limits their capacity to fully capture heterogeneous features and complex system dependencies. This constraint poses a bottleneck for large-scale PDE pre-training based on neural operators. To address these challenges, we propose a large-scale PDE pre-trained neural operator based on a nested Mixture-of-Experts (MoE) framework. In particular, the image-level MoE is designed to capture global dependencies, while the token-level Sub-MoE focuses on local dependencies. Our model can selectively activate the most suitable expert networks for a given input, thereby enhancing generalization and transferability. We conduct large-scale pre-training on twelve PDE datasets from diverse sources and successfully transfer the model to downstream tasks. Extensive experiments demonstrate the effectiveness of our approach.
