Search papers, labs, and topics across Lattice.
SPEAR addresses knowledge interference and expert redundancy in MoE neural operators for PDE pretraining by spectrally disentangling latent features into low-frequency (transferable) and high-frequency (specialized) components. A novel knowledge-guided expert aggregation strategy mitigates redundancy by identifying and consolidating similar experts based on dataset-specific learned knowledge and routing preferences. This approach achieves superior performance across diverse PDE benchmarks while significantly reducing model complexity.
Spectral disentanglement and knowledge-guided aggregation unlock efficient, high-performance MoE neural operators for large-scale PDE pretraining, slashing expert count by 50% without sacrificing accuracy.
Large-scale pre-training has improved the generalization of neural operators across diverse PDEs. However, existing PDE foundation models still struggle with heterogeneous dynamics, where shared representations may cause knowledge interference, while mixture-of-experts (MoE) architectures suffer from increasing expert redundancy. We propose SPEAR, a spectral-disentangled MoE neural operator with knowledge-guided expert aggregation for large-scale PDE pre-training. SPEAR decouples latent features into low- and high-frequency components, enabling shared modeling of transferable dynamics and specialized learning of PDE-specific patterns. To address expert redundancy, we design a knowledge-guided expert aggregation strategy that measures expert similarity from dataset-specific learned knowledge and routing preferences, enabling the identification and consolidation of similar experts. Experiments on twelve PDE datasets and multiple downstream benchmarks demonstrate superior performance in pre-training, fine-tuning, and transfer learning. Furthermore, our aggregation strategy reduces the number of experts by 50\% while maintaining or improving prediction accuracy, achieving a balance between model efficiency and generalization for PDE foundation models.
