Search papers, labs, and topics across Lattice.
The paper introduces FPMoE, a sparse Mixture-of-Experts model for functional code generation, comprising language-specific experts for Haskell, OCaml, and Scala, along with a shared expert for cross-language functional patterns. This architecture mitigates interference observed in multi-language fine-tuning while preserving shared functional abstractions missed by per-language fine-tuning. FPMoE achieves state-of-the-art performance on FPEval, matching the performance of models 2-10x larger with only 3B active parameters.
Functional programming languages get a MoE boost: a 3B-parameter model now rivals 30B parameter models in code generation.
Despite rapid progress in LLM-based code generation, existing models are predominantly trained on imperative languages, leaving functional programming languages (FPLs) such as Haskell, OCaml, and Scala chronically underexplored, with even frontier models performing substantially worse on FPLs. Fine-tuning is a natural remedy, but our experiments show that per-language fine-tuning fails to capture shared functional abstractions, while merged multi-language fine-tuning introduces cross-language interference. To address this, we introduce FPMoE, a lightweight, open-source code generation model built on a sparse Mixture-of-Experts (MoE) architecture with three language-specific routed experts (one each for Haskell, OCaml, and Scala) and a shared expert that captures cross-language functional patterns such as monadic reasoning and type-directed programming. This design resolves both failure modes simultaneously: dedicated experts eliminate interference, while the shared expert preserves abstractions that per-language models miss. On FPEval, FPMoE substantially outperforms fine-tuned baselines and, with only 3B active parameters, matches the performance of much larger models including DeepSeek-Coder-6.7B, Qwen2.5-Coder-14B-Instruct, and Qwen3-Coder-30B-A3B.
