Search papers, labs, and topics across Lattice.
This paper introduces MPC-Patch-Bench, a novel repository-level benchmark specifically designed for evaluating Large Language Model (LLM) code repair in Secure Multi-Party Computation (MPC) software. The benchmark addresses critical shortcomings in existing evaluation frameworks by implementing a Data Curation Framework that filters pull requests through cryptographic layers and synthesizes problem statements, alongside an MPC Verifier that conducts rigorous security and numerical-fidelity checks. Results show that even the most capable LLMs only resolve 22.9% of tasks, with further verification reducing this to 17.1%, highlighting the significant challenges in ensuring cryptographic safety in code repairs.
LLMs struggle with MPC code repair, achieving only 22.9% functional resolution and even lower verified success rates due to stringent security requirements.
Repository-level benchmarks for evaluating Large Language Model (LLM) code repair on Secure Multi-Party Computation (MPC) software do not yet exist, and directly transplanting general-purpose benchmarks such as SWE-bench fails on three structural fronts: (i) MPC repositories are dominated by generic Python infrastructure rather than cryptographic logic; (ii) high-value MPC fixes lack the standardized tests rigid extraction pipelines require; and (iii) standard fail-to-pass evaluation is insufficient for code that must also be cryptographically safe. MPC is increasingly deployed for privacy-preserving machine learning, biomedical collaboration, and secure analytics. Existing MPC-specific code-synthesis efforts cover only operator-level or single-framework tasks; evaluating LLM agents on real repository-level MPC repair instead demands MPC-aware data curation and a verifier matched to the security and numerical-fidelity guarantees MPC programs must obey neither of which existing benchmarks provide. We introduce MPC-Patch-Bench, a repository-level benchmark organised around two frameworks. (1)The Data Curation Framework combines a domain-specific curation agent that filters raw pull requests through three cryptographic layers with a human-AI completion engine that synthesizes missing problem statements and Fail-to-Pass/Pass-to-Pass tests, yielding 205 fully verified instances. (2)The MPC Verifier provides dedicated security and numerical-fidelity checks via dynamic differential testing against plaintext oracles and MPC-specific static analysis rules that flag unsafe reveals, insecure arithmetic, and illegal public/private casts. The strongest evaluated LLM functionally resolves only 22.9% of MPC-Patch-Bench tasks; the MPC Verifier further reduces verified resolution to 17.1%, with up to 40% of functionally-passing patches rejected for cryptographic or numerical-fidelity violations.
