Search papers, labs, and topics across Lattice.
The paper introduces Landseer, a modular framework for integrating and evaluating compositions of machine learning defenses against threats to robustness, privacy, and fairness. They identified 35 state-of-the-art defenses and, after filtering for reproducibility, analyzed their performance within Landseer's unified evaluation process. The study reveals significant gaps in the replicability of defenses and provides insights into the challenges of defense composition.
Many ML defenses don't replicate, and composing them is even harder, highlighting the need for systematic evaluation frameworks like Landseer.
Machine learning systems face diverse threats that undermine robustness, privacy, and fairness. Although many defenses have been proposed, each typically addresses a single risk in isolation. Real-world deployments, however, require these defenses to be composed to meet multiple guarantees simultaneously. The process of composing defenses is complex and not well understood, and its impact on performance and security remains unclear. We present Landseer, a modular framework for integrating machine learning (ML) defenses into the ML lifecycle and systematically evaluating their composition. Landseer encapsulates defenses as containerized modules, allowing existing and new techniques to be plugged in with minimal effort. Its evaluation engine automates experiments across multiple metrics, supporting the study of defenses both individually and in combination. In a preliminary study, we identified 35 state-of-the-art machine learning defenses. After filtering for reproducibility, we analyzed their performance using Landseer's unified evaluation process. Our findings reveal gaps in replicability across defense families and provide insights into the challenges and opportunities in integrating multiple defenses, establishing a foundation for improving the reliability of machine learning systems.
