Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study benchmarks code LLMs (Qwen2.5-Coder and DeepSeek-Coder) against a state-of-the-art rule-based injector (ProFIPy) to evaluate whether generative models can replace hand-engineered fault injection in distributed cloud infrastructure. Evaluating both paradigms across OpenStack's Nova and Cinder services reveals that while baseline activation and observable-failure rates match, the operational failure distributions diverge sharply. LLM-generated mutations induce significantly more catastrophic failures in Nova, whereas rule-based approaches surface predominantly silent and multi-component defects, showing that LLMs expand the behavioral test surface rather than strictly dominating fixed fault models.
LLMs don't just mimic human bugs—they fundamentally skew system failure distributions toward catastrophic crashes that standard deterministic fault-injection pipelines fail to surface.
Software Fault Injection (SFI) supports testing of cloud systems by introducing software defects and observing their manifestation. Rule-based injectors such as ProFIPy provide controlled and reproducible source-level mutations but require fault patterns to be encoded manually. Large Language Models (LLMs) offer a data-driven alternative by generating context-dependent software faults. We compare two code LLMs, Qwen2.5-Coder and DeepSeek-Coder, with ProFIPy in OpenStack's Nova and Cinder services. On shared injection targets, activation and observable-failure rates are comparable, but operational profiles differ: LLM-generated faults produce more Catastrophic outcomes on Nova, whereas ProFIPy produces more Silent and Multi-component effects. The sampled LLM outputs also differ in how they manifest failure, while showing greater agreement in their propagation scope. These findings show that LLM-based fault injection extends the behavioral coverage of fixed fault models without establishing general superiority, and that practical adoption still requires controlled generation, runtime validation, system-level oracles, and reproducible experimental provenance.