Search papers, labs, and topics across Lattice.
This paper revisits N-version programming by evaluating 48 AI-generated implementations against a shared oracle using 1,000,000 randomized test inputs. The study finds significant common-mode failures, echoing the Knight-Leveson experiment, particularly in areas of ambiguous specification. However, it also reveals that employing diverse coding agents can substantially reduce failure rates, with triples achieving a mean failure count drop from 387.44 to 130.99, and 11,844 units exhibiting zero failures.
N-version programming with coding agents not only mirrors historical failures but also shows a dramatic reduction in errors through diversity, challenging assumptions about AI reliability.
This paper revisits the classical concept on N-version programming in the setting of contemporary AI coding agents. Revisiting the seminal Knight-Leveson experiment, we study whether diversity across agent systems, models, and implementation languages creates diverse failure modes. Using the Knight-Leveson's, Launch Interceptor Program Specification, we evaluate 48 agent-generated implementations on a shared oracle and a campaign of 1,000,000 randomized test inputs. The results show substantial common-mode failure, along the findings of Knight-Leveson. Further analysis that many of those co-occuring failures can be traced to where is specification is particularly hard or ambiguous. We also demonstrate that diversity from coding agents provides practical benefit: across majority voting three-version units, the mean failure count drops from 387.44 for single versions to 130.99 for triples, and 11,844 N-version units exhibit zero observed failures. Our original results is the strongest evidence to date that N-Version Programming with coding agents is a useful engineering strategy.
Mila