Search papers, labs, and topics across Lattice.
This paper details MongoDB's development of a specification-based testing approach for its client libraries across twelve programming languages, addressing the challenge of ensuring consistent behavior in millions of lines of code. By transitioning to a Unified Test Format using YAML, the team was able to eliminate over 22,000 lines of test code and significantly reduce nonconformance bugs by up to 86% in libraries that adopted this method. The findings highlight the importance of declarative test design and the complexities of maintaining consistency across diverse implementations.
Achieving an 86% reduction in nonconformance bugs by standardizing tests across a dozen programming languages reveals the power of a unified testing approach.
MongoDB maintains client libraries in a dozen programming languages, used by tens of thousands of organizations and millions of developers. Most are implemented natively rather than as wrappers around a shared core. Ensuring consistent behavior across these libraries, comprising millions of lines of code, is hard but essential. Over eleven years, we developed a specification-based testing approach: tests are written once in YAML and executed by language-specific interpreters for each library. We describe the evolution from many ad-hoc formats to a Unified Test Format, which allowed us to delete over 22,000 lines of test code. The rate of nonconformance bugs fell up to 86% in drivers that adopted YAML tests (though results varied). We report lessons learned about declarative test design, test architecture, schema evolution, and the limits of unification.
