Search papers, labs, and topics across Lattice.
This paper introduces VNVSpec, an open-source framework designed to bridge the gap between high-level requirements and low-level tests in software development, particularly for AI-enabled and cyber-physical systems. By making verification and validation (V&V) specifications machine-readable and executable, VNVSpec allows users to directly state high-level requirements, decompose them into module-level requirements, and link these to test results through a traceability graph. The framework was evaluated through continuous integration against its own specifications, successfully verifying 36 requirements with 449 tests, demonstrating scalability to handle up to 10,000 requirements efficiently.
VNVSpec transforms high-level user requirements into actionable, machine-readable specifications that can be directly linked to test results, addressing a critical gap in software verification.
Modern software teams have mature tools for low-level testing, such as pytest, JUnit, and Jest, which make it inexpensive to write unit tests and run them on every commit. Systems engineering, in parallel, has developed rigorous principles for design verification and validation (V&V), which has worked very well across engineering discipline to align user expecations and requirements with developers'deliverables. In practice, however, the two rarely connect, and the link between users'high-level requirements and the low-level tests that machines actually run is maintained by hand, if at all. This gap is increasingly costly for AI-enabled and cyber-physical systems, for which regulators now ask for traceable evidence that high-level requirements are met, while raw test results provide little of the structure such evidence requires. We introduce VNVSpec, an open-source framework that makes V&V specifications machine-readable and executable. With this framework, users state high-level requirements directly or import them from catalogs derived from published standards. Then, the framework checks requirement quality, supports decomposition into module-level requirements with explicit metrics and acceptance criteria, links these requirements to test results through a traceability graph, and compiles the collected evidence into verdicts and audit-ready reports. We evaluate the framework by self-application, in which it is continuously assessed in CI against its own specification of 36 requirements verified by 449 tests, completed within limited time which scales linearly and thus can handle up to 10,000 requirements. We also discuss how the framework extends to testing black-box AI models and AI coding agents. The framework, its full test suite, the catalogs, and the benchmark scripts are available at https://github.com/ai-vnv/vnvspec.
