Search papers, labs, and topics across Lattice.
This paper analyzes the challenges of testing modern web applications that integrate large language model outputs, internationalization, and browser-driven interfaces, revealing that despite a robust automated test suite of 1,553 cases, user-facing defects persisted. The authors examined 252 bug-fix commits and identified that 44% of defects escaped through four critical seams鈥攍ive browser runtime, non-default markets, end-to-end flows, and whole-system levels鈥攖hat traditional unit tests fail to cover. By introducing a four-seam framework and sharing their practices, the authors provide actionable insights for improving testing efficacy in complex web applications.
Nearly half of the defects in LLM-integrated web apps slip through testing seams that standard unit tests can't catch, highlighting a critical gap in software verification.
Modern web applications increasingly combine three ingredients that are hard to test: output from large language models, multi-market internationalization, and browser-driven front-ends over external data sources. We report on a production rental-search assistant whose automated suite grew to 1,553 test cases in six weeks. The suite passed continuously, yet user-facing defects continued to reach production. We studied all 252 bug-fix commits in the project and classified each by the boundary, or seam, it escaped through. About 44 percent of the fixes fall in four seams that component-level unit tests cannot observe: the live browser runtime, the non-default market, the end-to-end flow, and the whole-system level. A fix without a guard at the seam let one defect ship twice. We present the four-seam framework, the measured defect distribution, and the practices we adopted, including a simple way for a team to find the seam that carries the most fixes.
