arXiv paper reporting on a production LLM-integrated, multi-market rental-search assistant whose 1,553-test automated suite passed continuously yet shipped user-facing defects. Introduces a four-seam defect classification framework based on analysis of 252 bug-fix commits.
Modern web applications combining LLM output, multi-market internationalisation, and browser-driven front-ends are particularly hard to test. Automated suites can achieve 100% pass rates while user-facing defects continue reaching production.
Studied all 252 bug-fix commits in a production rental assistant and classified each by the seam it escaped through. Identified four seams (live browser runtime, non-default market, end-to-end flow, whole-system) that component-level unit tests cannot observe. Presented a four-seam framework and practices adopted.
44% of fixes fall in the four unobservable seams. One defect shipped twice because a fix had no guard at the seam. Framework provides a simple method for teams to find which seam carries the most fixes, directly actionable for QA strategy.
Ask Bilal’s AI Clone