Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Self-Conditioned Positional HNSW (SCP-HNSW), a novel method designed to enhance chunked-document retrieval in retrieval-augmented generation (RAG) systems by addressing the issue of repeated evidence due to overlapping chunks. By appending a low-dimensional positional code to chunk embeddings and employing a two-pass query procedure, SCP-HNSW effectively reduces the retrieval of near-adjacent chunks while maintaining the integrity of the HNSW graph structure. The authors provide industrial evidence through a comprehensive quality audit of generated evidence, revealing that the majority of reviews rated 3/5 or higher, highlighting the method's practical applicability and the need for further controlled retrieval studies.
Overlap-aware retrieval can significantly enhance evidence quality in RAG systems, as shown by a 770-review audit revealing that most generated evidence meets acceptable standards.
Chunked-document retrieval is a common component of retrieval-augmented generation (RAG) systems. Documents are split into overlapping chunks, embedded, and indexed with approximate nearest-neighbor search such as hierarchical navigable small world graphs (HNSW). Overlap improves boundary coverage but induces a practical failure mode: top-k retrieval often returns near-adjacent chunks that repeat evidence and waste prompt budget. We propose Self-Conditioned Positional HNSW (SCP-HNSW), a lightweight modification that appends a low-dimensional positional code to chunk embeddings and uses a two-pass query procedure to estimate and apply a query-specific document-position prior. SCP-HNSW leaves HNSW graph construction and traversal unchanged while adding an auditable minimum-index-gap selector for final context construction. We also integrate industrial review artifacts for generated evidence quality: a 770-review text-evidence audit with 318 fully labeled reviews and a 70-case OCR audit with 350 ratings. The text audit shows that 574 of 770 projected reviews are rated 3/5, only 39 fall in the 1-2 range, and narrative reviewer detail appears much more often than structured issue flags. The OCR audit shows slice-level pass rates from 95% for clean chat screenshots to 45% for handwritten/blurry captures, with moderate to strong agreement. These results motivate overlap-aware, audit-friendly RAG retrieval and identify the remaining controlled retrieval ablations needed for causal performance claims.