Search papers, labs, and topics across Lattice.
Guardian Crawler is a novel retrieval-first framework designed for knowledge discovery and evidence-grounded summarization from noisy web data, particularly in sensitive domains. By integrating BM25 retrieval with risk-aware, embedding-augmented reranking, and constrained retrieval-augmented generation, the system achieved impressive retrieval scores, including P@10 = 1.00 and NDCG@10 = 0.94 on a synthetic corpus. While the results indicate high performance in controlled settings, the study highlights the need for further validation in real-world web environments.
Achieving perfect precision in retrieval tasks, Guardian Crawler sets a new standard for evidence-grounded summarization in noisy web contexts.
Retrieving relevant evidence from noisy web data is challenging, particularly in sensitive domains containing incomplete reports, heterogeneous language, and irrelevant content. We present Guardian Crawler, a reproducible retrieval-first testbed for controlled experiments on knowledge discovery and evidence-grounded summarization over synthetic web-like corpora. The architecture combines BM25 retrieval with risk-aware, embedding-augmented, and hybrid reranking, followed by constrained retrieval-augmented generation with explicit document citations. Experiments on a synthetic 900-document corpus and 10 queries produced the highest descriptive retrieval scores under risk-based reranking, with P@10 = 1.00 and NDCG@10 = 0.94, compared with 0.94 and 0.81 for BM25. The best hybrid and BM25+Semantic configurations reached NDCG@10 values of 0.94 and 0.88, respectively. All 41 evaluable generated bullets passed the lexical coverage threshold; an automated LLM judge classified 36 as supported, one as partially supported, and four as unsupported. These results demonstrate the feasibility of Guardian Crawler as a controlled testbed but do not establish statistical superiority, human-validated faithfulness, or transfer to live-web investigative environments.
