Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
QuanReview addresses the brittleness and auditability challenges of blending LLM and human structured span extractions through an open-source, character-level alignment and adjudication pipeline. Auditing dense, multi-field span annotations (such as quantities, units, and modifiers) is traditionally cost-prohibitive, risking silent error propagation when language models are introduced into annotation loops. Evaluated on a 4,457-record humanitarian extraction benchmark, the system fully auto-merged 8% of documents and applied automated policy resolutions to an additional 1,513 records, focusing human review on just 5.4 candidate conflicts per document.
Verifying LLM span extractions no longer requires full manual re-annotation: character-level policy reconciliation isolates genuine disagreements, slashing human review to just 5.4 focused conflicts per document.
Structured span annotations, such as quantities with their units, uncertainty modifiers, and event classes, are expensive to create and hard to keep trustworthy once language models enter the loop. We present QuanReview, an open-source system for auditing and correcting such annotation layers. QuanReview aligns two annotation streams over the same documents at character level, resolves unambiguous cases by an explicit and logged policy, and routes candidate conflicts to a browser-based adjudication interface where reviewers accept either side, build field-level hybrids, or flag items for re-annotation. A campaign manager assigns documents to multiple annotators with configurable redundancy, computes agreement at document and span level, auto-merges unanimous documents, and exports the corrected layer in the original file format, so that it can replace the original annotation files directly. Applied to a 4,457-record humanitarian benchmark and an LLM extraction stream, the system fully auto-merged 8% of documents, applied automatic policy decisions to a further 1,513 records, and concentrated human attention on 3,131 candidate conflicts, a mean of 5.4 per reviewed document.