Search papers, labs, and topics across Lattice.
This study investigates inference-time pattern-memory gating within a clinical NLP pipeline that uses a generator-verifier architecture to enhance extraction accuracy from patient narratives. Key findings reveal that traditional learning methods for filtering based on verifier rejections are ineffective at scale, while a simpler approach utilizing a fixed clinical ontology successfully identified violations. Additionally, the research highlights that the effectiveness of filtering mechanisms hinges on their alignment with the verifier's evaluation criteria, underscoring the importance of thoughtful design in generator-verifier systems.
The most sophisticated memory designs can fail silently in production, revealing that filtering effectiveness is determined by how well it aligns with the verifier's decision-making criteria.
We study inference-time pattern-memory gating in a production-scale clinical natural language processing (NLP) pipeline. The pipeline pairs a generator (Llama-3.3 70B) proposing extractions with a verifier (MMed-Llama-3.1 70B) accepting or rejecting them, over 167,034 PMC-Patients narratives, and adds a lightweight memory that learns at deployment which extractions to filter, so the verifier need not re-examine candidates already seen to fail. We report four findings. First, learning filtering rules directly from the verifier's rejections failed at full scale: the relation-extraction filter stayed empty despite 785,797 logged rejections, because they were spread too thinly across too many distinct forms to accumulate. Second, a simpler rule using a fixed clinical ontology produced the same filtering without the verifier, capturing 49,734 ontology-violating relations on a held-out 5,000-patient set. Third, of five versions of the question-answering filter, four failed for distinct, instructive reasons; the fifth succeeded by checking whether a patient's extracted entities support the question asked, and where it applies was 1.84 times likelier to flag an answer the verifier would reject than one it would accept. Fourth, one pattern held across all five: a filter is selective only when it tests the same evidence the verifier weighs, not when it imitates the verifier's output. Together these give a transferable result for any generator-verifier pipeline: the most natural memory design can fail silently at scale, and whether a pre-generation gate is selective is decided before any engineering effort, by whether its signal probes the question the verifier itself answers. Throughout, the system flags suspect extractions rather than deleting them, so every decision stays visible for clinical review. All code and test artefacts are released openly.
