Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a semi-automated pipeline for extracting Subject-Predicate-Object triplets from financial reports using LLMs, addressing the lack of ground truth data by employing ontology-driven proxy metrics like Ontology Conformance and Faithfulness. The authors compare a manually engineered ontology with a document-specific, automatically induced ontology, finding that the latter achieves 100% schema conformance and eliminates ontology drift. They also propose a hybrid verification strategy combining regex matching and LLM-as-a-judge to reduce subject hallucination rates, and identify asymmetries in subject/object hallucinations.
Financial reports are a goldmine of structured knowledge, and this pipeline unlocks it by using LLMs to extract relationships with high fidelity, even without labeled data.
Corporate financial reports are a valuable source of structured knowledge for Knowledge Graph construction, but the lack of annotated ground truth in this domain makes evaluation difficult. We present a semi-automated pipeline for Subject-Predicate-Object triplet extraction that uses ontology-driven proxy metrics, specifically Ontology Conformance and Faithfulness, instead of ground-truth-based evaluation. We compare a static, manually engineered ontology against a fully automated, document-specific ontology induction approach across different LLMs and two corporate annual reports. The automatically induced ontology achieves 100% schema conformance in all configurations, eliminating the ontology drift observed with the manual approach. We also propose a hybrid verification strategy that combines regex matching with an LLM-as-a-judge check, reducing apparent subject hallucination rates from 65.2% to 1.6% by filtering false positives caused by coreference resolution. Finally, we identify a systematic asymmetry between subject and object hallucinations, which we attribute to passive constructions and omitted agents in financial prose.