Search papers, labs, and topics across Lattice.
This paper introduces a schema-based framework that employs generative AI for the extraction of complex, hierarchical information from unstructured text, followed by an automated semantic evaluation against a gold standard. The approach allows for zero-shot extraction of nested attributes with variable cardinality, achieving an impressive F1 score of over 90% for 12 out of 14 attributes from health technology assessment documents. Notably, the extraction process is approximately 30 times faster than traditional human expert methods, showcasing the framework's efficiency and adaptability across various contexts and languages.
Achieving over 90% accuracy in extracting structured information from unstructured documents, this framework outpaces human experts by a factor of 30 in speed.
We present a schema-based framework for extracting complex, structured information from unstructured text documents using generative AI, followed by automated semantic evaluation of the extracted information against a gold standard. The schema, serving as an information model encoding domain knowledge, provides a unified, systematic, and consistent framework for extraction of hierarchical, nested information, with attributes of variable cardinality, and subsequent evaluation of the results. Information extraction from a document is performed in a single call to the model, in zero-shot mode. In the evaluation step, we introduce a path-based semantic matching algorithm to align the nested, variable-cardinality attributes in the extracted results with those in the gold standard. We use generative AI for semantic comparison of the extracted and gold standard values of an attribute, and introduce a rubric to classify the result of the comparison, according to domain-specific considerations, as an exact, semantic, useful, or non-match. We were able to extract 12 out of 14 attributes with an F1 score of $>$90\% from documents published by the health technology assessment organisation NICE, using the generative AI model Claude Opus 3. The time needed to extract the attributes from a document was $\sim$30 times lower than the time taken by a human domain expert. We further demonstrate generalisability of this framework across different generative AI models and transferability across different HTA organisations and languages.
