Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces LEDGER, a comprehensive benchmark consisting of 4,999 digitized corporate annual reports designed to facilitate the evaluation of large language models in financial contexts. By providing full documents with figures and tables, along with 31 labeled financial KPIs, the dataset enables rigorous testing across various tasks, including page-level retrieval and full KPI extraction. The authors validate LEDGER's utility through a case study that connects CEO-letter rhetoric to market reactions, highlighting the dataset's potential for advancing grounded financial retrieval and extraction capabilities.
Large language models can now be rigorously evaluated on their ability to extract and retrieve financial insights from complete corporate annual reports, not just plain-text filings.
Finance reporting is a natural proving ground for large language models, and the very-long-context capabilities of recent models across all sizes make rigorous evaluation in this domain an increasingly pressing need. Yet most public financial resources reduce the task to plain-text SEC 10-K filings paired with a handful of question-answer items. We release LEDGER (Long-context Evaluation of Documents for Grounded Extraction and Retrieval), a corpus of 4,999 digitized corporate annual reports - full documents with figures, tables, and narrative, not just regulatory filings. Each report is labeled with 31 consolidated financial KPIs to be extracted and linked to the market's reaction at the earnings date. From this data we derive three evaluation benchmarks spanning the difficulty spectrum: a pure page-level KPI retrieval task with TREC-style relevance judgments over 118,048 questions in natural language, a conversational"needle-in-a-haystack"single-value lookup, and a full KPI extraction task, both from long, numerically dense reports. We additionally provide human OCR-quality annotations with inter-annotator agreement and the complete extraction, validation, and scoring toolchain. We further demonstrate the dataset's research utility with a case study linking CEO-letter rhetoric to post-publication market impact.