Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors introduce the Structured Output Benchmark (SOB), a multi-source benchmark to evaluate structured output quality in LLMs across text, image, and audio modalities. By providing text-normalized context to the models, the benchmark isolates structured-output capability from raw vision or speech-processing quality. Evaluation of 21 models reveals near-perfect schema compliance but significantly lower value accuracy, especially in audio and image domains, highlighting the challenge of accurate information extraction from diverse sources.
LLMs can nail the JSON schema, but still struggle to extract the right values from images and audio, with accuracy plummeting to 23.7% for audio data.
Large Language Models are increasingly being deployed to extract structured data from unstructured and semi-structured sources: parsing invoices, medical records, and converting PDF documents to database entries. Yet existing benchmarks for structured output generation either focus on schema compliance alone, or evaluate value correctness within a single source domain. We introduce SOB (The Structured Output Benchmark), a multi-source benchmark spanning three source modalities: native text, images, and audio conversations. All models receive a text-normalized representation of their context regardless of source modality; this deliberate design isolates structured-output capability from raw vision or speech-processing quality, ensuring a fair, source-agnostic comparison. Our benchmark comprises 5,000 text evaluation records derived from multi-hop QA drawn from a 25,091-record full corpus, 209 image records from OCR-processed PDFs across seven document types including multi-column layouts, dense tables, scanned historical documents, small-print text, and mathematical typesetting, and 115 audio records from the AMI corpus. Each record pairs a natural-language question with a JSON schema that the model must follow and a ground-truth answer verified against the source context. We evaluate 21 frontier and open-weight models across three source domains and seven metrics. Our results reveal a consistent pattern: models achieve near-perfect schema compliance, yet the best Value Accuracy, measured by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and 23.7% on audio, where longer context makes extraction substantially harder. We release the dataset, evaluation pipeline, and all related code.