Search papers, labs, and topics across Lattice.
The authors introduce a reusable evaluation pipeline for AI meeting summaries, separating orchestration from task-specific semantics across stages like source intake, scoring, and reporting. They benchmarked the pipeline on 340 meeting-model pairs across three datasets (city council, private data, White House press briefings) and three models (gpt-4.1-mini, gpt-5-mini, and gpt-5.1). Results show that gpt-4.1-mini achieved the highest mean accuracy (0.583), while gpt-5.1 led in completeness (0.886) and coverage (0.942), demonstrating the pipeline's ability to identify nuanced performance differences.
GPT-4.1-mini wins on accuracy for meeting summarization, but GPT-5.1 crushes it on completeness and coverage, revealing that the best model depends on the specific metric you care about.
We present a reusable evaluation pipeline for generative AI applications, instantiated for AI meeting summaries and released with a public artifact package derived from a Dataset Pipeline. The system separates reusable orchestration from task-specific semantics across five stages: source intake, structured reference construction, candidate generation, structured scoring, and reporting. Unlike standalone claim scorers, it treats both ground truth and evaluator outputs as typed, persisted artifacts, enabling aggregation, issue analysis, and statistical testing. We benchmark the offline loop on a typed dataset of 114 meetings spanning city_council, private_data, and whitehouse_press_briefings, producing 340 meeting-model pairs and 680 judge runs across gpt-4.1-mini, gpt-5-mini, and gpt-5.1. Under this protocol, gpt-4.1-mini achieves the highest mean accuracy (0.583), while gpt-5.1 leads in completeness (0.886) and coverage (0.942). Paired sign tests with Holm correction show no significant accuracy winner but confirm significant retention gains for gpt-5.1. A typed DeepEval contrastive baseline preserves retention ordering but reports higher holistic accuracy, suggesting that reference-based scoring may overlook unsupported-specifics errors captured by claim-grounded evaluation. Typed analysis identifies whitehouse_press_briefings as an accuracy-challenging domain with frequent unsupported specifics. A deployment follow-up shows gpt-5.4 outperforming gpt-4.1 across all metrics, with statistically robust gains on retention metrics under the same protocol. The system benchmarks the offline loop and documents, but does not quantitatively evaluate, the online feedback-to-evaluation path.
