Search papers, labs, and topics across Lattice.
This paper introduces CorporateBench (CB), a large-scale Q&A benchmarking framework designed to evaluate LLMs on enterprise-scale document collections. By utilizing a human-validated multi-task approach and a temporally evolving knowledge base, CB assesses LLMs' performance in information extraction and knowledge base querying across diverse corporate scenarios. The findings indicate that LLMs struggle significantly with realistic input sizes, highlighting the need for improved models in corporate communication reasoning.
LLMs falter in corporate Q&A tasks, with performance dropping sharply as document complexity increases.
LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.
