Search papers, labs, and topics across Lattice.
This paper introduces Pre-Flight, a benchmark designed to evaluate large language models (LLMs) on aviation operational knowledge through 300 multiple-choice questions based on international standards and regulations. The study highlights a significant performance gap, with the best-performing model achieving only 82.7% accuracy compared to an expert reference of 95%, indicating that current LLMs struggle to meet the safety and correctness standards required in aviation. By providing an open-source dataset and evaluation framework, the authors emphasize the need for domain-specific assessments to ensure responsible AI deployment in aviation contexts.
Despite advancements, even the best LLMs lag far behind expert-level performance in aviation knowledge, achieving only 82.7% accuracy on critical operational questions.
Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants. General purpose benchmarks do not measure whether a model reasons safely and correctly about aviation specific operational knowledge, and the high stakes, regulated nature of the domain makes that gap consequential. We present Pre-Flight, an open source benchmark of 300 multiple choice questions drawn from international standards and airport ground operations material, covering international airport ground operations, ICAO and US FAA regulations, aviation general knowledge and complex operational scenarios. Questions were authored and reviewed by practitioners with experience in air traffic management, ground operations and commercial flying. We evaluate a range of contemporary commercial and open weight models using the Inspect evaluation framework, scoring by accuracy under a standard multiple choice protocol, and we maintain the leaderboard on a rolling basis as new models are released. Against an informal expert reference of around 95%, obtained from a low sample quiz of aviation professionals at a conference, even the strongest model evaluated (released in 2026) reaches 82.7%, having improved only gradually from roughly 75% in early 2025. A substantial and persistent gap below expert level reliability therefore remains. We release the dataset, the evaluation harness and the results, and the benchmark is available within the community evaluations package distributed with inspect_evals. We argue that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.
