Search papers, labs, and topics across Lattice.
This research developed a multimodal manual retriever (MMR) for the Cessna 172 Maintenance Manual (C172-MM), enhancing the retrieval of both textual and visual information to assist aircraft technicians. By evaluating retrieval performance with synthetic queries, the MMR achieved a recall of 93.37% at five results, while the subsequent multimodal retrieval augmented generation (MRAG) pipeline produced responses with 87.20% semantic similarity to ground-truth answers. The findings suggest that this approach significantly reduces the time technicians spend searching for critical maintenance information, addressing a key barrier to effective manual usage in aviation maintenance.
Technicians can now access critical maintenance information in under 12 seconds, transforming how they interact with complex manuals.
Proper use of the aircraft maintenance manual is essential for correct maintenance, providing procedures, diagrams, cautions, and specifications. However, technicians often avoid consulting it because it is difficult to navigate and time-consuming under strict schedules. Retrieval augmented generation (RAG) models have recently been introduced in aircraft maintenance, yet existing models focus solely on textual retrieval. This research therefore targeted the Cessna 172 Maintenance Manual (C172-MM), widely used in general aviation, and developed a multimodal manual retriever (MMR) capable of retrieving multimodal manual pages. Retrieval performance was evaluated using synthetic queries covering procedures, diagrams, caution/safety information, and specifications; the MMR achieved 93.37% recall@5. Beyond retrieval, a multimodal RAG (MRAG) pipeline was examined, in which retrieved pages were input to a vision-language model that generated responses to the synthetic queries, achieving 87.20% semantic similarity to ground-truth answers. Three practical feasibilities were also assessed: inference time, operational cost, and interpretability. Average retrieval time for five pages was 11.93 seconds and response generation took 4.95 seconds, at $0.0091 per query, while interpretability was validated through heatmap visualizations. These results indicate that the MRAG pipeline for the C172-MM can reduce the time technicians spend searching manuals and retrieving multimodal information.
