Search papers, labs, and topics across Lattice.
This paper introduces a multi-agent framework combining deep learning (Detectron2) with vision-language models (VLMs) to improve the accuracy and robustness of colony-forming unit (CFU) detection in pharmaceutical manufacturing quality control. The framework uses a VLM agent to pre-classify plate validity, then employs both DL and VLM agents for independent colony counting, routing discordant predictions for expert review. Integrating VLMs increased automation from 50% to 85% of samples, significantly reducing human verification and improving operational efficiency.
Achieve 85% automation in pharmaceutical quality control by combining deep learning with vision-language models, surpassing the limitations of standalone DL approaches on outlier samples.
Colony-forming unit (CFU) detection is critical in pharmaceutical manufacturing, serving as a key component of Environmental Monitoring programs and ensuring compliance with stringent quality standards. Manual counting is labor-intensive and error-prone, while deep learning (DL) approaches, though accurate, remain vulnerable to sample quality variations and artifacts. Building on our earlier CNN-based framework (Beznik et al., 2020), we evaluated YOLOv5, YOLOv7, and YOLOv8 for CFU detection; however, these achieved only 97.08 percent accuracy, insufficient for pharmaceutical-grade requirements. A custom Detectron2 model trained on GSK's dataset of over 50,000 Petri dish images achieved 99 percent detection rate with 2 percent false positives and 0.6 percent false negatives. Despite high validation accuracy, Detectron2 performance degrades on outlier cases including contaminated plates, plastic artifacts, or poor optical clarity. To address this, we developed a multi-agent framework combining DL with vision-language models (VLMs). The VLM agent first classifies plates as valid or invalid. For valid samples, both DL and VLM agents independently estimate colony counts. When predictions align within 5 percent, results are automatically recorded in Postgres and SAP; otherwise, samples are routed for expert review. Expert feedback enables continuous retraining and self-improvement. Initial DL-based automation reduced human verification by 50 percent across vaccine manufacturing sites. With VLM integration, this increased to 85 percent, delivering significant operational savings. The proposed system provides a scalable, auditable, and regulation-ready solution for microbiological quality control, advancing automation in biopharmaceutical production.
