Search papers, labs, and topics across Lattice.
This study introduces a YOLO- and CLIP-based vision-language framework to classify mosquito flight frames, distinguishing between uninfected and DENV2-infected mosquitoes. By isolating mosquito regions and aligning visual features with biologically relevant textual prompts, the model achieved an impressive 98.54% accuracy and 99.91% sensitivity at the frame level. The findings highlight the effectiveness of multimodal models in analyzing complex biological behaviors from video data, demonstrating their potential in disease diagnosis.
Achieving over 98% accuracy in classifying infected mosquitoes from video frames showcases the power of combining vision and language models for biological analysis.
Detecting infection-related behavioral changes in mosquitoes from video data is challenging because mosquitoes are small, move rapidly and irregularly, and are affected by environmental factors such as background, lighting, and shadows, which can make reliable feature extraction difficult. In this study, a YOLO- and Contrastive Language-Image Pre-training (CLIP)-based vision-language framework is proposed to classify mosquito flight frames of uninfected and Dengue virus serotype 2 (DENV2)-infected mosquitoes. First, YOLO is used to isolate mosquito regions from the background. Then, visual features extracted from video frames are aligned with biologically meaningful textual prompts in a shared embedding space. The multimodal model was fine-tuned using supervised bidirectional contrastive learning and evaluated through frame-level image-text similarity-based classification. The results show that the proposed method achieved 98.54% accuracy and 99.91% sensitivity at the frame level. After temporal aggregation of frame-level information, the model achieved complete video-level performance. The ablation results showed that fine-tuning and CLIP-based representations were essential for this domain, while the textual branch provided semantic image-text alignment rather than an accuracy advantage over the vision-only model. These findings suggest that vision-language models can provide a useful framework for analyzing infection-related biological behaviors from video data.
