Search papers, labs, and topics across Lattice.
This paper introduces an innovative on-site defect recognition method for UAV inspections by leveraging spatial reasoning learning with multimodal large models. By employing a combination of Variational Autoencoders and a Swin Transformer-based visual encoder, the approach effectively generates and identifies defect features in challenging environments characterized by small scale and complex backgrounds. Experimental results indicate that the method achieves high precision and efficiency in real-time edge inference, significantly enhancing the capability of drone inspections in power system monitoring.
Real-time UAV inspections can now achieve high-precision defect detection even in complex environments, thanks to a novel spatial reasoning approach that integrates multimodal learning.
Power drone inspections have become a crucial method for monitoring the status of transmission lines and substation equipment. However, in real-world scenarios, defect targets typically exhibit features such as small scale, weak texture, and complex backgrounds with strong interference. The traditional model struggles to meet the demand for rapid on-site closed-loop resolution. To address these challenges, this paper proposes an on-site defect identification method for drone inspection scenarios based on spatial reasoning learning from multimodal image-text large models. First, we develop a defect image generation method using Variational Autoencoders (VAE) and conditional score matching diffusion. By embedding semantic information into the feature layer of the U-Net decoder via the multi-layer Spatial Adaptive Normalization and Decomposition (SPADE) operator, we precisely control the generation location and morphology of defect features. Second, at the model level, a visual encoder based on Swin Transformer is constructed. A hierarchical window attention mechanism extracts multi-scale defect and scene topological features. A text encoder is built using the Next Token Prediction pre-training method and a large language model, combined with domain-specific fine-tuning using power system terminology databases and equipment ledger knowledge. At the cross-modal fusion layer, a contrastive learning mechanism aligns defect images and textual descriptions within a unified vector space. For spatial reasoning learning, we construct visually-linguistically intertwined regional inference samples. Region-level instruction fine-tuning drives the model to perform dynamic region cropping and multi-step reasoning analysis. Regarding on-site deployment, we combine CPU-NPU heterogeneous co-acceleration with memory optimization strategies. Through dynamic task allocation and data prefetching techniques, we achieve low-power, low-latency real-time edge inference. Experimental results demonstrate that our method delivers high-precision and high-efficiency defect detection in power inspection scenarios, effectively addressing complex backgrounds and resource-constrained field environments.
