Search papers, labs, and topics across Lattice.
This paper introduces FaLCon, a novel retrieval framework for text-based person anomaly search that effectively bridges the gap between synthetic training data and real-world pedestrian image retrieval. By employing a coarse-to-fine approach that integrates global semantic matching with fine-grained verification, the method enhances retrieval accuracy while managing computational costs. The framework achieves a remarkable 95.41% mAP@10 on the PAB benchmark, showcasing its ability to refine candidate pools through advanced multimodal techniques and consensus mechanisms.
Achieving a staggering 95.41% mAP@10, FaLCon revolutionizes text-based person anomaly search by combining global semantic matching with fine-grained verification to tackle the Sim2Real challenge.
Text-based person anomaly search requires retrieving real-world pedestrian images from detailed natural-language descriptions using models trained primarily on synthetic data. This Sim2Real setting is particularly challenging because visually similar candidates may differ only in subtle actions, object interactions, or appearance attributes, while applying multimodal large language models to the entire gallery is computationally expensive. We propose an anchor-constrained coarse-to-fine retrieval framework that combines global semantic matching with fine-grained verification. First, each query is represented by its original caption, a structured concatenation, and several semantic facets. Heterogeneous vision-language retrievers are then integrated through robust per-query score calibration and soft claim-aware fusion. Full and concatenated captions serve as anchors to preserve candidate recall, whereas appearance, action, and object facets provide bounded corrective evidence. The resulting candidate pool is further refined by a discriminative Qwen3 reranker and two complementary semantic verification modules based on anomaly-aware cloze completion and multi-agent evidence reasoning. Finally, an uncertainty-gated consensus module adaptively reweights the three experts on ambiguous queries. Experiments on the PAB benchmark show that the proposed soft claim-aware retrieval achieves 86.44% mAP@10, substantially outperforming individual retrieval backbones. The complete framework further improves performance to 95.41% mAP@10, 94.44% R@1, and 99.09% R@5. These results demonstrate that preserving strong global retrieval while restricting expensive semantic reasoning to a small candidate pool is effective for fine-grained Sim2Real person anomaly search. Our code will be available on Github.
