Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
HeteroPROMPT is a novel framework that enhances heterogeneous collaborative perception by aligning diverse sensor features into a unified ego-centric space without retraining existing models, thereby preserving privacy. This approach utilizes modular prompts and lightweight tuning to achieve real-time performance while keeping agent encoders frozen, addressing the challenges posed by feature-space shifts in collaborative environments. Experiments demonstrate that HeteroPROMPT significantly outperforms existing methods in Average Precision with vastly fewer trainable parameters, making it a scalable solution for real-world deployments.
HeteroPROMPT achieves over 99.99% accuracy in modality classification while dramatically reducing the number of trainable parameters needed for effective heterogeneous collaborative perception.
Collaborative Perception (CP) improves autonomous systems'awareness of their surroundings by sharing sensor data, intermediate features, and detection results. In real-world deployments, however, collaborating vehicles often use heterogeneous sensors, perception models, datasets, and training domains, creating feature-space shifts that degrade downstream fusion and detection. Existing approaches typically retrain fusion and detection components or introduce modality-specific feature interpreters. These methods scale poorly to newly joining agents and often require access to proprietary metadata, raising privacy concerns. We propose HeteroPROMPT, a real-time and privacy-preserving framework for heterogeneous collaborative perception. HeteroPROMPT rapidly aligns each heterogeneous agent's features with an ego-centric unified feature space through modular prompts and lightweight learning-based tuning, while keeping agent encoders and the collaborative fusion and detection stacks frozen. Its visual prompt-based training and inference modulate Bird's Eye View (BEV) features across channels and spatial locations with low computational overhead. For metadata-free deployment, an autoencoder learns a compact unified representation and extracts modality cues from shared features, enabling real-time modality classification and routing to the appropriate HeteroPROMPT modules without exposing proprietary agent information. Experiments on the OPV2V-H and V2XSet datasets show that HeteroPROMPT improves Average Precision over state-of-the-art heterogeneous CP methods while using orders of magnitude fewer trainable parameters. This offers a scalable and practical CP solution. The proposed modality classifier also predicts the joining agent's modality from compact features with greater than 99.99 percent accuracy during deployment. Code will be available at https://github.com/arminmaleki007/HeteroPROMPT.