Search papers, labs, and topics across Lattice.
FalconTrack introduces a novel framework for vision-based aerial tracking that automates the labeling process using a photorealistic simulator, generating approximately 10,000 labeled images in under 20 minutes. By integrating multi-head perception with physics-aware tracking, the system achieves impressive zero-shot sim-to-real transfer, demonstrating 96-100% class accuracy across diverse geometries and environments. In practical applications, FalconTrack's onboard system operates at 25 Hz, achieving 100% success in tracking tasks, significantly outperforming traditional methods in challenging scenarios.
FalconTrack automates the generation of photorealistic labeled data, achieving 100% success in real-world tracking while traditional methods falter under pressure.
Vision-based aerial tracking is critical in GPS-denied environments. Reliable perception for tracking depends on large-scale labeled data, yet most photorealistic datasets rely on heavy manual annotation and are time-consuming to produce. We present FalconTrack, a unified perception-and-tracking framework that (i) leverages a photorealistic editable simulator for automated label generation and (ii) combines multi-head perception with physics-aware tracking for zero-shot sim-to-real transfer. FalconTrack provides an automated labeling pipeline in a Gaussian Splatting simulator that isolates target Gaussians from short object videos and composites them with randomized backgrounds to generate RGB, mask, class, and 6-DoF pose labels, producing about 10k labeled images in under 20 minutes. Using this dataset, we train a multi-head perception module with staged learning and reprojection consistency, and fuse its outputs with class-conditioned dynamics priors in an EKF for tracking. Our perception model outperforms two baselines and reaches 96-100% class accuracy in zero-shot sim-to-real transfer on three geometrically diverse objects and two environments, while maintaining consistent performance in unseen simulated and real scenes. In real hardware closed-loop visual tracking, the onboard system runs at about 25 Hz and achieves 100% success in sim-to-real F1-tenth and gate tracking in five trajectories across two environments, while a mask-centered vision baseline drops to 60% success on F1-tenth during fast out-of-view scenarios.
