Search papers, labs, and topics across Lattice.
This study enhances the roboreg framework for real-time, marker-free pose estimation of surgical robots by implementing sequential optimization and CUDA stream parallelization, achieving localization at 30 fps for 1080p video. The method demonstrates near-1 cm accuracy, significantly outperforming existing approaches, particularly in occlusion scenarios, with a 63% improvement over FoundationPose. These advancements not only improve tracking reliability in cluttered surgical environments but also facilitate faster setup and safer multi-robot interactions.
Real-time, marker-free tracking of surgical robots now achieves near-1 cm accuracy at 30 fps, outpacing traditional methods even in occluded environments.
PURPOSE We evaluate stereo-differentiable rendering-based pose estimation for marker-free real-time surgical robots tracking, mitigating occlusion-prone marker-based tracking in cluttered surgical environments, potentially improving safety, reducing setup times, and enabling intelligent multi-robot interaction. METHODS This work extends the differentiable rendering-based markerless robot pose estimation framework roboreg for online real-time dynamic tracking in two ways. (i) Sequential optimisation propagates pose estimates across consecutive frames, with motion-adaptive hyperparameter tuning balancing convergence and precision during estimation. (ii) Integrate CUDA stream parallelisation for segmentation and the optimisation steps and combines it with CUDA-graph accelerated segmentation. We collect 38 displacement video sequence datasets with unobstructed robot and 5 occluded-robot dataset with static start/end ground-truth pose calibrations and dynamic marker-based reference tracking in between for accuracy evaluation under different scenarios. RESULTS Real-time localisation at 30 fps for 1080p video sequence is achieved, accelerating from 14 fps in the vanilla roboreg, thereby matching the camera frame rate. Near-1 cm accuracy is demonstrated, with 1.7 cm translational and 0. 6 ∘ rotational error against static ground-truth pose calibration; and with 1.2 cm average 3D error across 27,460 frames against a marker-based reference standard (1.53 cm in over 1242 frames in occlusion evaluation). Our method outperforms FoundationPose by 11% (63% in occlusion dataset) in dynamic estimation and 250% in static estimation, while achieving 6 × faster inference. CONCLUSIONS We demonstrate real-time high-resolution marker-free tracking of surgical robots through stereo-differentiable rendering. Localisation accuracy performed on par with marker-based approaches and improved upon foundational baselines.
