Search papers, labs, and topics across Lattice.
This paper introduces BiliVLA, a scene-aware Vision-Language-Action framework designed for autonomous biliary endoscopic navigation, addressing the challenges posed by anatomical variability and visual artifacts in ERCP procedures. By formulating navigation as an instruction-conditioned visuomotor learning problem, BiliVLA effectively predicts target categories, bounding boxes, and motor commands while incorporating scene-aware and safety-aware supervision. The two-stage training paradigm, combining supervised fine-tuning and Group Relative Policy Optimization, leads to a remarkable action precision of 91.96% and an overall success rate of 84.85% in real-world experiments, demonstrating significant improvements in perception-action alignment and decision consistency.
Action precision in autonomous biliary navigation reaches 91.96% by integrating scene-aware learning with instruction conditioning.
Endoscopic retrograde cholangiopancreatography (ERCP) demands precise endoscopic navigation and stable biliary cannulation within a narrow monocular field characterized by specular reflections, partial occlusions, and frequent tissue contact. Although recent robotic systems and vision-based assistance techniques improve operator ergonomics and provide perceptual cues, their performance degrades under pronounced anatomical variability and safety-critical visual artifacts, which hinders reliable autonomy in cannulation-grade procedures. Here, we present BiliVLA, a scene-aware Vision-Language-Action (VLA) framework that formulates biliary endoscopic navigation as an instruction-conditioned visuomotor learning problem. Given an endoscopic observation and a stage-specific language instruction, BiliVLA jointly predicts the target category, a grounded bounding box, and a discrete three degrees of freedom (DoF) motor command for a continuum endoscope. The proposed framework incorporates scene-aware supervision to enhance semantic target consistency and safety-aware recovery supervision to induce conservative retreat behaviors under luminal wall contact. A key component of BiliVLA is a two-stage training paradigm that combines grounding-enhanced supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO), which significantly improves action reliability and decision consistency during closed-loop navigation. Across three ERCP subtasks, BiliVLA achieves an average action precision of 91.96\% and an overall success rate (SR) of 84.85\% in real-world phantom experiments. These results indicate that integrating semantic grounding, scene-aware learning, and reward-guided optimization improves perception-action alignment and enables robust autonomous endoscopic navigation.
