Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the human-aware safety gap in UAV navigation by developing an onboard pipeline that translates visual observations into context-dependent safety constraints. The system leverages a BLIP encoder for scene understanding and a lightweight LLM (Qwen-2.5-0.5B) for category-specific risk inference, employing retrieval-augmented generation and grammar-constrained decoding for planner-compatible JSON output. Experimental results demonstrate a significant reduction in collision events and improved success rates in complex urban navigation scenarios compared to purely geometric planners.
Lightweight LLMs can infer category-specific risk for real-time UAV navigation, enabling a 3x reduction in collisions and boosting landing success from 34% to 96% in urban environments.
Aerial robots operating near people must treat a pedestrian, a parked car, and a wall differently, yet conventional perception--planning stacks reduce all of them to occupied geometry---a \emph{human-aware safety gap} in which category-specific clearance is unavailable to the planner and temporary occlusions remove pedestrian-specific constraints from its map. We present a modular, fully onboard pipeline that closes this gap by turning visual observations into context-dependent safety constraints for real-time UAV navigation. A shared BLIP encoder drives scene captioning and text-conditioned segmentation, which is fused with depth to localize obstacles in 3D. Obstacle-specific risk is inferred by a lightweight LLM (Qwen-2.5-0.5B) stabilized with retrieval-augmented generation and grammar-constrained decoding that restricts its output to a planner-compatible JSON schema. The LLM reasons once per object category rather than per frame: its verdicts are cached per class, so previously seen categories are resolved without inference and new ones trigger a fresh retrieval-grounded query; detections are additionally given persistent identities and held in the map across occlusions. The perception pipeline runs at 10\,Hz on a Jetson Orin NX, while fallback LLM queries take approximately 0.8\,s and remain off the control-critical path. In Gazebo, low-altitude urban navigation among pedestrians and vehicles shows a roughly $3\times$ reduction in collision events over a geometric planner, and semantics-aware landing among moving actors raises success from 34\% to 96\%. The 224-pixel encoder input limits reliable small-object perception at range; conservative pedestrian margins and short-term persistence mitigate this limitation during navigation.