Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
QueryArt estimates 3D articulation parameters from a single RGB image and a 2D query point by predicting relative geometry conditioned on the query's depth, thus avoiding explicit part segmentation and its associated vulnerabilities. This query-conditioned approach, trained on mixed synthetic and real data, recovers metric parameters with a single depth measurement at the query point. QueryArt outperforms existing methods on benchmarks and achieves a 70.2% success rate in real-world robotic manipulation trials.
A single query point and its depth are sufficient to unlock accurate 3D articulation estimation from a single image, bypassing the need for explicit part segmentation.
Enabling robots to estimate the kinematic parameters of articulated objects unlocks a wide range of capabilities for interaction and manipulation. The estimation has to happen from the information the robot currently observes, often just a single RGB image of an object it has never seen before. Current single-image approaches couple articulation part segmentation with articulation estimation, making their predictions vulnerable to missed detections and incorrect part associations, and they regress metric 3D geometry that a single view fixes only up to scale. We present QueryArt, a model that estimates articulation parameters from a single RGB image, a 2D query point, and camera intrinsics. QueryArt is trained to estimate the 3D articulation geometry relative to the queried point and in units of its depth, which keeps its target identifiable from the image alone. A single depth measurement at the query point then supplies the scale and recovers the metric parameters. We train QueryArt on a curated mixture of synthetic and real-world articulation datasets. We evaluate QueryArt on several benchmarks and compare it against recent baselines. QueryArt outperforms recent baselines on most articulation metrics, including on out-of-distribution data. To demonstrate the model's capabilities in real-world settings, we evaluate QueryArt on a mobile manipulator across 57 manipulation trials spanning 16 object parts and five viewpoint classes, achieving a 70.2% success rate. We provide code and videos at: https://abwerby.github.io/queryart/