Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces a semantic-to-physical framework for microscopic robots that maps language instructions to constrained geometric operators, leveraging frozen open-vocabulary perception and integrating focal-plane trajectories via confidence weighting and dynamic programming. This approach achieves low pixel RMSE (6.30-6.59) on unseen parts and configurations, significantly outperforming part-specific U-Net training in terms of labeling effort and inference time. The trajectory-space integration method demonstrably reduces error and improves task completion in physical robot experiments.
Microscopic robots can now execute complex tasks from natural language instructions with unprecedented accuracy, achieving 6.30-pixel RMSE using zero new labels and outperforming specialized models by over 50%.
Microscopic robots require accurate task geometry despite changes in language, parts, and focus. We present a semantic-to-physical framework that maps instructions to constrained geometric operators, reuses frozen open-vocabulary perception, and integrates locally reliable focal-plane trajectories by confidence weighting and dynamic programming. Calibrated multi-view geometry connects 2-D paths to physical execution. Prompt, unseen-part, and geometry reconfiguration tests yield 6.30-6.59-pixel RMSE. Relative to part-specific U-Net training with 20-100 labels, the proposed zero-new-label configuration takes 15 rather than 72-165 min. Across nine part-illumination conditions, trajectory-space integration reduces RMSE from 14.41 to 6.28 pixels (56.4%) and P95 error from 20.07 to 8.13 pixels (59.5%) compared with image-first multi-focus fusion. An ablation isolates the roles of confidence and path-wise selection. In representative robot experiments, target-region coverage improves from 83.5% to 92.9%. Dispensing provides a measurable physical trace, not a task-specific limitation of the method.
CMU ML