Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
This paper introduces LICOA, a novel system that treats projected light as an embodied action for mobile spatial augmented reality, enabling a robot to autonomously position and scale projections based on natural language instructions and visual observations. By integrating a differential-drive chassis, a 2DoF steerable mirror, and projection scale control, LICOA achieves room-scale repositioning and rapid local beam manipulation. The system demonstrates significant real-world performance gains (57.50% success rate) by training on a large simulated corpus and then adapting to unseen physical environments, outperforming models trained from scratch.
Mobile robots can now dynamically place and scale augmented reality projections in real-world environments based solely on language instructions, achieving over 57% success in unseen tasks.
Spatial augmented reality (SAR) augments physical environments with projected content, but most systems assume that the projector pose, target region, or motion path has been fixed or specified. We instead make projected light an embodied action: given a language instruction and current visual observations, a system must decide how to move and where and at what scale to place the projection. We introduce LICOA to realize this capability. Its differential-drive chassis provides room-scale approach and repositioning, while a steerable mirror with two degrees of freedom (2DoF) and projection scale provide rapid local control of the beam. A shared interface maps synchronized external and onboard RGB cameras with robot control state to physical optical actions across pretrained action models. Because instruction-conditioned optical action lacks dedicated training data and a common evaluation protocol, we construct a matched simulation embodiment and a corpus with 150 tasks and 7,500 rendered episodes, then evaluate models trained only on this corpus in previously unseen real-world environments. Across 480 complete closed-loop real-world execution trials (rollouts) on 12 tasks, the best adapted action model achieves a 57.50% success rate and 40.64% success weighted by path length, compared with 8.33% and 5.15% for the same architecture trained from scratch. A further 150 real-world ablation rollouts support the need for both camera views and coordinated control of the chassis and mirror. These results establish a physical platform, shared perception and optical action interface, corpus, and real-world evaluation protocol for treating projected light as an embodied action.