Search papers, labs, and topics across Lattice.
IMPASTO, a robotic oil-painting system, learns to reproduce paintings from image sequences by integrating learned pixel dynamics models with model-based planning. The system uses dynamics models to predict canvas updates from parameterized stroke actions, and a receding-horizon MPC optimizer plans trajectories and forces executed by a force-sensitive 7-DoF robot arm. IMPASTO learns entirely from robot self-play and outperforms baselines in reproduction accuracy when approximating human artists' single-stroke datasets and multi-stroke artworks.
Robots can now learn to reproduce oil paintings with impressive accuracy through self-play and model-based planning, even without human demonstrations or high-fidelity simulators.
Robotic reproduction of oil paintings using soft brushes and pigments requires force-sensitive control of deformable tools, prediction of brushstroke effects, and multi-step stroke planning, often without human step-by-step demonstrations or faithful simulators. Given only a sequence of target oil painting images, can a robot infer and execute the stroke trajectories, forces, and colors needed to reproduce it? We present IMPASTO, a robotic oil-painting system that integrates learned pixel dynamics models with model-based planning. The dynamics models predict canvas updates from image observations and parameterized stroke actions; a receding-horizon model predictive control optimizer then plans trajectories and forces, while a force-sensitive controller executes strokes on a 7-DoF robot arm. IMPASTO integrates low-level force control, learned dynamics models, and high-level closed-loop planning, learns solely from robot self-play, and approximates human artists'single-stroke datasets and multi-stroke artworks, outperforming baselines in reproduction accuracy. Project website: https://impasto-robopainting.github.io/
