Search papers, labs, and topics across Lattice.
This paper introduces SPyCE, a novel framework for multimodal agents that enables the co-evolution of skills and policies during reinforcement learning. By distilling multimodal reasoning trajectories into a hierarchical skill library that updates in tandem with the policy, SPyCE allows agents to leverage reusable skills rather than relying solely on scalar rewards or static memory retrieval. Experiments across eight benchmarks show that SPyCE significantly outperforms existing reinforcement learning and memory-based approaches, highlighting the importance of joint skill-policy optimization for enhancing agent capabilities.
Multimodal agents can achieve superior performance by co-evolving skills and policies, rather than treating them as separate entities.
Multimodal agents that think with images iteratively manipulate visual evidence and invoke tools across many steps. Existing reinforcement learning methods reduce trajectories to scalar rewards, forcing the policy to discover reusable tool-use patterns from scratch on every new task; memory-based alternatives retain past experience, yet they rely on test-time retrieval, without updating the policy to absorb reusable patterns from that experience. Our key insight is that multimodal reasoning trajectories should be distilled into reusable skills that co-evolve with the policy during training, rather than being consumed as rewards or retrieved from a static store. To this end, we propose SPyCE (Skill-Policy Co-evolution), a framework that distills trajectories into a hierarchical skill library and updates it throughout reinforcement learning. Execution skills capture local visual operations, while workflow skills encode high-level priors that orchestrate tool use. During training, the policy model conditions on retrieved skills to guide its rollouts, while the skill library evolves using valuable rollouts generated by the policy. This creates a closed loop in which improved policies yield better skills, and the evolving skill library, in turn, provides stronger priors for policy rollouts. Experiments across eight benchmarks demonstrate that SPyCE consistently outperforms both RL-based and memory-based baselines. Further analysis reveals that both the hierarchical skill design and the co-evolution mechanism are critical to our design. These results suggest joint skill-policy optimization as a promising paradigm for building capable multimodal agents.
