Search papers, labs, and topics across Lattice.
This study investigates the impact of human feedback on self-evolving agents, which traditionally improve through self-play but risk safety degradation. The authors introduce ANCHOR, a framework that simulates human oversight during various phases of agent evolution, and evaluate its effectiveness on two open-source systems in coding, mathematical reasoning, and safety. Results indicate that even minimal human supervision significantly reduces safety risks while maintaining performance, with output verification being the most critical phase for effective intervention.
Limited human oversight can dramatically enhance the safety and stability of self-evolving agents, challenging the notion that autonomy must come without guidance.
Self-evolving agents improve through continual self-play and self-generated learning signals, but autonomous evolution can also cause capability degradation and safety drift. Although human feedback has proven effective for static and post-trained agents, its role in self-evolving systems remains underexplored. We introduce Agent Norm Correction through Human-like Oversight and Review (ANCHOR), an LLM-based framework that simulates human supervision and delivers feedback at various phases of self-evolution. With ANCHOR, we evaluate two representative open-source self-evolving agent systems across coding, mathematical reasoning, and safety. Our results show that even limited supervision substantially mitigates safety degradation while preserving stable performance on core evolutionary objectives. Further analysis shows that supervision over the output verification phase is the most effective for intervention, whereas increasing supervision frequency yields diminishing returns. These findings provide empirical evidence and practical guidance for designing more stable, controllable, and human-aligned self-evolving agent systems.
