Search papers, labs, and topics across Lattice.
This paper introduces Orchestration Reward Modeling (OrchRM), a self-supervised framework designed to enhance the orchestration of multi-agent systems (MAS) built on large language models (LLMs). By utilizing intermediate artifacts from multi-agent executions to create win-lose pairs for training a Bradley-Terry reward model, OrchRM significantly reduces the need for human supervision and computational resources. The approach achieves up to 10x improvement in token usage efficiency and an 8% increase in accuracy for MAS test-time scaling across various domains, highlighting its potential for robust multi-agent orchestration.
OrchRM slashes training costs while boosting orchestration accuracy, proving that self-supervised reward modeling can revolutionize multi-agent coordination.
Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.
