Search papers, labs, and topics across Lattice.
VisHarness is introduced as a trainable visual agent that learns to harness a suite of heterogeneous visual expert models for complex visual reasoning tasks. It decouples high-level reasoning from low-level task execution, allowing the agent to leverage specialized visual models without sacrificing general intelligence. The agent learns a generalizable expert-harnessing policy through reinforcement learning, using a dynamic visual memory archiving technique to manage the overhead of multi-turn interactions.
Forget end-to-end training for complex vision tasks; VisHarness shows that a lightweight agent can outperform task-specific models by learning to orchestrate a diverse set of pre-trained visual experts.
Recent progress in computer vision has produced a wide range of powerful specialized models for detection, segmentation, counting, and other visual tasks. However, these models are usually optimized for isolated task formulations, making it difficult to directly support general-purpose visual intelligence, especially when a task requires complex language understanding and dense small-object perception. In this paper, we propose VisHarness, a trainable visual agent that decouples high-level perception, reasoning, and decision-making from low-level task execution. Instead of training a model to solve a specific visual task, VisHarness learns to harness a set of carefully designed heterogeneous visual experts. This paradigm preserves the general intelligence of the agent while fully leveraging the precision advantages of specialized visual models in concrete visual tasks. With only lightweight training, VisHarness learns a generalizable visual expert-harnessing policy and can solve common fundamental vision tasks under various complex conditions through multi-turn interactions with visual expert models. To enable efficient on-policy reinforcement learning training in a live environment, we introduce dynamic visual memory archiving, which mitigates the rapidly accumulating visual-token overhead caused by multi-turn interactions with visual expert models. Experiments on four representative benchmarks covering reasoning segmentation, generalized referring segmentation, dense small-object detection, and referring counting demonstrate that VisHarness substantially outperforms existing general-purpose models and achieves competitive or superior performance compared with task-specific models.
