Search papers, labs, and topics across Lattice.
This paper introduces StateFactory, a method for learning factorized world state representations using language models to predict rewards across diverse domains. StateFactory transforms unstructured observations into hierarchical object-attribute structures, enabling reward estimation based on the semantic similarity between the current and goal states. Experiments on the RewardPrediction benchmark show StateFactory achieves superior zero-shot reward prediction compared to VLWM-critic and LLM-as-a-Judge models, which translates to improved agent planning performance in AlfWorld and ScienceWorld.
Factorizing world states with language unlocks surprisingly strong zero-shot reward prediction across diverse environments, outperforming end-to-end learned critics and LLM-as-a-judge approaches.
Agents must infer action outcomes and select actions that maximize a reward signal indicating how close the goal is to being reached. Supervised learning of reward models could introduce biases inherent to training data, limiting generalization to novel goals and environments. In this paper, we investigate whether well-defined world state representations alone can enable accurate reward prediction across domains. To address this, we introduce StateFactory, a factorized representation method that transforms unstructured observations into a hierarchical object-attribute structure using language models. This structured representation allows rewards to be estimated naturally as the semantic similarity between the current state and the goal state under hierarchical constraint. Overall, the compact representation structure induced by StateFactory enables strong reward generalization capabilities. We evaluate on RewardPrediction, a new benchmark dataset spanning five diverse domains and comprising 2,454 unique action-observation trajectories with step-wise ground-truth rewards. Our method shows promising zero-shot results against both VLWM-critic and LLM-as-a-Judge reward models, achieving 60% and 8% lower EPIC distance, respectively. Furthermore, this superior reward quality successfully translates into improved agent planning performance, yielding success rate gains of +21.64% on AlfWorld and +12.40% on ScienceWorld over reactive system-1 policies and enhancing system-2 agent planning. Project Page: https://statefactory.github.io
