Search papers, labs, and topics across Lattice.
To prevent vision-language models from hallucinating unobserved objects or generating physically impossible action sequences during embodied planning, the authors present CLAMP, an inference-time neurosymbolic decoding framework. The system combines scene-derived hard token masks with an HMM-based world-state lookahead module that dynamically reweights generation probabilities according to symbolic preconditions and goal reachability. Across VLABench, SafeAgentBench, and TaPA, CLAMP enforces strict affordance and safety guarantees on a frozen VLM, effectively shifting the primary planning bottleneck from model hallucination to upstream visual perception errors.
Hallucinated objects and unexecutable actions in embodied planning don't require fine-tuning to fix: pairing a frozen VLM with inference-time symbolic masking and an HMM state lookahead guarantees feasible, grounded plans out of the box.
Embodied planning increasingly relies on vision-language models (VLMs) to translate instructions and visual observations into executable action sequences. However, fluent plans are not always executable. A VLM may refer to objects that are not visually observed, select actions whose required affordances are unavailable, or violate syntax and action constraints. We introduce CLAMP, a multimodal constraint-grounding framework that turns scene evidence into decoding-time constraints for a frozen VLM planner. CLAMP uses the initial observation to restrict object references to those supported by the scene, while a provided symbolic action model specifies state transitions and goals. During decoding, hard masks eliminate invalid next-token candidates, while a Hidden Markov Model (HMM)-based world-state lookahead module reweights the probabilities of the remaining feasible candidates based on action preconditions and goal reachability. This allows the planner to retain the VLM's language prior while preventing visually unsupported, unsafe, or infeasible candidates from entering the plan. For unseen tasks and environments, CLAMP adapts the HMM at test time using label-free continuations sampled from the frozen VLM. Experiments on VLABench, SafeAgentBench, and TaPA show that scene-grounded constraints improve object grounding and safety, while most remaining failures stem from perception errors or misaligned constraint specifications.
