Search papers, labs, and topics across Lattice.
This paper introduces CEFITO, a novel procedure planning approach that learns an action-conditioned representation space to enhance inference-time planning in instructional videos. By reformulating procedure planning as a task-constrained optimization problem, CEFITO effectively omits irrelevant actions, addressing a significant limitation of existing methods that treat all actions as plausible. The approach achieves state-of-the-art accuracy on two established benchmarks, demonstrating its effectiveness in enforcing task-specific logical constraints.
CEFITO's innovative action-conditioned representation space allows for precise inference-time planning by eliminating irrelevant actions, setting a new standard in procedure planning accuracy.
Procedure planning seeks to estimate a sequence of actions to transition from an observed initial state to a given goal state. Current procedure planning approaches directly predict action sequences from latent representations using feed-forward neural networks or diffusion-based inference. These paradigms treat every action as plausible, lacking the ability to enforce task-specific logical constraints that render certain actions irrelevant or not plausible. We propose CEFITO, a procedure planning approach that learns a predictor to express an action-conditioned representation space. Based on this representation space, we formulate procedure planning as a task-constrained optimization problem. Unlike prior methods, CEFITO explicitly reasons over the action space by omitting irrelevant actions during inference-time planning. This reformulation enables effective procedure planning and achieves state-of-the-art accuracy on two established procedure planning benchmarks.
