Search papers, labs, and topics across Lattice.
The paper introduces EXACT, a domain-specific language for representing human motions as underspecified motion programs that generate rewards for zero-shot policy inference. By composing individual policies based on the structure of EXACT programs, the approach creates executable neuro-symbolic models of human actions. Experiments on motion capture data demonstrate that these executable models improve data efficiency and capture more intuitive relationships between actions for segmentation and anomaly detection compared to monolithic approaches.
Modeling human actions as executable programs not only improves action recognition, but also unlocks more data-efficient and interpretable models compared to standard monolithic approaches.
Human-centred systems require an understanding of human actions in the physical world. Temporally extended sequences of actions are intentional and structured, yet existing methods for recognising what actions are performed often do not attempt to capture their structure, particularly how the actions are executed. This, however, is crucial for assessing the quality of the action's execution and its differences from other actions. To capture the internal mechanics of actions, we introduce a domain-specific language EXACT that represents human motions as underspecified motion programs, interpreted as reward-generating functions for zero-shot policy inference using forward-backwards representations. By leveraging the compositional nature of EXACT motion programs, we combine individual policies into an executable neuro-symbolic model that uses program structure for compositional modelling. We evaluate the utility of the proposed pipeline for creating executable action models by analysing motion-capture data to understand human actions, for the tasks of human action segmentation and action anomaly detection. Our results show that the use of executable action models improves data efficiency and captures intuitive relationships between actions compared with monolithic, task-specific approaches.
