Search papers, labs, and topics across Lattice.
This paper introduces Structured Image Representations (SIR), which utilize Scene Graphs (SGs) to enhance the interpretability and robustness of robot learning policies. By constructing a fully connected graph from image-derived features and learning to sparsify it end-to-end, the method yields task-relevant sub-graphs that improve decision-making clarity and performance. Evaluations on the RoboCasa platform demonstrate that SIR policies achieve a 19.5% success rate compared to 14.81% for traditional image-based approaches, while also revealing dataset biases through sub-graph analysis.
Sparse graph policies not only outperform traditional image-based methods but also expose hidden dataset biases, enhancing both performance and interpretability in robot learning.
Existing robot policies based on learned visual embeddings lack explicit structure and are sensitive to visual distractions. Thus, the representations that drive their behaviour are often opaque, making their decision-making process difficult to interpret. To address this, we introduce Structured Image Representations (SIR), a method that leverages Scene Graphs (SGs) as an intermediate representation for robot policy learning. Our approach first constructs a fully connected graph, using image-derived features as initial node representations. Then, a module learns to sparsify this graph end-to-end, creating a task-relevant sub-graph that is passed to the action generation model. This process makes our model intrinsically explainable. Evaluations on RoboCasa show that our sparse graph policies outperform image-based baselines on average with 19.5% vs 14.81% success rate. Most importantly, we show that the learned sparse graphs are a powerful tool for model analysis. By analysing when the model's sub-graph deviates from human expectation, such as by including distractor nodes or omitting key objects, we successfully uncover dataset biases, including spurious correlations and positional biases. https://github.com/intuitive-robots/SIR_Model
