Search papers, labs, and topics across Lattice.
This paper introduces Counterfactual Nuisance Behaviour Cloning (CFNBC), an offline data-selection framework aimed at enhancing the robustness of visuomotor imitation learning policies against visual nuisances. By generating paired clean and nuisance observations, CFNBC measures action drift to identify which demonstrations can effectively repair policy performance without requiring online execution or success labels. The results demonstrate that targeted selection of just 20-30 candidates based on action drift significantly outperforms random selection, highlighting the importance of focusing on specific response modes rather than sheer data volume for robustness improvement.
Targeted data selection can outperform random sampling in robustness repair, achieving near-optimal performance with just a fraction of the data.
Visuomotor imitation learning has demonstrated success for manipulation tasks. However, the trained policies remain brittle to visual `nuisances', with even minor task-preserving variations such as lighting, distractions or changes in colour result in heavy degradation of the trained policy's performance. While increasing data diversity can improve robustness, it is unclear which additional demonstrations are informative for a particular trained policy. We propose Counterfactual Nuisance Behaviour Cloning (CFNBC), an offline data-selection framework for targeted robustness repair. Starting from a nominal policy trained on `clean'demonstrations, CFNBC generates paired clean and nuisance observations that preserve the expert action, then measures \emph{action drift}: the change in the policy's predicted action under a nuisance that should not alter the desired behaviour. This provides a policy-specific sensitivity signal for selecting a compact, response-diverse repair set from a larger candidate pool, without requiring rollout success labels or online policy execution. We show in MuJoCo bimanual cube transfer and SimplerEnv cube stacking that action drift correlates with nuisance-induced failure, and that response-guided repair with only $20$--$30$ selected candidates substantially outperforms matched-budget random selection while approaching the performance of much larger random repair budgets. These results support a data-centric view of robustness repair: the most useful data are not necessarily the most numerous, visually diverse, or obviously difficult, but the examples that cover fragile response modes of the current policy.
