Search papers, labs, and topics across Lattice.
This paper introduces a transformer-based framework for predicting ball states in robotic table tennis, addressing the limitations of both physics-based and learning-based methods. By leveraging attention mechanisms to capture long-range temporal dependencies from a large-scale real-world dataset, the model achieves accurate long-horizon forecasting without relying on explicit flight or bounce models. The authors also present a novel sim-to-real transfer strategy, Swap Predictor at Deployment (SPAD), which enhances policy transferability by substituting the physics-based simulator with the real-world-trained predictor during deployment, effectively narrowing the sim-to-real gap.
A transformer-based predictor narrows the sim-to-real gap in robotic table tennis by leveraging extensive real-world data and attention mechanisms for long-range forecasting.
Robotic table tennis is a representative benchmark for high-speed, closed-loop robotic control in dynamic environments, where accurate and fast prediction of ball states is critical for reliable planning and control. Physics-based approaches rely heavily on accurate parameter identification and precise initial state, while learning-based methods often struggle to capture long-range temporal dependencies and are typically trained on limited or simulated data. We propose a transformer-based framework for table tennis ball state prediction that leverages attention mechanisms to model long-range temporal correlations directly from historical observations, without relying on explicit flight or bounce models. To support robust learning and generalization, we collected a large-scale real-world dataset from players of varying skill levels and diverse ball cannon configurations. The combination of a high-capacity transformer architecture and extensive real-world data enables accurate long-horizon forecasting. Building on this capability, we introduce a plug-and-play sim-to-real transfer strategy, Swap Predictor at Deployment (SPAD), which replaces the physics-based simulator used during training with the proposed real-world-trained predictor at deployment, improving the sim-to-real transferability of the policy without requiring retraining. We demonstrate that this simple substitution effectively narrows the sim-to-real gap while preserving the efficiency and scalability of simulation-based training.
