Search papers, labs, and topics across Lattice.
This paper presents an event-based active vision system designed for real-time tracking and spin estimation of balls in professional sports, addressing the challenges posed by the small size, high speed, and rapid rotation of the balls. By integrating a high-speed event camera, galvanometer mirrors, and a focus-tunable lens, the system achieves state-of-the-art accuracy in spin estimation for various sports, including table tennis, baseball, tennis, and golf. The proposed methods yield impressive results, with a latency of just 3 ms and a throughput of 750 Hz during live matches, demonstrating the system's practical applicability in competitive environments.
Achieving real-time spin estimation with just 3 ms latency could revolutionize how we analyze and enhance performance in professional sports.
Spin plays a crucial role in many ball sports due to its effect on the trajectory of the ball. Vision-based estimation of the ball's spin during a game with conventional cameras is challenging due to the ball's small size, high speed, and fast rotation. To address these challenges, we propose an event-based active vision system that can track unmodified balls and measure their spin in real-time. The system consists of an event camera for its high temporal resolution and minimal motion blur, high-speed pan/tilt galvanometer mirrors to keep the ball in the field of view, and a low-latency focus-tunable telephoto lens to increase the spatial resolution on the ball and keep it in focus. To track the ball, we use a hybrid approach that combines 2D event-based detection for centering and 3D positions from a ball localization system for re-initialization. For high-accuracy spin estimation, we propose an offline method that performs contrast maximization on the sphere (s-CMax). This method achieves state-of-the-art accuracy on static balls across multiple sports (table tennis, baseball, tennis, and golf), with mean magnitude and axis errors of 2.1% and 4.0 degrees, respectively. We then develop a low-latency online method for table tennis as a case study in real-time applications. This method uses an uncertainty-aware convolutional neural network trained on pseudo-ground-truth spin labels from the offline approach, combined with a GPU-accelerated batch implementation of contrast maximization for refinement. We demonstrate reliable tracking and spin estimation with a three-view setup during professional table tennis matches, with high accuracy (8.8% magnitude and 6.4 degrees axis mismatch), 3 ms latency, and 750 Hz throughput.
