Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates how intelligent visual systems, specifically human and macaque vision, integrate object appearance and motion while maintaining robustness to changes in appearance. By comparing neural activity in the macaque inferior temporal cortex with various neural network models, the researchers found that while temporal integration enhanced object representations, most video recognition models struggled with generalization under appearance disruption. The key finding is that predictive world models exhibited strong cross-appearance generalization and closely matched the neural responses of macaques, highlighting the importance of motion integration for robust dynamic vision in both biological and artificial systems.
Predictive world models outperform traditional video recognition systems by achieving robust object representation through motion integration, a principle not yet realized in AI.
How does an intelligent visual system combine what objects look like with how they move while remaining robust as appearance changes? We addressed this question by comparing human perception and neural activity in macaque inferior temporal cortex with representations from image- and video-based neural networks spanning recognition, segmentation, optic-flow processing and predictive world modeling. Temporal integration improved object representations, but most video recognition models generalized poorly when appearance was disrupted while motion structure was preserved. Humans and macaque IT remained robust. Notably, predictive world models combined strong cross-appearance generalization with the closest correspondence to IT, outperforming other video-modeling approaches in neural fidelity. Yet no model reproduced the cortical transformation from early appearance-dominated responses toward later appearance-invariant motion coding. These results identify progressive integration of motion into object representations as a principle of robust dynamic vision and implicate predictive learning as a promising route toward realizing this computation in artificial systems.