Search papers, labs, and topics across Lattice.
This paper introduces MeshPriorDiT, a hierarchical model that enhances action-conditioned cloth dynamics prediction by combining a mesh-based GNN for local deformation with a Residual DiT for long-range coordination. By decomposing future cloth motion into a structured mesh prior and a generative residual, the model effectively captures both local topological responses and global interactions. Evaluation across three cloth manipulation tasks shows that MeshPriorDiT significantly reduces average Global MSE by 43.42% compared to a GNN-only baseline and 75.03% compared to a DiT-DDPM baseline, while maintaining competitive edge-strain MSE.
MeshPriorDiT achieves a groundbreaking 75% reduction in prediction error for cloth dynamics by integrating local and global modeling strategies.
Action-conditioned cloth dynamics prediction requires both locally plausible deformation and long-range coordination. Existing approaches largely follow two paradigms. Mesh-based GNNs capture local physical responses through material connectivity. However, their finite message-passing range limits coordination between topologically distant regions, while autoregressive rollouts tend to accumulate prediction errors. Transformer-based dynamics models capture long-range interactions through global attention, but often operate without explicit material connectivity and must learn local topological responses directly from data. We propose MeshPriorDiT, a hierarchical dynamics model that decomposes future cloth motion into a structured mesh prior and a generative residual. An action-conditioned mesh GNN first predicts multi-step vertex displacements, yielding a reference trajectory that respects material topology and grasp constraints. Conditioned on historical states, planned actions, and the mesh prior, a Residual DiT then uses conditional flow matching to jointly generate the residual motion not captured by the prior. The generated residual is further rescaled and decoded using material adjacency to coordinate corrections across neighboring vertices. We evaluate MeshPriorDiT on 15-step autoregressive rollouts across three cloth manipulation tasks. Averaged over the three tasks, MeshPriorDiT reduces average Global MSE by 43.42% relative to the GNN-Only baseline and by 75.03% relative to the DiT-DDPM baseline, while maintaining a favorable Edge-strain MSE comparable to that of GNN-Only.
