Search papers, labs, and topics across Lattice.
This paper introduces AppDeltaWorld, a transition-grounded delta code world model designed to enhance mobile GUI agents' interaction capabilities by predicting the next GUI state as a code update rather than an image. By leveraging app-specific HTML references and generating executable HTML conditioned on various contextual factors, AppDeltaWorld demonstrates significant improvements in structural layout and UI element reconstruction over traditional image-only and code-only approaches. The model not only achieves the highest fidelity on CMGUIBench-500 but also enables state-of-the-art performance for the AppDeltaAgent in various mobile interaction benchmarks through innovative training and reinforcement learning techniques.
Achieving the highest fidelity in mobile GUI interaction, AppDeltaWorld redefines how agents predict and interact with app interfaces by focusing on code updates instead of images.
Mobile GUI agents can operate apps through pixel perception and touch actions, making them a promising interface for collecting and improving long-horizon mobile interaction policies. However, real trajectories are difficult to obtain for sensitive apps and privacy-critical operations. At the same time, existing simulated environments are costly to scale up, and GUI world models still suffer from unstable generation, limited modality coverage, and inconsistent action-transition logic. To address these limitations, we propose AppDeltaWorld, a transition-grounded delta code world model that predicts the next GUI as a reachable code update rather than as an unconstrained image or text description. AppDeltaWorld retrieves app-specific Level-1 HTML references under an action-transition constraint, generates Level-2 executable HTML conditioned on the current screen, action, predicted next-screen text, and retrieved structure, and inserts generated visual assets into image slots before browser rendering. As a world model, AppDeltaWorld achieves the highest fidelity on CMGUIBench-500 under Code2World evaluation, with clear gains in structural layout and UI element reconstruction over image-only and code-only baselines. As a training environment, AppDeltaWorld supports filtered closed-loop SFT data construction that, when combined with public supervision, enables AppDeltaAgent to achieve state-of-the-art performance on AndroidLens and consistent gains on MobileGym and MobileWorld. Moreover, world-model-based test-time reinforcement learning enables policy adaptation and shows further improvements without additional interaction with real apps.
