Search papers, labs, and topics across Lattice.
SenWorld is a deterministic digital-twin simulation designed to generate context-rich evaluation data for smartphone personal assistants while preserving user privacy. By simulating personas navigating a real-world environment, the system produces data that closely aligns with actual user behavior, achieving a Jensen-Shannon divergence of 0.070 in category distribution and below 0.1 for communication rhythms. The evaluation revealed 78 specific failures in a production assistant, highlighting retrieval errors without reliance on post-hoc annotations or large language model assessments.
SenWorld uncovers 78 critical failures in smartphone assistants by generating contextually rich evaluation data that mirrors real user interactions.
Smartphone personal assistants reason over longitudinal personal data, yet evaluating them requires context-rich evaluation data whose correct answers are known, and real device traces are too privacy-sensitive to share. To address this challenge, we present SenWorld, a physically grounded, deterministic, event-sourced digital-twin simulation that generates such data with ground truth fixed by construction. In SenWorld, personas live through a full day in a world built from real map, weather, holiday, and network data; every observable signal is archived in full-system snapshots; and each evaluation case is labeled by a pointer to an existing record rather than by post-hoc annotation or a large language model (LLM) judge. We evaluate this method with 16 personas in Beijing. The generated data closely matches the held-out real-user benchmark in category distribution (Jensen--Shannon divergence (JSD) 0.070) and in the daily rhythm of communication records (JSD below 0.1), though generated records remain shorter than real ones. Without scripted interaction, personas form a fully reciprocated dialogue subgraph and differentiated behavioral repertoires. Projected into 717 evaluation cases, the generated data exposes 78 failures in a production smartphone assistant, concentrating on call and Short Message Service (SMS) records while contacts, schedules, and alarms never fail. The snapshot pointer confirms each failure as an assistant-side retrieval error, with no LLM judge involved. Overall, SenWorld offers a privacy-safe, reproducible, and distribution-checked path to evaluation data whose labels are fixed by construction.
