Search papers, labs, and topics across Lattice.
This paper introduces a novel system for generating and replaying temporal enterprise scenarios to evaluate AI agents in dynamic environments. By leveraging a schema-inferred temporal description and a deterministic-plus-LLM approach, the authors create a realistic, persona-driven enterprise world that reflects the evolving nature of data across multiple applications. The key finding reveals that this method allows for efficient, reproducible evaluations of agents at any moment in time, overcoming the limitations of static snapshots in traditional assessments.
Evaluating AI agents in a dynamic enterprise environment reveals that static snapshots miss critical context, but a new system allows for real-time, persona-driven assessments of agent performance across time.
Enterprise AI agents act across many apps whose data changes continuously, so an answer is correct only relative to what data existed and who could see it at the moment it was asked. Offline evaluation today grades against a single static snapshot, effectively the end of the episode. So, it can only evaluate one situation, the final one, even though every earlier moment of the episode is a different situation that invites its own realistic questions with its own correct answers. Recreating each of those moments as a separate snapshot would mean re-provisioning a whole tenant per instant, which is prohibitively costly; and even a single snapshot leaks future state hidden inside records and cannot represent the multi-app, time-ordered way real work happens. Our system closes two gaps at once: it generates a realistic, persona-driven, temporally-evolving enterprise world from real research, and replays that world at any chosen moment to evaluate any pluggable agent. A schema-inferred temporal description drives a deterministic-plus-LLM rebuild of each record's past state; because the queryable moments are finite, all rebuilds are precomputed into a compact difference cache, making evaluation a fast, reproducible lookup with no model in the path. We describe the design, an architecture spanning both flows, and early experience evaluating enterprise agents.
Microsoft Research