Search papers, labs, and topics across Lattice.
This paper introduces PPT-Eval, a comprehensive benchmark consisting of 120 PowerPoint tasks designed to evaluate computer-use agents in both content creation and presentation editing. The evaluation framework incorporates nuanced rubrics that allow for partial credit and qualitative feedback, addressing the limitations of binary success metrics in capturing the complexity of these multimodal tasks. Results show that even advanced models like Claude-4.5-Opus achieve only a 45% success rate, highlighting the challenges agents face in this domain and the need for improved methodologies.
Even top-performing AI models struggle with PowerPoint tasks, achieving only 45% success rates despite a robust evaluation framework that rewards nuanced performance.
Creating and editing slides is a rich, multimodal activity that is ubiquitous in professional and educational settings, making it an ideal testbed for real-world computer-use agents. Microsoft PowerPoint is among the most widely adopted and feature-rich environments for presentation creation. We introduce PPT-Eval, a benchmark of 120 PowerPoint tasks across 12 files that cover both content creation and presentation editing scenarios, organized by difficulty. A central challenge in this domain is evaluation: tasks are complex, multimodal, and often admit many valid solutions. Moreover, today's agents frequently make only partial progress, which binary success metrics fail to capture. To address this, we design a robust evaluation framework to help create task-specific rubrics for PowerPoint tasks, taking inspiration from and building on past works for rubric-based evaluation. These rubrics award partial credit for intermediate steps, penalize unnecessary changes and poor aesthetics, and provide natural language feedback. This nuanced approach proves highly effective, achieving a Kendall's 蟿-b correlation of 0.77 with human judgments. We find that existing frontier agents still struggle with solving PowerPoint tasks, with strong models like Claude-4.5-Opus achieving only a 45% success rate and an average partial score of 57%. The benchmark is located at: https://microsoft.github.io/ppteval.
CMU ML