Search papers, labs, and topics across Lattice.
This paper introduces ASIL (Agent-Software Interaction Layer), a structured interface that replaces traditional screenshot-and-click methods with JSON observations and semantic actions for software interaction. By evaluating ASIL across 15 applications, the authors demonstrate significant improvements in task success rates, achieving over 80% success with fewer than five actions per task, compared to a maximum of 26.6% success using screenshot-and-click methods. The structured approach not only enhances performance but also facilitates better training outcomes for models, with notable increases in success rates for various configurations of the Qwen model.
ASIL achieves over 80% task success with fewer than five actions, dramatically outpacing traditional screenshot-and-click methods.
Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks. ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches draw.io's MCP content contract. The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.
