Search papers, labs, and topics across Lattice.
This paper introduces LUMOS, a semantic interaction layer designed to bridge the gap between AI agents and traditional operating systems by leveraging accessibility metadata and UI structures. By converting these elements into machine-readable semantic blueprints, LUMOS enables AI agents to interact with operating systems using stable identifiers and grounded actions, significantly reducing reliance on visual interpretation methods such as screenshots and OCR. The findings indicate that this approach not only enhances the efficiency of AI interactions but also paves the way for the development of AI-native operating systems that prioritize semantic understanding.
AI agents can now interact with operating systems semantically, cutting down on the inefficiencies of visual interpretation and opening new avenues for AI-native environments.
Current operating systems expose interfaces optimized for human users but not for AI agents. Humans benefit from pixels, icons, windows, visual grouping, mouse movement, and keyboard shortcuts; AI agents instead need compact semantic state, grounded actions, and reliable feedback. As a result, many computer-use agents are forced to interpret screenshots, OCR output, and visual crops, introducing high token costs, visual ambiguity, latency, and coordinate uncertainty. This paper introduces LUMOS (Language Model Unified Machine-Readable Operating-System Semantics), a semantic interaction layer between AI agents and operating systems. LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure. These results suggest a path toward AI-native operating systems and machine-readable interaction layers.
