Search papers, labs, and topics across Lattice.
The paper introduces AIvaluateXR, a benchmarking framework for evaluating the performance of on-device LLMs on XR devices, addressing the challenge of selecting optimal model-device combinations for human-AI interaction. They deployed and evaluated 17 LLMs across four XR platforms (Magic Leap 2, Meta Quest 3, Vivo X100s Pro, Apple Vision Pro), measuring performance consistency, processing speed, memory usage, and battery consumption under varying conditions. The study uses 3D Pareto Optimality to identify optimal device-model pairs and compares on-device performance with client-server and cloud-based setups, providing insights for optimizing LLM deployment in XR.
Forget cloud costs: this benchmark reveals the real-world performance tradeoffs of running LLMs directly on XR devices like Apple Vision Pro and Meta Quest 3.
The deployment of large language models (LLMs) on extended reality (XR) devices has great potential to advance the field of human-AI interaction. In the case of direct, on-device model inference, selecting the appropriate model and device for specific tasks remains challenging. In this paper, we present AIvaluateXR, a comprehensive evaluation framework for benchmarking LLMs running on XR devices. To demonstrate the framework, we deploy 17 selected LLMs across four XR platforms: Magic Leap 2, Meta Quest 3, Vivo X100s Pro, and Apple Vision Pro, and conduct an extensive evaluation. Our experimental setup measures four key metrics: performance consistency, processing speed, memory usage, and battery consumption. For each of the 68 model-device pairs, we assess performance under varying string lengths, batch sizes, and thread counts, analyzing the trade-offs for real-time XR applications. We propose a unified evaluation method based on the 3D Pareto Optimality theory to select the optimal device-model pairs from quality and speed objectives. Additionally, we compare the efficiency of on-device LLMs with client-server and cloud-based setups, and evaluate their accuracy on two interactive tasks. We believe our findings offer valuable insight to guide future optimization efforts for LLM deployment on XR devices. Our evaluation method can be used as standard groundwork for further research and development in this emerging field. The source code and supplementary materials are available at: www.nanovis.org/AIvaluateXR.html
