Search papers, labs, and topics across Lattice.
This paper introduces Vortex, a novel system designed to streamline the deployment and evaluation of sparse attention algorithms for large language models (LLMs). By integrating a Python-embedded frontend with a page-centric tensor abstraction, Vortex allows for rapid prototyping and testing of various sparse attention strategies, achieving up to 3.46 times higher throughput compared to full attention while maintaining accuracy. The system also facilitates experimentation with emerging architectures and large models, demonstrating throughput improvements of up to 4.7 times on specific LLMs, significantly accelerating the design process for sparse attention algorithms.
Vortex achieves up to 4.7 times higher throughput for large language models, revolutionizing how researchers can prototype and evaluate sparse attention algorithms.
Sparse attention is becoming increasingly important for serving large language models (LLMs) as generation lengths continue to grow. However, deploying and evaluating new sparse attention algorithms at scale remains highly engineering-intensive, slowing both human researchers and AI agents in exploring the sparse attention design. To address this challenge, we present Vortex, a system that combines a Python-embedded frontend language atop a page-centric tensor abstraction for expressing a broad range of sparse attention algorithms, with an efficient backend tightly integrated into modern LLM serving stacks. Vortex enables rapid prototyping, deployment, and evaluation of sparse attention algorithms, effectively translating their theoretical efficiency gains into real-world throughput improvements. As a result, Vortex substantially accelerates the design and iteration of sparse attention algorithms. First, AI agents use Vortex to automatically generate and refine diverse algorithms, the best reaching up to $3.46\times$ higher throughput than full attention while preserving accuracy. Second, Vortex extends sparse attention to emerging architectures and very large models that are otherwise hard to experiment with, reaching up to $4.7\times$ higher throughput on the MLA-based GLM-4.7-Flash and $1.37\times$ on the 229B-parameter MiniMax-M2.7 on NVIDIA B200 GPUs.
CMU ML
NUS