Search papers, labs, and topics across Lattice.
FreeToken is an edge-native mixture of experts (MoE) serving system that optimizes the deployment of large open-weight models on personal machines by dynamically adapting to the unique hardware resources available. By co-designing the serving stack with considerations for changing agent workloads and heterogeneous edge resources, FreeToken enables efficient execution of models ranging from 35B to 753B parameters on various consumer-grade hardware. This approach significantly expands the practical capabilities of local AI deployment, allowing users to leverage their existing machines for advanced AI tasks.
FreeToken transforms personal machines into powerful platforms for running massive AI models, enabling users to deploy frontier-scale intelligence without specialized infrastructure.
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
