Search papers, labs, and topics across Lattice.
ArcLight is a new LLM inference architecture designed to maximize the utilization of many-core CPUs, which are often underutilized by existing frameworks due to cross-NUMA memory access overhead. It tackles this issue with efficient memory management, thread scheduling, and fine-grained tensor parallelism to reduce cross-node memory access. Experiments demonstrate that ArcLight achieves up to 46% higher inference throughput compared to mainstream frameworks while maintaining compatibility with various CPU devices.
Squeeze 46% more LLM inference throughput from your many-core CPUs with ArcLight, a new architecture that overcomes the cross-NUMA memory access bottleneck.
Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.
