Search papers, labs, and topics across Lattice.
This paper benchmarks CPU-based deep learning inference scalability for ResNet models on legacy (Intel Xeon E5-2403 v2) and modern (Intel Xeon "Granite Rapids") hardware, revealing performance bottlenecks and architectural saturation points. The study demonstrates that legacy CPUs saturate quickly at small batch sizes, while modern CPUs leveraging Intel AMX achieve higher throughput but suffer from performance degradation due to oversubscription beyond physical core limits. The authors introduce GDEV-AI, a reproducible benchmarking framework, to analyze these scalability behaviors and provide a vendor-neutral baseline for capacity planning.
Modern CPUs with AMX significantly boost deep learning inference throughput, but oversubscribing cores can lead to performance degradation and amplified tail latency.
The deployment of deep learning inference in production environments continues to grow, where throughput, latency, and hardware efficiency are critical. Although specialized accelerators are increasingly adopted, many inference workloads still run on CPU-only systems, particularly in legacy data centers and cost-sensitive environments. This study investigates the scalability limits of CPU-based inference for convolutional neural networks by benchmarking ResNet models across varying batch sizes on two hardware tiers: a legacy Intel Xeon E5-2403 v2 processor and a modern Intel Xeon 6 "Granite Rapids" platform. Results show that legacy CPUs quickly reach throughput saturation, with limited scaling beyond small batch sizes due to instruction-level and memory constraints. In contrast, the Granite Rapids system leverages Intel Advanced Matrix Extensions (AMX) to achieve substantially higher throughput. However, oversubscription beyond physical core limits introduces execution contention and tail-latency amplification, revealing a performance degradation regime in modern architectures. We introduce GDEV-AI, a reproducible benchmarking framework for analyzing scalability behavior and architectural saturation in CPU-based inference. By establishing a vendor-neutral baseline, this work provides empirical insight into performance bottlenecks and informs capacity planning in heterogeneous data center environments.
