Search papers, labs, and topics across Lattice.
This paper investigates the performance of a hand-written kernel for fp32 GEMM operations on Apple Silicon's M1 AMX, revealing that the throughput advantage over the Accelerate library is not due to a faster inner loop but rather underutilized deployment strategies. By microbenchmarking, the authors identify that the inner loop is load-issue bound, with throughput significantly dropping when operand loads interleave with the FMA32 stream. The key findings show that optimizing for multi-threaded panel filling and pre-packing weights can yield a bit-exact kernel that outperforms all Accelerate paths, achieving a 1.44x increase in full-forward throughput for LLM prefill tasks.
Optimizing deployment strategies on the M1 AMX can double throughput for LLM prefill operations, far exceeding existing libraries.
On Apple Silicon the fp32 GEMMs dominating LLM prefill are dispatched by Accelerate to a matrix coprocessor (AMX) on the M1-M3. We ask where a hand-written kernel's throughput over Accelerate comes from on the M1 AMX, and reach a structural conclusion: not a faster inner loop. By microbenchmark the inner loop is load-issue bound -- once any operand load interleaves with the FMA32 stream, single-thread throughput falls to a 610-to-680 GFLOPS band, under half the load-free rate. The gain comes from two deployment-level levers Accelerate underuses: fine multi-thread panels filling the M1's second on-chip AMX block (winning the K >= N shapes), and pre-packing the constant weight at load (winning the N > K shapes). A bit-exact direct-AMX kernel using both is the fastest bit-exact fp32 GEMM path we find on the M1: it exceeds all three Accelerate fp32 paths (cblas_sgemm, BNNSMatMul, and the BNNS Graph compiler) at all twelve LLM prefill GEMMs at S = 128 (GPT-2 to Llama-7B scale), leading the fastest, BNNS Graph, by 1.17 -- and by 1.09 at the three shapes where it too holds fp32 -- with geometric means of 1.58 over BNNSMatMul and about 2.0x over cblas_sgemm. Every output is bit-identical to Accelerate, whereas BNNS Graph is bit-exact at only three of twelve shapes, the rest at reduced precision (error up to 1.4e-3). Dropped into llama.cpp for its cblas_sgemm prefill matmuls, it raises measured full-forward throughput from 291 to 420 tokens/s (1.44x, bit-identical) at 128-token prefill -- end-to-end, not a GEMM-only ratio. The contribution is this shape-resolved M1-AMX characterization (microbenchmark, two-block aggregate, per-core occupancy probe), leaving fine-panel scheduling and pre-packing as the only two levers above an inner loop at the hardware limit; mis-tuning the single column-panel width costs nearly 2x.
