Search papers, labs, and topics across Lattice.
This paper introduces Hydra, a phase-aware workload characterization framework designed to analyze LLM inference on edge System-on-Chips (SoCs) by integrating prompt timing data with hardware telemetry. By evaluating multiple SoC generations and a diverse set of instruction-tuned LLMs, the study reveals that aggregate latency can obscure critical insights about performance and resource utilization, particularly in relation to backend structure and quantization effects. The findings emphasize the importance of phase-level analysis in understanding the efficiency and deployment characteristics of edge LLMs, with a comprehensive dataset of over 107K records made publicly available.
Aggregate latency masks crucial insights into LLM inference efficiency, revealing that backend changes and quantization effects significantly influence performance metrics.
Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a common-schema, phase-aware workload characterization framework for LLM inference on edge SoCs. Hydra instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema and fuses those records with hardware telemetry, enabling a multi-dimensional characterization of performance, system-resource utilization, and efficiency across prefill and decode phases. Using Hydra, we evaluate three consecutive edge System-on-Chip (SoC) generations (AGX Xavier, AGX Orin, and AGX Thor), 13 instruction-tuned LLMs from seven families, five execution formats, and consider input/output-length sensitivity. The resulting artifact contains roughly 107K per-prompt records and is publicly released with Hydra. Our analysis shows that aggregate latency alone hides key deployment effects: backend structure changes where latency is introduced, quantization reduces memory traffic and energy but does not predict power monotonically, and SoC generation changes how utilization and efficiency should be interpreted. By connecting phase-level timing with system-resource utilization and efficiency metrics, Hydra enables reproducible, phase-aware characterization of edge LLM inference. Hydra's source code and the collected per-prompt trace corpus are available open-source at: https://github.com/amirtaherin/hydra
