Search papers, labs, and topics across Lattice.
This paper introduces HYDRA, a design space exploration framework that optimizes the serving of hybrid Transformer-Mamba large language models (LLMs) on heterogeneous chiplet architectures. By jointly considering chiplet composition, placement, inter-chiplet bandwidth, dynamic batching, and runtime scheduling, HYDRA achieves significant performance improvements. The framework yields an average throughput increase of 1.55x and a 43.7% reduction in time-to-first-token, demonstrating the importance of co-designing architecture and runtime policies for efficient LLM serving.
Achieving up to 2.3x throughput gains, HYDRA reveals that co-designing architecture and runtime policies is essential for optimizing hybrid LLM workloads on chiplet systems.
Hybrid Transformer-Mamba large language models (LLMs) enhance long-context efficiency, but their heterogeneous computation and communication patterns complicate efficient hardware acceleration. Chiplet-based architectures offer a scalable solution by integrating specialized compute and memory units. However, the design space spanning static architectural configurations and dynamic runtime policies is prohibitively large to explore exhaustively. To address this challenge, we present HYDRA, a comprehensive design space exploration framework for hybrid LLM serving on heterogeneous chiplet systems. HYDRA jointly explores chiplet composition, placement, inter-chiplet bandwidth provisioning, dynamic batching, and runtime scheduling. It integrates communication-aware placement, dynamic batching, elastic task scheduling, and a fast Markov-based performance estimator that captures multi-tenant runtime dynamics for efficient and accurate exploration. Across all workloads, HYDRA delivers 1.55x the throughput and 43.7 percent lower time-to-first-token on average, with throughput gains reaching up to 2.3x compared to state-of-the-art baselines. These results highlight that co-designing architecture and runtime policies is critical for efficient large-scale LLM serving on heterogeneous chiplet systems.
