Search papers, labs, and topics across Lattice.
This paper introduces MONET, a framework extending the Stream inference modeling tool to simulate the training of neural networks on heterogeneous dataflow accelerators. MONET addresses the distinct memory and backpropagation challenges of training workloads, which are often overlooked by inference-focused tools. The framework is used to explore the design space of ResNet-18 and a small GPT-2, optimizing layer fusion configurations and activation checkpointing strategies using a genetic algorithm to improve hardware architectures.
MONET reveals the potential for significant hardware architecture improvements by modeling and optimizing neural network training, a domain often overshadowed by inference-centric design.
While hardware-software co-design has significantly improved the efficiency of neural network inference, modeling the training phase remains a critical yet underexplored challenge. Training workloads impose distinct constraints, particularly regarding memory footprint and backpropagation complexity, which existing inference-focused tools fail to capture. This paper introduces MONET, a framework designed to model the training of neural networks on heterogeneous dataflow accelerators. MONET builds upon Stream, an experimentally verified framework that that models the inference of neural networks on heterogeneous dataflow accelerators with layer fusion. Using MONET, we explore the design space of ResNet-18 and a small GPT-2, demonstrating the framework's capability to model training workflows and find better hardware architectures. We then further examine problems that become more complex in neural network training due to the larger design space, such as determining the best layer-fusion configuration. Additionally, we use our framework to find interesting trade-offs in activation checkpointing, with the help of a genetic algorithm. Our findings highlight the importance of a holistic approach to hardware-software co-design for scalable and efficient deep learning deployment.
