Search papers, labs, and topics across Lattice.
This paper analyzes the rapid advancements in vision generative AI, emphasizing the need for models that function effectively under hardware constraints in edge applications like autonomous vehicles and mobile devices. It highlights the reactive evolution of hardware in response to the increasing demands of these models, quantifying their parameter costs and energy efficiency across various accelerator platforms. The authors advocate for a software-hardware co-design approach to ensure that generative AI models are optimally matched to their deployment environments, enhancing sustainability and accessibility.
Vision generative AI models could revolutionize edge applications, but only if we rethink how they are designed in relation to hardware constraints from the outset.
Vision generative artificial intelligence (AI) has emerged as one of the most rapidly advancing areas of deep learning. The explosion of multimodal models has made them widely associated with text-to-image applications running on large datacentres. However, vision generative models are equally needed in applications that operate under strict hardware constraints at the edge, including autonomous vehicles, agricultural sensors, and mobile devices. In this Perspective, we argue that progress in vision generative AI has been driven by output quality, with hardware evolving reactively to accommodate growing model demands. We quantify the parameter cost and energy efficiency of these models across a range of accelerator platforms, and map four generative model families against seven real-world application domains. Finally, we advocate a software-hardware co-design approach, where deployment constraints are considered from the start of the design process, ensuring that the"right model"runs on the"right hardware"to serve the"right application", making generative AI deployment sustainable and accessible across a much broader range of platforms.
