Search papers, labs, and topics across Lattice.
This paper investigates the impact of hardware noise on diffusion models during inference, revealing that the early denoising stages are particularly susceptible to errors from analog compute-in-memory (CIM) hardware. By employing a noise model validated against real CIM chip measurements, the authors identify that correlated prediction errors propagate more significantly in the early stages of denoising. They introduce ASSERT, a training-free sampler that enhances stochasticity in early iterations, achieving substantial improvements in image generation quality, with up to 2.58脳 lower FID compared to deterministic methods, without altering model parameters or evaluations.
Early denoising stages in diffusion models are up to 2.58脳 more vulnerable to hardware noise, but a novel approach can mitigate this without retraining.
Diffusion models achieve strong image generation quality but incur high iterative denoising costs. Analog compute-in-memory (CIM) can accelerate matrix-vector multiplications, yet spatial memory variations perturb weights and accumulate during sampling. Unlike conventional neural networks, diffusion models' temporal sensitivity to hardware noise remains underexplored. We investigate diffusion inference using a noise model calibrated and validated against measurements collected from multiple physical CIM chips. Our results show that the early, high-noise denoising stage is substantially more vulnerable than the final refinement stage. A first-order trajectory analysis attributes this behavior to the repeated propagation of correlated prediction errors induced by a fixed hardware mapping. Based on this observation, we propose ASSERT, a training-free sampler that uses higher stochasticity early and smoothly transitions to deterministic denoising. The injected stochasticity changes subsequent activation trajectories and thereby reduces their alignment with persistent spatial errors. Across the evaluated settings, ASSERT achieves up to 2.58$\times$ lower FID than deterministic DDIM on high-resolution datasets and 7.68$\times$ lower FID in the CIFAR-10 step-count study, without changing model parameters or the number of network evaluations.
Tsinghua AI