Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ABNAT, an attention-based noise-aware training framework designed to enhance Transformer robustness on analog compute-in-memory (CIM) systems by explicitly guiding the model to preserve attention map fidelity under hardware noise. The research highlights that CIM noise disproportionately affects deeper Transformer layers and that conventional noise-aware training methods can be overconfident despite significant attention divergence. ABNAT achieves substantial improvements in noise-degraded accuracy (up to 11.28%) and reduced attention deviation (up to 85.77%) across various Transformer modalities.
Transformers trained with ABNAT learn to mimic clean attention maps, achieving up to 85.77% less attention deviation and 11.28% higher accuracy on noisy analog hardware.
Transformer-based models have become the backbone of a wide range of tasks, but their intensive computation and data movement incur substantial energy overhead. Analog compute-in-memory (CIM) offers an energy-efficient alternative, yet its inherent hardware non-idealities perturb stored weights and can significantly degrade model performance. Although existing noise-aware training (NAT) methods improve robustness by injecting noise during training, they largely overlook the critical role of attention mechanisms in Transformer inference.To bridge this gap, we analyze how CIM-induced noise reshapes attention in Transformers and identify two key observations: (1) the divergence between clean and noisy attention maps increases substantially in deeper layers, and (2) conventional NAT can remain overconfident even when its attention routes deviate markedly from the clean reference. Motivated by these findings, we propose ABNAT, an attention-based noise-aware training framework that teaches the noisy student to preserve the clean teacher's headwise attention routes rather than merely reproduce answers, without hardware changes or inference overhead; selective variants further reduce training cost.Extensive experiments across vision, language, and multimodal Transformers show that ABNAT consistently improves robustness over corresponding NAT baselines, improving noise-degraded accuracy by up to 11.28% and reducing attention deviation by up to 85.77%.