Search papers, labs, and topics across Lattice.
This study explores the Recursive-Transformer architecture for Automatic Speech Recognition (ASR), focusing on the effects of recursion depth and layer allocation. By applying layer sharing through depth recursion, the authors achieve a significant reduction in model size鈥攂y 66%鈥攚hile maintaining competitive performance levels. The findings suggest that the Recursive-Transformer could be a practical solution to the challenges posed by large model sizes in ASR systems.
Achieving a 66% reduction in parameters without sacrificing ASR performance opens new avenues for efficient model design.
Transformer-based architectures have led to significant improvements in Automatic Speech Recognition (ASR), often at the cost of substantially increased model sizes. A promising approach to address this issue is layer sharing through depth recursion, commonly referred to as the Recursive-Transformer, which involves repeatedly applying the same layers within the model. Despite its potential shown in other fields, this technique remains relatively unexplored in ASR. In this paper, we present an experimental study of the Recursive-Transformer applied to ASR encoder architectures. We systematically investigate the impact of recursion depth and layer allocation within the Recursive-based Transformer. Our results demonstrate that the Recursive-Transformer is a viable alternative, especially when recurrence is applied in the latent space with a restricted number of loops, obtaining comparable performance while reducing the parameter count by 66%.
