Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the training efficiency of distilling a large speech foundation model (SFM) into a more efficient student model by introducing a novel interleaved stacking method. The authors demonstrate that this approach not only accelerates the training process but also preserves the performance of the model by maintaining layer-specific knowledge throughout the stacking process. Validation on the SUPERB benchmark shows that interleaved stacking significantly enhances the deployment speed of SFMs without compromising their effectiveness.
Interleaved stacking preserves critical layer-specific knowledge in speech model distillation, leading to faster training without sacrificing performance.
Distilling a large speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM distillation remains underexplored. In this work, we explore training acceleration of SFM distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.