Search papers, labs, and topics across Lattice.
This paper introduces LatentASR, a novel method for enhancing end-to-end automatic speech recognition (ASR) by implementing continuous latent test-time scaling on a frozen ASR backbone. By utilizing two small trainable modules鈥攁 Latent Adapter for refining latent prefixes and a Value Head for optimizing computation鈥擫atentASR effectively reduces word error rates (WER) on challenging inputs without requiring extensive retraining. The approach demonstrates significant performance improvements, achieving a 16.0% relative character error rate reduction on accented and code-switched speech, while maintaining efficiency by dynamically halting processing on easier utterances.
LatentASR transforms frozen ASR models into adaptive systems that intelligently allocate compute resources, achieving significant WER reductions on challenging inputs without the need for extensive retraining.
End-to-end ASR models transcribe in a single pass, leaving no room for the decoder to revisit hard inputs. We propose LatentASR, a parameter-efficient method that adds continuous latent test-time scaling to a frozen ASR backbone. Two small trainable modules drive it: a Latent Adapter that iteratively refines a few latent prefix positions through bounded, stabilized updates, and a Value Head that predicts whether extra computation will help and halts the loop early. The Qwen3-ASR-0.6B backbone stays fully frozen, and we train only ~4M extra parameters. We activate this loop with a deliberately small, diverse 500-utterance training set. Under this minimal-data regime, standard adaptation methods all regress: full fine-tuning, LoRA, and prompt tuning each increase WER. LatentASR is the only tested method that reduces WER on both clean benchmarks (FLEURS -2.54% and VoxPopuli -0.47% relative). The reductions are concentrated on intrinsically hard inputs. On accented and code-switched speech (ASCEND), LatentASR achieves a 16.0% relative CER reduction. Across 30 FLEURS languages (23,049 utterances), the multilingual WER decreases uniformly across resource tiers, confirming that the adapter generalizes without overfitting. Dynamic halting preserves most of the clean-set reduction at a fraction of the compute, skipping roughly half of all utterances at the entry gate. Our results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.
