Search papers, labs, and topics across Lattice.
This paper addresses the issue of timestamp drift in autoregressive automatic speech recognition (ASR) systems, which can lead to misalignment between generated timestamps and the actual audio during non-speech segments. The authors introduce REDDIT, a two-stage post-training framework that corrects these timestamps without causing catastrophic forgetting of the model's non-target behaviors. Their approach achieves significant improvements in timestamp accuracy, raising the long-gap mean Intersection over Union (mIoU) from 38.7% to 95.0% while maintaining a low update rate of model parameters and preserving overall ASR performance.
Timestamp drift in ASR can be corrected with minimal parameter updates, achieving near-perfect alignment without sacrificing model performance.
Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing. We show that these generated timestamps can drift across long non-speech spans: the transcript may remain plausible, but the decoded time axis drifts away from the audio. We study this non-speech-induced timestamp drift with self-built gap and long-gap benchmarks across 15 evaluated timestamp-producing ASR and audio-language systems. Naive timestamp-corrected fine-tuning improves alignment but can severely degrade non-target ASR behavior, exposing a forgetting problem. We propose REDDIT(REplay-based Distribution eDITing), a lightweight two-stage post-training framework that corrects timestamps while avoiding this catastrophic forgetting: it first edits timestamp targets under the model's own replayed decoder context while matching the frozen base distribution on non-timestamp tokens, then applies a short edited-prefix refinement stage. In this framework, we construct correction supervision without human transcripts or human timestamp annotations by combining VAD-trimmed speech spans with inserted non-speech gaps and known concatenation offsets. On Whisper-tiny, 34.9 hours of targeted correction audio used and only 1.6% of model parameters updated, raising long-gap mIoU from 38.7% to 95.0% and reducing mixed-gap out-of-domain AAS from 2752 ms to 223 ms while preserving CV-en MER at 41.3% (versus 524.2% for ordinary SFT decoder tuning).
CMU ML