Search papers, labs, and topics across Lattice.
This paper introduces CSSEL-P2P, a novel approach to simultaneous speech translation (SimulST) that leverages fixed-length chunks and a rewind-based committed prefix for cumulative streaming decoding, avoiding the need for architectural modifications. By employing teacher-labeled prefix-to-prefix (P2P) targets during fine-tuning, the method effectively addresses the challenges of limited context and cross-lingual reordering in decoder-only LLMs. The results show a significant improvement in streaming quality, with a +1.54 COMETKiwi score over the baseline while maintaining comparable latency, indicating a promising direction for SimulST without complex architectural changes.
Achieving superior simultaneous speech translation quality without altering LLM architecture could redefine efficiency in real-time translation tasks.
Simultaneous speech translation (SimulST) requires incremental translation under strict latency constraints, yet remains challenging for decoder-only LLM systems due to limited context and cross-lingual reordering. Recent approaches often introduce architectural changes or explicit read/write policies to control output timing, which can be brittle in conversational speech where segmentation boundaries are ambiguous. We present a simple data-driven alternative: fixed-length chunks for cumulative streaming decoding with a rewind-based committed prefix, and teacher-labeled prefix-to-prefix (P2P) targets with bounded waiting for fine-tuning, yielding CSSEL-P2P, where CSSEL is our proposed chunked streaming speech encoder LLM. In our in-house conversational speech evaluation, CSSEL-P2P improves streaming quality by +1.54 COMETKiwi over the CSSEL streaming baseline at comparable latency (+0.15s Average Lagging), suggesting effective SimulST without architectural changes via P2P supervision.
