Search papers, labs, and topics across Lattice.
The paper introduces AVERT, a novel approach to spoken dialogue state tracking that combines cross-turn agreement with an audio-conditioned verifier to address common errors in slot-value recovery. By employing three targeted operators鈥攙ote, add, and swap鈥擜VERT effectively resolves inconsistencies, omissions, and unsupported values in dialogue states. The method achieves a Joint Goal Accuracy (JGA) of 40.13 on the SpokenWOZ dataset, surpassing traditional text editing methods and demonstrating competitive performance against larger end-to-end systems without requiring retraining.
AVERT achieves a new state-of-the-art in spoken dialogue state tracking by leveraging audio verification to correct persistent ASR errors, outperforming conventional text-based methods.
Spoken dialogue state tracking recovers slot-value pairs from speech, where ASR errors concentrate in entity values and persist across turns, making it both a generation and an editing problem. A strong per-turn text editor corrects much of this but, operating on the transcript alone, leaves three recoverable errors: a value predicted inconsistently across turns, an omitted slot, and a value the audio does not support. We present AVERT, which scores each candidate value by combining cross-turn agreement with a trained audio-conditioned verifier and resolves the three error types with three operators, vote, add, and swap, each restricted to the slots where its error is common. On SpokenWOZ, a base speech-LLM reaches 33.04 JGA, a text editor 38.34, and AVERT 40.13, without retraining either. This is in the range of a 1B end-to-end system that consumes the full spoken history (39.32), though AVERT uses two 1B decoders rather than one. The audio verifier contributes a statistically significant gain, and restricting each operator to a selected slot subset matters: removing it lets unrestricted voting overwrite correct categorical values and fall below the editor.
