Search papers, labs, and topics across Lattice.
This study introduces SpeakPay, a voice-first digital wallet designed to enhance accessibility for visually impaired users in Nepal by enabling low-resource financial speech recognition. By fine-tuning the Whisper large-v2 model with LoRA on a newly created dataset, NepFinSpeech-403, the authors achieve a significant reduction in Word Error Rate from 129.95% to 42.58%, alongside a dramatic increase in Transaction Success Rate from 1.67% to 33.33%. The findings highlight the effectiveness of domain adaptation in improving practical outcomes in financial transactions, demonstrating that as few as 100 domain-specific utterances can substantially enhance model performance.
Achieving a 20x increase in transaction success rates for financial speech recognition in Nepali with just 300 training examples reveals the power of domain adaptation in low-resource settings.
Mobile payment applications in Nepal are graphically mediated and largely inaccessible to visually impaired users. This paper presents SpeakPay, a voice-first digital wallet, and documents the central technical contribution: a controlled study of domain adaptation for low-resource financial speech recognition. We introduce NepFinSpeech-403, a 403-utterance dataset of Nepali financial voice commands (send, load, and balance operations spanning 237 unique numerals), and fine-tune Whisper large-v2 with LoRA. On the held-out test set, the domain-adapted model reduces Word Error Rate from 129.95% (zero-shot baseline) to 42.58% --- a 67.2% relative reduction --- and improves Devanagari numeral recognition accuracy from 0.0% to 73.9%. We find that word-level metrics understate the practical task-level impact: domain adaptation improves the Transaction Success Rate from 1.67% to 33.33%, a roughly 20x gain. The improvement is consistent at the individual-utterance level (sign test, $p<10^{-17}$) and across all command types. A data efficiency analysis shows that as few as 100 domain-specific utterances are sufficient to halve the zero-shot WER, with performance plateauing around 300 examples. Error analysis reveals systematic numeral confusion patterns (zero insertion/deletion, prefix hallucination) that account for the majority of remaining transaction failures. The trained system is deployed as a publicly accessible voice-first web application. All code, dataset, model weights, and this paper are released at https://github.com/subedibiraj/speakpay.
