Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a benchmark and reference system for live captioning of Sikh Kirtan, addressing the unique challenges of closed-vocabulary transcription where accuracy is paramount due to religious considerations. The authors formalize the task into a structured framework and release a dataset comprising 12 evaluation cases, achieving a notable 57.9% frame accuracy with their fine-tuned IndicConformer model. By contrasting their approach with traditional ASR metrics, they highlight the need for specialized evaluation criteria that align with the task's requirements.
Achieving 57.9% frame accuracy in live Kirtan captioning underscores the critical need for tailored metrics in closed-vocabulary transcription tasks.
We present a benchmark and reference system for live captioning of Sikh Kirtan - the continuous, sung recitation of verses from the Sri Guru Granth Sahib Ji (SGGS). Unlike open-vocabulary lyrics transcription, Kirtan captioning is a closed-vocabulary problem: every displayed line must be an exact, word-for-word line from the canonical scripture, because displaying misspelled Gurmukhi is considered religiously inappropriate. We formalize the task as predicting, at every time t, a pair (shabad_id, line_idx) or null, and organize the problem space into a 2x2 matrix along two orthogonal axes: live vs. offline (causal vs. full-audio access) and blind vs. oracle (shabad identity discovered vs. given). We release v1 of the benchmark - 4 hand-annotated Kirtan recordings x 3 cold-start offsets = 12 evaluation cases, ~57 minutes of scored audio - together with a scorer that computes frame accuracy at 1s resolution over a scored region, with a 1s collar and gap-tolerant scoring at segment boundaries. We describe a reference system (fine-tuned 120M IndicConformer ->fuzzy matcher ->state machine; INT8 ONNX; RTF ~0.05 on one Apple Silicon core) that achieves 57.9% overall frame accuracy across all 12 cases (10/12 correct shabad locks) on the hardest variant (live x blind). We compare against three trivial baselines (empty, shifted-5s, perfect) and discuss why standard ASR metrics (WER/CER) measure transcription accuracy rather than the display accuracy this task requires. The benchmark, reference system, and a live deployment are released under permissive licenses to facilitate further improvements.