Search papers, labs, and topics across Lattice.
This paper introduces CANDLE, a lightweight system for character-level Arabic noise deduplication that effectively distinguishes between correct spelling and informal character elongation without relying on traditional linguistic resources. Utilizing a novel application of Connectionist Temporal Classification (CTC), CANDLE frames the normalization task as a sequence alignment problem, achieving a Sentence Error Rate (SER) as low as 5.37% across various benchmarks. Additionally, the model's distillation into a more efficient 2-layer version reduces inference overhead while maintaining performance, leading to significant improvements in tokenizer efficiency for Arabic LLMs.
CANDLE reduces Arabic text noise with a 5.37% error rate while slashing inference costs by optimizing tokenizer efficiency.
Handling repeated characters in text can be tricky, since they can represent either the correct spelling of a word or informal character elongation often seen in social media posts. We present CANDLE, a lightweight system for character-level Arabic noise deduplication that addresses this challenge without relying on handcrafted rules, dictionaries, or morphological analyzers. At the heart of CANDLE is a novel application of Connectionist Temporal Classification (CTC) to this task, a formulation not previously explored for character deduplication, which frames normalization as a sequence alignment problem over a character-based encoder. Evaluated on three benchmarks spanning clean newspaper, manually curated ambiguous cases, and real-world social media text, the CTC model achieves a Sentence Error Rate (SER) as low as $5.37\%$ and consistently outperforms a classification-based baseline by a large margin. To reduce inference overhead, we distill the 6-layer CTC model into a 2-layer student, achieving a $3\times$ depth reduction with minimal performance degradation. Beyond deduplication accuracy, normalization yields a practical downstream benefit: a relative reduction in tokenizer fertility of up to $12.8\%$ across a diverse set of Arabic LLM tokenizers, directly lowering inference costs and improving context window utilization. We release all code and models publicly to support reproducibility and advance future research\footnote{https://github.com/abjadai/candle}.
