Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates the performance of 11 commercial and one open-source LLM on minimal-edit benchmarks for Ukrainian grammatical error correction (GEC), revealing that the best configuration, Gemini 3.1-Pro, achieves an F0.5 score of 69.22, narrowing the gap to the fine-tuned state-of-the-art model. The research highlights that while zero-shot prompting yields limited benefits, leveraging Ukrainian-specific minimal-edit prompts significantly enhances performance across all models. Error analysis uncovers five overcorrection patterns linked to unique Ukrainian linguistic features, providing insights for future improvements in GEC systems.
Ukrainian minimal-edit prompts can boost LLM performance, achieving near state-of-the-art results in grammatical error correction.
Fine-tuned Large Language Models (LLMs) dominate in Ukrainian grammatical error correction (GEC), while API-accessed LLMs remain nearly untested on minimal-edit benchmarks. We evaluate 11 commercial LLMs from four providers and one open-source Ukrainian model on the UNLP 2023 GEC-only benchmark, comparing zero-shot, few-shot, minimal-edits, and LLM-assisted prompt optimization strategies. Our best configuration (Gemini 3.1-Pro) reaches F0.5=69.22, closing over 90% of the gap to fine-tuned SOTA (F0.5=73.14). For zero-shot prompts, only Claude models benefit from Ukrainian instructions. However, the best overall results for all models use Ukrainian minimal-edits prompts, whose language-specific rules require Ukrainian to express precisely. LLM-assisted prompt optimization on top of minimal-edits + few-shot achieves the highest score. Detailed minimal-edits instructions yield the largest gains for punctuation and case errors but cause the model to abandon several low-frequency categories. Delving into error analysis, we identify five recurring overcorrection patterns tied to Ukrainian-specific linguistic phenomena. Code, prompts, and outputs are publicly available.