Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper evaluates the effectiveness of large language models (LLMs) like OpenAI's gpt-oss-120b and GPT-5.4 in generating machine-processable grammars using newly developed Cantonese ParGram resources as gold standards. The study finds that while GPT-5.4 significantly outperforms gpt-oss-120b, both models struggle with coordinating complex formal constraints, indicating their limitations in grammar engineering. Ultimately, the findings suggest that while LLMs can assist in the initial stages of grammar development, human expertise remains crucial for thorough analysis and refinement.
GPT-5.4 outshines gpt-oss-120b in generating grammars, but both models falter in managing complex formal constraints, highlighting the need for human oversight.
This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradigm. Using Cantonese ParGram resources as gold standards, with corresponding English baselines, we investigate whether OpenAI's gpt-oss-120b and GPT-5.4 can generate machine-processable grammars from sentences and target formal structures under systematically varied prompting conditions. GPT-5.4 outperformed gpt-oss-120b, while grammars generated from target formal structures generally outperformed those generated from sentences. Although both models could generate locally plausible phrase-structure rules, lexical entries, and templates, they often struggled to coordinate interacting formal constraints, especially in multi-construction settings. The results characterize both the capabilities and limitations of current LLMs for potential integration into AI-assisted expert workflows: LLMs may support intermediate stages of grammar development, but human linguistic expertise remains central to analysis, validation, and refinement. The study also contributes new Cantonese symbolic grammatical resources.