Search papers, labs, and topics across Lattice.
This paper introduces COBOL-Coder, a domain-adapted LLM fine-tuned for COBOL code generation and translation, built using a novel data curation pipeline that combines compiler-guided validation and similarity-based filtering. Evaluated on code generation (COBOLEval, COBOLCodeBench) and translation (COBOL-JavaTrans), COBOL-Coder significantly outperforms general-purpose LLMs like GPT-4o and open-source models, achieving up to 73.95% compilation success and 49.33 Pass-1 on COBOLEval. User studies with experienced COBOL developers confirm COBOL-Coder's superior COBOL awareness, program structure, and alignment with enterprise practices compared to general-purpose LLMs.
GPT-4o can't even compile COBOL, but COBOL-Coder, a new domain-adapted LLM, achieves a 74% compilation success rate, finally bringing modern AI to legacy mainframe systems.
COBOL remains a critical language for mainframe systems, yet existing large language models (LLMs) struggle to generate and translate COBOL code correctly. This paper reports our experience in developing and evaluating domain-adapted LLMs for COBOL and mainframe software engineering. We introduce (1) an automated data curation pipeline that combines compiler-guided validation with multi-stage similarity-based filtering to construct high-quality COBOL training data, and (2) COBOL-Coder, a COBOL-specialized LLM fine-tuned on the curated COBOL domain data. We evaluate COBOL-Coder on two tasks: code generation (on COBOLEval and COBOLCodeBench) and code translation (on COBOL-JavaTrans, our proposed benchmark for bidirectional COBOL-Java translation). In our experiments, COBOL-Coder achieves up to a 73.95 percent compilation success rate and 49.33 Pass-1 on COBOLEval, compared to 41.8 percent and 16.4 for GPT-4o, while most open-source baselines (e.g., CodeGemma, CodeLlama, StarCoder2) fail to produce compilable programs. For Java-to-COBOL translation, COBOL-Coder reaches 34.93 Pass-1, whereas general-purpose LLMs achieve near-zero scores. To assess the usability of LLM-generated code in real-world settings, we conduct a survey with experienced COBOL developers. Participants consistently report that COBOL-Coder exhibits stronger COBOL awareness, has more reliable program structure, and is better aligned with enterprise practices than general-purpose LLMs.
