Search papers, labs, and topics across Lattice.
The paper investigates supervised fine-tuning (SFT) of open-source LLMs (Qwen3-4B, Llama-3.1-8B, and Qwen3-32B) to explain C compiler errors to CS1/2 students, addressing limitations of proprietary LLMs in educational settings. They created a dataset of 40,000 C compiler error/explanation pairs and evaluated model performance using expert human reviewers and an LLM-as-judge ensemble. Results show SFT significantly improves smaller open-source models' ability to provide helpful explanations, making them a viable alternative to larger, proprietary models for pedagogical tools.
Fine-tuning smaller, open-source LLMs on a targeted dataset can rival the performance of larger, proprietary models for explaining compiler errors, offering a cost-effective and privacy-preserving solution for educational tools.
Cryptic compiler error messages continue to present a significant barrier for novice programmers, especially in foundational languages like C. Although large language models (LLMs) can generate accurate and comprehensible error explanations, their computational requirements, propensity for over-assistance, and privacy concerns constrain their suitability for widespread adoption in educational tools. This work investigates how Supervised Fine-Tuning (SFT) can enhance the performance of smaller, open-source models when explaining C compiler errors to students in introductory programming courses (CS1/2). We derive a training dataset of 40,000 input-output pairs from CS1/2 student C compiler errors to fine-tune three open-source models: Qwen3-4B, Llama-3.1-8B, and Qwen3-32B. Model performance was assessed through a dual evaluation framework involving expert human reviewers and a large-scale automated analysis of 8,000 responses using an ensemble of models as judges. Our results indicate that SFT significantly improves both expert and LLM-as-judge ratings in smaller open-source models, with reduced gains in the larger model. We analyse the trade-offs between model size and quality, and validate LLM-as-judge by demonstrating inter-rater agreement with experts. Our findings demonstrate that fine-tuning smaller models on high-quality data is a viable strategy for creating specialised pedagogical tools. We provide a replicable methodology for enabling broader access to advanced AI capabilities within educational contexts, especially with smaller, economical models.
