Search papers, labs, and topics across Lattice.
This work enhances code completion for the low-resource programming language Pharo by developing an end-to-end pipeline that includes data curation, continued pre-training, and fine-tuning of existing LLMs. The authors establish new benchmarks to assess the models' understanding of Pharo's syntax and their ability to complete masked code accurately. Results show that Pharo-specialized models significantly outperform both their original base checkpoints and larger code LLMs, enabling effective real-time in-IDE support for developers.
Pharo-specialized LLMs achieve superior code completion accuracy, outperforming larger models and paving the way for robust support in low-resource programming languages.
Large Language Models (LLMs) unlocked new possibilities in automated code writing, becoming the backbone of most code completion tools. While LLMs excel in mainstream languages, they often lack support for the so-called low-resource languages where training data is scarce. As a result, these languages lag behind in the quality of code completion tooling available to their communities. A concrete example is Pharo, a Smalltalk-inspired language whose IDE currently offers only single-token completion. In this work, we report on our experience bringing LLM-based code completion to Pharo. First, we describe an end-to-end pipeline that combines Pharo-specific data curation, continued pre-training and fine-tuning of open code LLMs. Second, we introduce a set of Pharo code completion benchmarks designed to evaluate whether models (i) learn Pharo's syntax and (ii) accurately complete masked Pharo code from real-world GitHub repositories. Third, we show empirically that Pharo-specialized models substantially outperform their original base checkpoints and also exceed the accuracy of substantially larger code LLMs on Pharo completion. Overall, our case study demonstrates the feasibility of bringing strong LLM-based code completion to low-resource programming languages, with models small enough to provide ``real-time''in-IDE support.
