Search papers, labs, and topics across Lattice.
This paper explores customizing smaller language models for domain-specific code generation using synthetic datasets across Python, Scikit-learn, and OpenCV domains. They compare few-shot prompting, retrieval-augmented generation (RAG), and LoRA fine-tuning, evaluating performance with benchmark and similarity-based metrics. LoRA fine-tuning achieves higher accuracy and domain alignment, while prompting-based methods offer cost-effective domain relevance improvements.
LoRA fine-tuning beats prompting and RAG for adapting smaller language models to domain-specific code generation tasks, offering a path to higher accuracy and domain alignment.
Large language models (LLMs) have demonstrated strong capabilities in generating executable code from natural language descriptions. However, general-purpose models often struggle in specialized programming contexts where domain-specific libraries, APIs, or conventions must be used. Customizing smaller open-source models offers a cost-effective alternative to relying on large proprietary systems. In this work, we investigate how smaller language models can be adapted for domain-specific code generation using synthetic datasets. We construct datasets of programming exercises across three domains within the Python ecosystem: general Python programming, Scikit-learn machine learning workflows, and OpenCV-based computer vision tasks. Using these datasets, we evaluate three customization strategies: few-shot prompting, retrieval-augmented generation (RAG), and parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA). Performance is evaluated using both benchmark-based metrics and similarity-based metrics that measure alignment with domain-specific code. Our results show that prompting-based approaches such as few-shot learning and RAG can improve domain relevance in a cost-effective manner, although their impact on benchmark accuracy is limited. In contrast, LoRA-based fine-tuning consistently achieves higher accuracy and stronger domain alignment across most tasks. These findings highlight practical trade-offs between flexibility, computational cost, and performance when adapting smaller language models for specialized programming tasks.
