Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of various domain adaptation strategies for large language models (LLMs) in the context of French medical question-answering (QA). By comparing continual pretraining (CPT), supervised fine-tuning (SFT), and their combination across different model families, sizes, and initialization types, the authors reveal nuanced trade-offs in performance. The findings indicate that while CPT+SFT often yields the best results for multiple-choice QA, SFT remains a strong, cost-effective default, and that CPT enhances overlap-based metrics for open-ended QA despite potential degradation in generation quality from SFT.
CPT+SFT may offer the best performance for multiple-choice QA, but SFT stands out as a cost-effective strategy that shouldn't be overlooked.
The development of large language models (LLMs) has led to an increased focus on their adaptation to specialized domains and languages, yet the effectiveness of domain adaptation strategies remains unclear. We present a study of medical domain adaptation using French medical question-answering (QA) as a case study. We compare continual pretraining (CPT), supervised fine-tuning (SFT), and their combination across three model families, multiple sizes, and three initialization types, explicitly disentangling adaptation effects from base model choice. We evaluate both multiple-choice (MCQA) and open-ended QA (OEQA) under greedy and constrained decoding using automatic metrics and LLM-as-a-Judge evaluation. For MCQA, CPT+SFT most often achieves the best scores, but gains over SFT are small and frequently not statistically significant, making SFT a strong and cost-effective default. For OEQA, CPT consistently improves overlap-based metrics, while SFT often degrades generation quality; instruction tuning and CPT+SFT are preferred by LLM-based evaluation. Cross-lingual experiments further show effective transfer from French adaptation to English benchmarks. Overall, we provide practical guidelines for selecting adaptation strategies under computational constraints.
