Search papers, labs, and topics across Lattice.
The authors introduce MedCL-Bench, a new benchmark for evaluating continual learning (CL) strategies in biomedical NLP, addressing the lack of standardized evaluation for catastrophic forgetting in this domain. They evaluated eleven CL strategies across ten datasets and eight task orders, measuring retention, transfer, and GPU-hour cost. Their results show that direct sequential fine-tuning leads to catastrophic forgetting, and different CL methods offer distinct retention-compute trade-offs, with parameter isolation being the most efficient and replay offering strong protection at a higher cost.
Biomedical language models suffer severe catastrophic forgetting when sequentially updated, but parameter isolation offers the best retention per GPU-hour, revealing a crucial efficiency-stability tradeoff.
Medical language models must be updated as evidence and terminology evolve, yet sequential updating can trigger catastrophic forgetting. Although biomedical NLP has many static benchmarks, no unified, task-diverse benchmark exists for evaluating continual learning under standardized protocols, robustness to task order and compute-aware reporting. We introduce MedCL-Bench, which streams ten biomedical NLP datasets spanning five task families and evaluates eleven continual learning strategies across eight task orders, reporting retention, transfer, and GPU-hour cost. Across backbones and task orders, direct sequential fine-tuning on incoming tasks induces catastrophic forgetting, causing update-induced performance regressions on prior tasks. Continual learning methods occupy distinct retention-compute frontiers: parameter-isolation provides the best retention per GPU-hour, replay offers strong protection at higher cost, and regularization yields limited benefit. Forgetting is task-dependent, with multi-label topic classification most vulnerable and constrained-output tasks more robust. MedCL-Bench provides a reproducible framework for auditing model updates before deployment.
