Search papers, labs, and topics across Lattice.
This paper introduces a model-agnostic framework enabling frozen LLMs and VLMs to learn from deployment experience by augmenting them with a Skill for reasoning/tool use, a Knowledge Memory for facts, and a Multimodal Knowledge Base for visual examples. This approach addresses limitations of traditional fine-tuning and parameter-free methods by validating updates against new cases to prevent overfitting and degradation. The framework significantly improves performance on medical AI tasks by up to 34.2% across various benchmarks and models, demonstrating generalization and transferability.
Frozen medical AI models can now continuously learn and improve from real-world deployment experience without requiring weight updates or risking catastrophic forgetting.
Large language models (LLMs) and vision-language models (VLMs) are usually frozen after deployment, so they do not learn from the cases they solve. This is especially concerning in medicine, where new clinical evidence, updated guidelines, and new therapies can change established practice. Fine-tuning can update the model, but it requires access to model weights and additional training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by saving experience only as text. To address these limitations, we present a model-agnostic framework that allows frozen LLMs and VLMs to learn from deployment experience through three forms of external expertise: a Skill that guides reasoning and tool use, a Knowledge Memory that stores reliable facts supported by earlier cases or trusted external evidence, and a Multimodal Knowledge Base that keeps visual examples and guides the model to relate each retrieved case to the current image. Instead of relying on a fixed validation set, a validation strategy keeps an update only if it helps on new cases without degrading performance on earlier ones. Across six benchmarks covering clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning, and with four open-weight and closed-source base models, our framework improves performance during online deployment by up to 34.2% over the base model on medical tasks, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.
