Search papers, labs, and topics across Lattice.
This study explores parameter-efficient adaptation strategies for generating volumetric CT reports using large language models (LLMs), addressing challenges like overfitting and clinical hallucination. The authors introduce RAD3D-Prefix, a lightweight framework that integrates image embeddings with diagnostic classification logits while keeping the LLM frozen, thus minimizing the need for extensive parameter training. Results indicate that freezing larger LLMs and training only lightweight layers yields better performance and generalization across various metrics compared to fully fine-tuned models, particularly for smaller LLMs.
Freezing large LLMs while using a lightweight diagnostic-prior framework can significantly enhance performance in 3D CT report generation with minimal training overhead.
Recent advances in multimodal learning, including large language models (LLMs) and vision-language models (VLMs), have demonstrated strong adaptability to natural images. However, extending their use to the medical domain, particularly for volumetric (3D) images, is challenging due to high computational complexity, volumetric dependencies and the semantic gap between visual features and clinical terminology. Naively fine-tuning LLMs on limited medical data often leads to overfitting and clinical hallucination, where linguistic fluency is prioritized over clinical factuality. In this study, we investigate parameter-efficient adaptation strategies for volumetric CT report generation and introduce RAD3D-Prefix, a lightweight diagnostic-prior conditioning framework that minimizes the need for extensive parameter training. This module integrates image embeddings with multi-label diagnostic classification logits, preserving critical clinical details while bridging the semantic gap. By keeping the LLM frozen, our method requires minimal trainable parameters and mitigates the risk of overfitting on small, domain-specific datasets. Through a systematic study spanning LLMs from 96.1M to 1.6B parameters, we find that fine-tuning is most beneficial for smaller LLMs, whereas freezing larger (~1B+ LLMs and training only lightweight projection layers provides a superior trade-off between performance, generalization, and computational efficiency. Across multiple automatic metrics and a clinical reader study, RAD3D-Prefix outperforms comparable parameter-efficient baselines and demonstrates strong out-of-domain generalization while using substantially fewer trainable parameters than fully fine-tuned alternatives.
