Search papers, labs, and topics across Lattice.
This paper introduces a two-stage framework for Computed Tomography Report Generation (CTRG) that leverages structure-wise image-text contrastive learning. In the first stage, learnable structure-specific visual queries extract observation tokens from CT images, which are then contrasted with structure-specific textual features from radiology reports, using a novel contrastive loss with soft pseudo targets and a dynamic negative queue. The second stage freezes the visual structure queries to select critical image patches and trains a text decoder for report generation, achieving state-of-the-art results on two public datasets.
Structure-aware contrastive learning with visual queries and soft pseudo-targets unlocks state-of-the-art performance in automated CT report generation.
Computed Tomography Report Generation (CTRG) aims to automate the clinical radiology reporting process, thereby reducing the workload of report writing and facilitating patient care. While deep learning approaches have achieved remarkable advances in X-ray report generation, their effectiveness may be limited in CTRG due to larger data volumes of CT images and more intricate details required to describe them. This work introduces a novel two-stage (structure- and report-learning) framework tailored for CTRG featuring effective structure-wise image-text contrasting. In the first stage, a set of learnable structure-specific visual queries observe corresponding structures in a CT image. The resulting observation tokens are contrasted with structure-specific textual features extracted from the accompanying radiology report with a structure-wise image-text contrastive loss. In addition, text-text similarity-based soft pseudo targets are proposed to mitigate the impact of false negatives, i.e., semantically identical image structures and texts from non-paired images and reports. Thus, the model learns structure-level semantic correspondences between CT images and reports. Further, a dynamic, diversity-enhanced negative queue is proposed to guide the network in learning to discriminate various abnormalities. In the second stage, the visual structure queries are frozen and used to select the critical image patch embeddings depicting each anatomical structure, minimizing distractions from irrelevant areas while reducing memory consumption. Also, a text decoder is added and trained for report generation.Our extensive experiments on two public datasets demonstrate that our framework establishes new state-of-the-art performance for CTRG in clinical efficiency, and its components are effective.
