Search papers, labs, and topics across Lattice.
This study integrates human visual attention into pathologist report generation for prostate histopathology by creating a multimodal dataset of annotated whole-slide images (WSIs) that captures pathologists' viewport trajectories and verbal descriptions. The authors fine-tune report generation models using an attention-alignment loss to ensure that model attention mirrors the distribution of human attention, leading to significant improvements in report accuracy and NLP metrics. Results demonstrate an average increase of 10.9% in NLP metrics and 19.3% in accuracy for clinically relevant report components, while also enhancing the interpretability of model outputs through aligned attention maps.
Aligning AI-generated reports with human pathologist attention boosts accuracy and interpretability, revealing the critical regions that inform cancer diagnoses.
The allocation of visual attention by pathologists during cancer diagnosis is a highly selective process that critically shapes the information extracted from whole-slide images (WSIs). Human attention helps medical imaging tasks such as classification and segmentation, and becomes a strong semantic cue for identifying diagnostically informative regions for report generation. In this paper, we introduce human attention into the training of pathologist report generation models. To this end, we collected a multimodal human-attention dataset of 121 prostate WSIs annotated with pathologists'multi-scale viewport trajectories synchronized with the pathologists'verbal descriptions and cursor movements for five clinically relevant components (e.g., Gleason patterns). Using this dataset, we finetune two report generation models with an attention-alignment loss that regularizes the model attention over image patches to match the distribution of pathologist attention. We evaluate our approach on prostate cancer report generation and visual question answering using two models with different internal attention mechanisms (i.e., how image tokens are integrated into the language decoder). Experiments show average gains of 10.9% on NLP-based metrics and 19.3% in accuracy across five clinically relevant report components. Further, model attention maps extracted at inference time, with minimal computational overhead, align more closely with pathologist attention, providing stronger visual support for the generated reports by highlighting the regions that most influence the output.
