Search papers, labs, and topics across Lattice.
PathMem is introduced as a memory-centric multimodal framework for pathology MLLMs, designed to explicitly integrate structured pathology knowledge. It organizes this knowledge as a long-term memory (LTM) and employs a Memory Transformer to model the dynamic transition from LTM to working memory (WM) via multimodal memory activation and context-aware knowledge grounding. PathMem achieves state-of-the-art performance on pathology benchmarks, significantly improving WSI-Bench report generation and open-ended diagnosis compared to existing WSI-based models.
Pathology MLLMs can now better incorporate diagnostic standards during reasoning, thanks to a new memory architecture inspired by how human pathologists process information.
Computational pathology demands both visual pattern recognition and dynamic integration of structured domain knowledge, including taxonomy, grading criteria, and clinical evidence. In practice, diagnostic reasoning requires linking morphological evidence with formal diagnostic and grading criteria. Although multimodal large language models (MLLMs) demonstrate strong vision language reasoning capabilities, they lack explicit mechanisms for structured knowledge integration and interpretable memory control. As a result, existing models struggle to consistently incorporate pathology-specific diagnostic standards during reasoning. Inspired by the hierarchical memory process of human pathologists, we propose PathMem, a memory-centric multimodal framework for pathology MLLMs. PathMem organizes structured pathology knowledge as a long-term memory (LTM) and introduces a Memory Transformer that models the dynamic transition from LTM to working memory (WM) through multimodal memory activation and context-aware knowledge grounding, enabling context-aware memory refinement for downstream reasoning. PathMem achieves SOTA performance across benchmarks, improving WSI-Bench report generation (12.8% WSI-Precision, 10.1% WSI-Relevance) and open-ended diagnosis by 9.7% and 8.9% over prior WSI-based models.
Tsinghua AI