Search papers, labs, and topics across Lattice.
This paper introduces HounsWorld, a multimodal world model that integrates volumetric medical imaging and clinical language to infer a patient's underlying condition through a shared latent state. By employing Joint Understanding-Generation Learning, HounsWorld effectively addresses three interrelated tasks: state readout, reconstruction, and simulation, utilizing a shared transformer architecture. The model demonstrates superior performance across these tasks while enhancing CT understanding through structured completion, highlighting its potential for advancing clinical intelligence in patient care.
HounsWorld achieves state-of-the-art performance in multimodal patient-state inference by seamlessly integrating CT imaging and clinical language, transforming how we interpret medical data.
Clinical intelligence requires estimating a patient's underlying condition from incomplete observations rather than learning isolated mappings from scans to answers. Volumetric medical images provide dense observations of anatomy, attenuation, and lesions, whereas clinical language provides sparse but complementary semantic observations. We formulate CT-centered intelligence as inference over a shared latent patient state, under which readout, reconstruction, and simulation all become state-dependent prediction problems. To operationalize this view, we introduce HounsBench, a computed tomography (CT) centric patient-state benchmark that unifies these three task families with patient-disjoint splits and per-family metrics, and HounsWorld, a 3B multimodal world model that treats volumetric scans and language as observations of the shared state through Joint Understanding-Generation Learning. A shared transformer forms an implicit patient-state estimate and supports three outputs: query-conditioned answers that read out the state, reports and captions that reconstruct it in language, and condition-specific CT volumes for low-dose denoising, virtual contrast enhancement, and anatomy-constrained text-and-mask-to-volume generation. Zero-initialized CT adapters preserve pretrained multimodal mappings, while condition-explicit Hounsfield-unit window sampling exposes clinically meaningful density observations. HounsWorld shows strong performance across all three task families while consistently improving CT understanding through clinically structured completion. Our project is available at https://github.com/byhwhite/HounsWorld.git
DAMO