Search papers, labs, and topics across Lattice.
This study introduces a novel method for generating a 3D image-text dataset specifically for brain oncology, utilizing 3D MRI scans of glioma and meningioma cases. By employing a collaborative system of multiple large language models (LLMs) to generate and validate reports, the researchers achieved significant improvements in both report generation and visual question answering tasks compared to existing 2D and 3D approaches. The resulting visual-language model (VLM) not only enhances report quality but also contributes to more accurate diagnoses and treatment strategies in brain oncology.
Collaborative LLMs can significantly enhance the accuracy and clarity of MRI report generation, outperforming traditional methods in brain oncology.
Recent advances in large language models (LLMs) and their extension to vision-language models (VLMs) have made it easier to combine text and images for tasks such as report generation. Existing VLMs in medicine typically focus on 2D images (chest X-rays), and their extension to 3D imaging has been difficult because of the lack of paired 3D imaging-text data. Thus, we introduce a new method for creating a 3D image-text dataset for brain oncology using 3D MRI scans of glioma and meningioma cases. We use a cooperative system in which several LLMs work together to generate and check reports, ensuring that they are accurate and clear. By leveraging the new 3D MRI-text dataset, we further build a VLM that converts MRI scans into tokens and aligns them with text instructions. Our VLM performed better in report generation and visual question answering tasks than other 2D and 3D methods. Our method not only improves the quality of reports but also helps with better diagnosis and treatment in brain oncology.
