Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
OntoBook introduces a novel method for leveraging medical ontology structures to enhance the pretraining of encoder language models. By conducting random walks through ontology graphs to capture hierarchical and causal relationships, the method reformulates these walks into fluent textbook-style prose, which is then used to train ModernCamemBERT on dual objectives of masked language modeling and relation prediction. The approach yields significant performance gains on French medical coding benchmarks, with notable improvements of +2.5 micro-F1 on FRACCO and +8.0 micro-F1 on Distemist, underscoring the importance of alignment between training objectives.
Aligning training objectives can lead to a staggering 30-point performance drop if mismanaged, revealing the critical importance of cohesive data usage in medical language model pretraining.
We present OntoBook, a method that converts medical ontology structure into pretraining signal for encoder language models. Our approach has three stages: random walks through ontology graphs capture hierarchical and causal relations between medical codes, a large language model reformulates these walks into fluent textbook-style prose, and the resulting text is used to train ModernCamemBERT, a 149M-parameter French encoder, with two objectives on the same data: masked language modeling and relation prediction between code pairs. On three French medical coding benchmarks (FRACCO, Cantemist-FR, Distemist-FR), OntoBook achieves significant improvements over MLM-only pretraining, with +2.5 micro-F1 on FRACCO and +8.0 micro-F1 on Distemist. We find that alignment between objectives is necessary: misaligned training, where each task uses different data, causes a 30-point degradation. We release 1.3 million LLM-reformulated medical textbooks across three French ontologies (CIM-10, CCAM, ATC) and pretrained model checkpoints.