Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
RefineMix, a novel framework for discrete diffusion models, addresses severe data scarcity by strategically incorporating out-of-distribution data at specific diffusion timesteps to enhance generalization without compromising the sampling distribution. This approach leverages the unique properties of discrete diffusion, where domain information is preserved in surviving tokens at high noise levels, while disjoint supports at low noise levels enable effective learning from both in-domain and out-of-distribution data. Experimental results demonstrate that RefineMix matches or surpasses in-domain finetuning and data mixing across various domain-shift scenarios, notably achieving significant improvements in protein sequence generation with minimal in-domain data.
Out-of-distribution data can dramatically boost discrete diffusion model performance under extreme data scarcity by strategically injecting it at low noise levels.
We introduce RefineMix, a framework for training discrete diffusion models under severe data scarcity, a common constraint in scientific applications. RefineMix uses out-of-distribution data at selected diffusion times to improve generalization without biasing the sampling distribution. Although this strategy has been explored in continuous diffusion, discrete diffusion presents a distinct challenge: unlike Gaussian noise, masking preserves domain information in surviving tokens, limiting the use of related data at high noise levels. At low noise levels, however, the domains effectively disjoint supports become an advantage, allowing the model to learn from both in-domain and out-of-distribution data without biasing the sampler. We formalize these intuitions and provide a theoretical analysis for the proposed method. Experimentally, across five domain-shift settings, RefineMix matches or outperforms in-domain finetuning and data mixing. For protein sequence generation, finetuning with just 197 in-domain examples nearly doubles the fraction of generated proteins that are simultaneously novel, foldable, and in-family compared to standard finetuning.