Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces InfoSEDD, a novel discrete diffusion-based framework for estimating information-theoretic measures like Mutual Information on discrete data, bypassing the need for continuous embeddings. Leveraging Continuous Time Markov Chains, InfoSEDD offers a lightweight and scalable approach to compute Kullback-Leibler divergences, directly addressing limitations of existing methods that struggle with high dimensionality and require extensive engineering. Experiments demonstrate InfoSEDD's superior performance over "embedding trick" alternatives across diverse applications including genetic motif discovery and semantic-aware model selection.
Discrete diffusion directly estimates mutual information, outperforming continuous embedding methods and unlocking robust information-theoretic analysis for complex discrete datasets.
Information-theoretic measures, such as Mutual Information (MI), play a crucial role in understanding non-linear relationships between random variables and are widely used across scientific disciplines. Yet, their use on real-world discrete data remains challenging. Existing methods typically rely on embedding discrete data into a continuous space and apply neural estimators originally designed for continuous distributions. This process requires careful engineering for both the embedding model and estimator architecture, but suffers from issues related to high data dimensionality. In this work, we introduce InfoSEDD, a discrete diffusion-based approach that bridges information-theoretic estimation and generative modeling such that they can be used to compute Kullback-Leibler divergences. Backed by Continuous Time Markov Chains theory principles, the design of InfoSEDD is lightweight and scalable and allows seamless integration with pretrained models. We showcase the versatility of our approach through applications on motif discovery in genetic promoter data, semantic-aware model selection in text summarization, and entropy estimation in Ising models. Finally, we construct consistency tests on real-world textual and genomics data. Our experiments demonstrate that InfoSEDD outperforms alternatives that rely on the"embedding trick". Our results position InfoSEDD as a robust and scalable tool for information-theoretic analysis of discrete data.