Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors formulate the Chernoff bottleneck to directly optimize downstream binary hypothesis testing error exponents under a mutual information rate constraint, addressing the classical Information Bottleneck's disconnect from operational decision error. They prove that the optimal error exponent $C(R)$ strictly increases up to the entropy of likelihood-ratio-merged symbols $H(V)$, is not necessarily concave, and requires an alphabet size of at most $|V|+1$. Leveraging an alternating optimization scheme combining a generalized Blahut–Arimoto update with a 1D Chernoff parameter solver, the approach preserves 90% of the uncompressed error exponent on 20 Newsgroups classification while compressing word representations to just 17% entropy.
Optimizing representation bottlenecks directly for operational decision error rather than mutual information breaks the classical concavity guarantee, proving that features can be compressed by 83% while retaining 90% of the optimal classification exponent.
The classical information bottleneck (IB) measures the relevance of a representation $U$ of $X$ to a target $Y$ by $I(U;Y)$, which does not directly characterize the error of downstream decisions. For a binary hypothesis $Y$ inferred from many separately encoded observations, the optimal error exponent is the Chernoff information between the two conditional distributions of $U$ given $Y$. We study the mutual information constrained Chernoff bottleneck, which seeks an encoder that maximizes this Chernoff information subject to a rate constraint $I(U;X) \leq R$. We show that its optimal value $C(R)$ increases strictly up to $R = H(V)$, where $V$ merges the symbols of $X$ with equal likelihood ratio, remains at the uncompressed exponent beyond, and, unlike the IB curve, need not be concave. We further show that $k+1$ outputs suffice to attain $C(R)$, where $k$ is the cardinality of $V$. We propose an alternating algorithm that updates the encoder via a generalized Blahut--Arimoto algorithm and the Chernoff parameter $s$ via a nonlinear equation, and prove that its iterates remain feasible, with nondecreasing and convergent Chernoff information. Numerical experiments confirm the theory, and on real topic-detection data from the 20 Newsgroups corpus, compressing each word to only $17\%$ of its entropy retains $90\%$ of the error exponent and nearly the accuracy of the uncompressed classifier.