Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the cardinality of optimal representations within the Information Bottleneck (IB) framework, specifically for binary sources. It proves that for binary $X$ and finite $Y$, the optimal representation $U$ is always binary, tightening the general bound of $|\mathcal{U}| \leq |\mathcal{X}|+1$ to $|\mathcal{U}| \leq |\mathcal{X}|$. This result is achieved through a combination of separating hyperplane arguments and an analysis of the concavity of the ratio of second derivatives of relevant entropy functions.
Binary sources allow for strictly smaller optimal representations in the Information Bottleneck than previously thought.
The information bottleneck (IB) seeks a representation $U$ of a source $X$ that retains as much information as possible about a target $Y$, subject to a constraint on $I(U;X)$. A classical argument shows that it suffices to consider representations with at most $|\mathcal{X}|+1$ symbols, and this bound is known to be tight whenever $|\mathcal{X}| \geq 3$. We show that the binary case behaves differently: if $X$ is binary and $Y$ is finite, then for every joint distribution of $(X,Y)$ and every rate constraint, the IB optimum is attained by a binary $U$. Hence the bound $|\mathcal{U}| \leq |\mathcal{X}|+1$ sharpens to $|\mathcal{U}| \leq |\mathcal{X}|$ for binary sources. The proof combines a separating hyperplane argument with the observation that, for a binary source, the ratio of the second derivatives of the two entropy functions involved is concave.