Search papers, labs, and topics across Lattice.
This paper establishes a new stochastic separability theorem for embedding manifolds of distinct object categories, addressing a gap in theoretical validation for previously observed low-dimensional characteristics in high-dimensional neural spaces. By employing a two-layer measure concentration analysis technique, the authors derive inequalities that demonstrate the conditions under which samples from two datasets become linearly separable with high probability. The findings reveal a non-singularity condition that, when satisfied, allows for effective representation learning in deep networks, enhancing our understanding of geometric and statistical properties in neural representations.
Stochastic separability of embedding manifolds reveals that distinct object categories can be linearly separated in high-dimensional spaces, provided certain conditions are met.
Neurobiological studies and representation learning have observed that representations of objects belonging to the same category in high-dimensional neural spaces exhibit low-dimensional object manifold characteristics, and different object manifolds are linearly separable in these neural spaces. However, these experimentally observed phenomena lack rigorous theoretical validation to date. This paper proposes a new stochastic separability theorem for embedding manifolds of two different object categories. First, we establish a projection measure concentration theorem for embedding manifolds under general conditions. We develop a new two-layer measure concentration analysis technique, which unifies two estimation bounds via the law of total expectation to derive measure concentration inequalities. Based on the measure concentration theorem, we further prove a stochastic separability theorem for embedding manifolds of two different object categories. If two datasets have distinct means and bounded total variances, their samples become linearly separable with high probability, provided that the projection direction satisfies a non-singularity condition. The main contributions of this paper are twofold: 1. We prove the projection concentration properties of embedding manifolds in high-dimensional spaces by using two-lawyer tail-bound inequalities. 2. We identify a non-singularity condition for the stochastic separability between embedding manifolds, and rigorously prove the stochastic projection separability theorem. The theorem not only uncovers geometric and statistical properties of the object embedding manifolds, but also provides a novel mechanism for representation learning in deep networks.
