Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the concept of conceptual separation in neural networks, focusing on how well-defined and ambiguous concepts are represented internally in Convolutional Neural Networks (CNNs) and Large Language Models (LLMs). Through geometric and distributional analysis of internal activations, the authors find that while CNNs maintain coherent representations for familiar concepts, this coherence diminishes for unseen concepts and under domain shifts. In contrast, LLMs exhibit clear separation for distinct domains, with related subdomains clustering together, revealing that conceptual separation offers insights into model robustness that output metrics alone cannot provide.
CNNs struggle with unseen concepts, while LLMs show surprising clustering of related topics, challenging our understanding of internal representations in neural architectures.
Neural networks are increasingly employed to identify both well-defined and ambiguous concepts, yet output-level metrics reveal little about how those concepts are represented internally. Our study asks if these networks exhibit \textit{conceptual separation}: if examples of the same concept form coherent representations, and whether related concepts lie closer together in the representation space. We examine this conceptual organisation in Convolutional Neural Networks (CNNs) and Large Language Models (LLMs) through geometric and distributional analysis of their internal activations. In CNNs, familiar ImageNet concepts form coherent and semantically ordered representations, while this coherence weakens for unseen concepts and suffers within-class domain shift. In LLMs, clearly distinct domains remain well separated, related subdomains move closer together, and the distinction between ambiguous topics collapses at both the mean and covariance level. These results suggest that conceptual separation can reveal structure that output accuracy alone cannot, and may serve as a useful diagnostic of how robustly a model represents the concepts it is asked to identify. Code and data available on \href{https://github.com/JaeeRoshniCapstoneProject/Are-You-Thinking-What-I-m-Thinking-Examining-Conceptual-Separation-in-Neural-Architectures}{GitHub}.