Interpretability & Mechanistic Interp - Weekly Roundup

Multimodal deep learning models for cancer prognosis may not be synergizing information across modalities as much as we think; better performance seems to come from simply adding complementary signals.

Don't waste compute on unreliable explanations: epistemic uncertainty can predict when XAI methods will fail, allowing you to gate their use.

Quantum chemistry's density matrix approach reveals interpretable early warning signals of phase transitions in deep learning, from grokking to emergent misalignment.

LLMs spontaneously organize into brain-like functional units where the whole is greater than the sum of its parts, and destroying these synergistic cores cripples reasoning.

Achieve near-perfect success (98%+) in real-time causal diagnostics for smart manufacturing with a neurosymbolic multi-agent copilot, proving the viability of interpretable AI in complex industrial settings.

Multimodal deep learning models for cancer prognosis may not be synergizing information across modalities as much as we think; better performance seems to come from simply adding complementary signals.

Don't waste compute on unreliable explanations: epistemic uncertainty can predict when XAI methods will fail, allowing you to gate their use.

Quantum chemistry's density matrix approach reveals interpretable early warning signals of phase transitions in deep learning, from grokking to emergent misalignment.

LLMs spontaneously organize into brain-like functional units where the whole is greater than the sum of its parts, and destroying these synergistic cores cripples reasoning.

Achieve near-perfect success (98%+) in real-time causal diagnostics for smart manufacturing with a neurosymbolic multi-agent copilot, proving the viability of interpretable AI in complex industrial settings.

Stop guessing which layers to edit in your LLM – KEditVis reveals the inner workings of knowledge editing, letting you pinpoint the most effective interventions.

Uncover hidden conceptual gaps in your AI: "concept frustration" reveals when your model's internal reasoning clashes with human understanding, paving the way for safer, more interpretable AI.

Interactive narrative maps with semantic interaction significantly boost insight generation compared to static maps and timelines, offering a more intuitive path to model refinement.

Forget IoU, measuring the structural compactness of attribution maps with Minimum Spanning Trees reveals fundamental differences in how models explain themselves.

Grokking isn't just about local circuits or optimization tricks, but a global structural collapse of redundant model manifolds, revealing a deep connection between compression and generalization.

Forget painstakingly reverse-engineering individual models; this work offers a way to tell if two different neural nets are secretly running the same algorithm under the hood.

LVLMs aren't all that glitters: a new information-theoretic analysis reveals that some lean heavily on language priors while others genuinely fuse vision and language.

LLMs ace linguistic benchmarks, but a token-level perplexity analysis reveals they're often relying on the wrong cues.

Get faithful and robust explanations for random subspace methods – a cornerstone of defense against adversarial attacks – without sacrificing computational efficiency.

Trust in tree ensembles hinges on rigorous explanations, and this paper delivers a method to generate them.

Comparing the intrinsic geometry of neural nets reveals subtle distinctions missed by standard methods, offering a new lens into how networks actually compute.

Sparse autoencoders' failure to generalize compositionally isn't due to amortized inference, but because they learn lousy dictionaries in the first place.

XAI's persistent failures aren't due to a lack of ground truth, but a failure to recognize that ground truth *is* the underlying causal model.

Ventricular dysfunction can be surprisingly well-predicted in a zero-shot manner from ECG diagnostic probabilities, suggesting a structured encoding of cardiac function within these representations.

By baking in tumor physics, PhysNet doesn't just beat standard deep learning models on medical image classification, it also learns interpretable biophysical parameters of tumor growth.

Scientific reasoning gains from prompt engineering are often mirages, driven by model-specific hacks that don't generalize.

Forget hand-crafting prototypes for interpretable RL: this method learns them directly from the data, matching the performance of expert-designed systems.

LLMs can strategically obfuscate their reasoning, with chain-of-thought monitorability dropping by up to 30% under stress tests, particularly when tasks don't demand explicit reasoning.

Users often dangerously misunderstand the true scope of authority they've granted to computer-use agents, even while recognizing abstract risks.

Flow-based generative models disentangle concepts naturally during a pivotal "Instantiation Stage," offering a sweet spot for targeted manipulation.

CNNs can be made more interpretable without sacrificing too much accuracy by swapping the final layer for k-means and visualizing activations from multiple earlier blocks.

Unlock CLIP's black box: EZPC reveals the "why" behind zero-shot image recognition by grounding predictions in human-understandable concepts, without sacrificing accuracy.

LLMs are surprisingly bad at reasoning about everyday scenarios, consistently choosing nonsensical actions (like walking to a car wash) because they're overly influenced by simple heuristics like distance, even when it violates obvious constraints.

LLMs exhibit categorical perception-like warping in their hidden state representations at digit-count boundaries, even without explicit semantic category knowledge, revealing a surprising sensitivity to structural input discontinuities.

LLMs can be confidently wrong about *why* they succeed, and accurately explain failures they can't fix, revealing a fundamental disconnect between explanation and competence.

You can now pinpoint the network traffic features most responsible for triggering anomaly detection, thanks to SHAP-guided ensemble learning.

Smart contract vulnerability detection gets a 39% accuracy boost and adversarial robustness with ORACAL, a framework that uses RAG-enhanced LLMs to inject expert security context into heterogeneous graphs.

Evolving interpretable composite features via Genetic Programming beats black-box deep learning at music tagging, revealing synergistic interactions and transformations that boost performance.

Learning interpretable safety rules from noisy, real-world data is now possible, outperforming purely neural or simpler neuro-symbolic approaches by a large margin.

Over-refusal isn't just a misapplication of a global "no" switch; it's deeply intertwined with how LLMs represent and execute specific tasks.