Interpretability & Mechanistic Interp - Weekly Roundup

Most observed IPv6 Fragment-EH patterns are simple within-packet co-occurrences, not complex temporal network rules, a finding rigorously established by a novel negative-control protocol.

Spurious forgetting in LLMs is a reversible access loss, not permanent knowledge erasure, and can be mitigated by decoupling shared representational shifts from fact-specific updates.

Generalization-heavy models are demonstrably more fragile during unlearning, suffering significantly higher performance degradation on retained data.

Conformal prediction set size reduction is formally justified as an information gain metric, bridging uncertainty quantification with classical information theory.

Diffusion models reveal feature generation order quantitatively, exposing architectural differences and suggesting ordered training benefits.

Spurious forgetting in LLMs is a reversible access loss, not permanent knowledge erasure, and can be mitigated by decoupling shared representational shifts from fact-specific updates.

Generalization-heavy models are demonstrably more fragile during unlearning, suffering significantly higher performance degradation on retained data.

Conformal prediction set size reduction is formally justified as an information gain metric, bridging uncertainty quantification with classical information theory.

Diffusion models reveal feature generation order quantitatively, exposing architectural differences and suggesting ordered training benefits.

Training-free TFMs can outperform trained GCNs in calibration for node classification, and a novel homophily-gated approach significantly shrinks prediction sets without sacrificing validity.

Activation steering debiasing methods reduce LLM bias not by correcting underlying preferences, but by making the model less confident and more likely to abstain from answering.

Discovering task-specific, compact concept vocabularies from unsupervised pools unlocks inspectable AI without the need for manual annotations or massive, static concept sets.

A novel explainable power modeling framework achieves state-of-the-art prediction accuracy and generalization on diverse workloads with negligible hardware overhead, unlike opaque black-box models.

Unfrozen video-language models implicitly know when they've seen enough evidence, a signal that can be extracted and used to boost answer accuracy by nearly 10% without retraining.

Counterfactual explanations can now achieve near-perfect numerical sparsity and proximity by treating feature changes as a sparse transport problem solved with flow matching.

Latent signals for unanswerability generalize robustly across diverse question-answering datasets, but their utility in multi-turn dialogue hinges on capturing conversational dynamics rather than just static missing information.

Connectome-based GNNs achieve high accuracy but are dangerously overconfident, with misclassified patients receiving up to 95% confidence.

Tabular foundation models exhibit joint label processing, deviating from fixed-weight in-context learning and suggesting context labels dynamically alter each other's influence on predictions.

A small, causally identified set of attention heads reliably detects network infrastructure information across diverse LLMs, but their responsibility may reside in single neurons or be distributed across the head depending on the model.

LLMs can decode partner intents but struggle to translate this knowledge into cooperative actions, with explicit action recommendations proving far more effective than abstract rule statements.

Most observed IPv6 Fragment-EH patterns are simple within-packet co-occurrences, not complex temporal network rules, a finding rigorously established by a novel negative-control protocol.

Time-series foundation models, despite their promise for "what-if" analysis, exhibit attenuated dynamics and memorylessness, requiring synthetic fine-tuning to recover accurate causal inference.

Unlocking interpretable uncertainty in LLMs by mapping semantic doubt and certainty anchors to a low-dimensional subspace, outperforming baselines even when controlling for output length.

OmniLLMs can now confess which specific evidence tokens rely on, enabling targeted hallucination correction without retraining.

Latent reasoning models can achieve 97% of explicit CoT accuracy at 25% of the latency by shaping a continuous latent space with flow matching and self-verification.

LLM interrogations reveal diffusion model's internal tokens capture scene semantics surprisingly early, even without explicit prompts, and correlate with generation quality.

Causal representation learning now achieves block disentanglement under weaker intervention assumptions, directly enabling label-free visual state estimation in embodied agents.

Learned AI simulators can pass standard forecast metrics yet fundamentally fail to capture system dynamics, as demonstrated by a new response theory probe that reveals critical divergences in forced responses.

Prior-data fitted networks unlock state-of-the-art tabular anomaly detection without requiring anomalous samples during training.

A single space token transforms Transformers from arithmetic failures to 89% accurate multipliers by enabling learnable carry propagation.

Deep GNNs can achieve superior depth scaling and accuracy by leveraging controlled Turing instability or stable near-critical propagation to combat over-smoothing.

Faithful concept bottleneck explanations are impossible without including all concepts, but probabilistic modeling and group lasso enable compact, faithful explanations.

Influence functions, not just activation averages, unlock more thematic and robust concept steering in LLMs.

Most LLM vulnerability reports are right about the bug but wrong about *why*, with free-form reasoning hiding critical flaws that VERA's structured audit uncovers.

Spurious features are not just learned, they actively undermine worst-group accuracy during head retraining, even with existing robust methods.

Spatial directionality, a critical but overlooked capability, is readily recoverable from frozen VLM encoders with a simple, parameter-free readout.

Unsupervised EEG domain adaptation is unlocked by treating distribution shifts as linear transformations on the SPD manifold, enabling interpretable alignment without labeled calibration data.

Diffusion language models can be steered to generate biased outputs by exploiting their iterative denoising process with a closed-loop controller, achieving significantly higher bias injection than fixed steering.

The internal journey of an LLM's prediction, not just its destination, reveals crucial information about input difficulty and generator identity.

Key contribution not extracted.

Unlocking latent topological structures with discrete Morse theory boosts graph diffusion model performance, offering theoretical stability guarantees.

Teacher averaging in MOPD, not just raw gradients, dictates student learning, with Adam's momentum and BF16 precision masking significant parameter shifts.

A probabilistic deep learning emulator for a stochastic climate model not only achieves high fidelity but also learns a latent space that unsupervisedly disentangles metastable regimes and transition dynamics.

Language model "self-repair" is a predictable consequence of pre-existing counterweight mechanisms, not emergent compensation, governed by an affine law.

Current interpretability objectives can misrank circuits, favoring less faithful ones that mimic model behavior imperfectly, even when identical circuits are available.

A novel hybrid deep learning approach achieves robust few-shot malware detection by leveraging autoencoder-derived features for rapid meta-learning adaptation.

Unlearning PII from VLMs is now feasible without sacrificing retained knowledge by directly manipulating attention-value representations.

A simple "safe response centroid" embedding fails to reliably distinguish safe from unsafe LLM outputs, often performing worse than chance and requiring an explicit unsafe reference to orient the safety direction.

Learned attention weights actively harm in-context learning performance in Chronos-2, while simple uniform pooling provides consistent benefits.

Discover the "why" behind ML benchmark gains with MECHVAR, a variance-guided approach that achieves near-optimal mechanism discrimination 100x faster than EIG.

LLM concepts form non-linear manifolds, with alignment metrics revealing distinct layer-wise structures and training-dependent multilingual concept sharing that linear methods miss.

Acoustic noise, even when not diagnostically informative, can systematically manipulate Alzheimer's predictions in SSL models, revealing a critical vulnerability in current speech-based assessments.

Novel dashboards can bridge the gap between complex topological data analysis and user interpretation by explicitly acknowledging and managing methodological uncertainty.

A learned linear skip connection eradicates spurious local minima in shallow ReLU networks, even when overparameterized, by confining student features to the teacher's subspace.

Transformer model explanations diverge significantly under diagnostic uncertainty, highlighting the critical need for multi-method auditing and careful ontology selection in high-stakes medical classification.

A subtle structural signal in protein interactomes predicts GNN prediction failures, even when accounting for node degree and network complexity.

Reinforcement learning with sparse autoencoder features (RL-SAE) unlocks precise, interpretable, and composable design of intrinsically disordered protein regions, activating 90% of targeted functional features compared to 24% with activation steering.

Humans can directly control complex embodied agents using a handful of learned, composable skills discovered through self-play, bypassing the need for low-level action control.

Property-specific probing unlocks efficient, confidential multi-group fairness auditing, revealing bias structure without model reconstruction and resolving open sample complexity questions.

A single model's representation can predict how it disagrees with others, unlocking efficient input reliability estimation without needing multiple models at inference.

A single learned linear layer, not the shared Transformer weights, dictates distinct computations within each recurrent loop of a Looped Transformer, with attention patterns causally routing these operations.

Most LLM lie detectors fail when models adopt anti-factual personas, instead latching onto spurious correlations like instruction following.

The Green's operator reveals how computation is globally routed and reused across tasks and time in recurrent networks, offering a new lens beyond activity geometry.

Legal LLMs can now be optimized for nuanced quality beyond correctness, with rubrics that dissect response evaluation into Style, Element, and Chain of reasoning.

AI-driven music comparison tools can effectively surface subtle, expert-validated differences by letting users precisely scope comparisons and inspect AI-interpreted features.

Weight decay, not just learning rate, dictates the peak magnitude and lifespan of massive activations in transformers.

Students trained with OPMD learn to mimic teacher reasoning processes, not just outputs, leading to demonstrably improved reasoning capabilities and altered solution strategies.

Auditing just 5% of token positions catches nearly all model threats—and simple chat structure picks those tokens better than internal computational signals without requiring a forward pass.

Generative models can only rely on efficient local denoising up to the exact moment they commit to global semantics, after which both phenomena collapse into a single sharp phase transition as system size grows.

Context expansion inflates an LLM's active token budget not to retrieve new facts, but simply to counteract softmax mass dilution caused by distractor token competition.

Far from merely clustering tokens, full Transformers act as pattern-forming dynamical systems that natively amplify traveling and rotating waves—an inductive bias that can be engineered at initialization to accelerate training.

Training data attribution on 32B-parameter models no longer requires intractable gradient storage: extreme 1-bit eigenbasis projection achieves competitive influence estimation accuracy at over 100× lower per-example footprint than standard baselines.

Most directions in a transformer's residual stream are functionally invisible to the terminal loss, exposing a thin observable subspace whose pullback Fisher geometry consistently outperforms standard attention weights for compression and distillation.

Key contribution not extracted.

Where an LLM stores a fact is rarely where you should erase it: high-precision storage localization picks the optimal unlearning intervention on just 17 of 36 targets, exposing a core blind spot in mechanistic model editing.

Models don't need to manipulate curved manifolds to compute over them: Qwen executes number comparison via simple linear projections and localized MLP boundaries, largely ignoring the non-linear manifold geometry of its own representations.

Even after 50,000 shared optimization steps and near-identical accuracy, neural networks preserve permanent internal footprints of their earliest training curricula.

Early transformer layers act as in-context PCA engines, compressing high-dimensional visual demonstrations into a universal, low-dimensional subspace where classification becomes trivially separable.

Swapping deep networks for symbolic logic rules specifically at taxonomic leaf nodes delivers verifiable, human-readable explanations across hundreds of granular classes without sacrificing hierarchical context.

Existing "private" graph explanations quietly leak sensitive edges with an empirical AUC of 1.0 and implied privacy budgets exceeding $\varepsilon = 700$, exposing a structural flaw that can only be solved by enforcing data-independent intervention universes.

Parameter matrices are no longer inscrutable black boxes: decomposing weights directly into the activation features they read and write enables scalable, causally editable circuit discovery at 8B scale.

A single trained neural network can harbor vastly different internal computations and representations within its local solution space, a diversity standard optimization misses.

On-policy LLM post-training's generalization advantage stems from its dynamic parameter update direction, a principle transferable to SFT for improved performance and efficiency.