Search papers, labs, and topics across Lattice.
This paper investigates the role of cross-entropy (CE) loss in the K-way energy probe, which approximates a monotone function of the log-softmax margin in discriminative predictive coding networks. By training predictive coding networks with MSE loss instead of CE and using bidirectional predictive coding (bPC), the study demonstrates that CE is a critical component for the probe's behavior. Results show that removing CE significantly reduces the probe-softmax gap, with temperature scaling ablations revealing that this gap is attributable to both logit-scale effects and a scale-invariant ranking advantage of CE-trained representations.
Cross-entropy loss isn't just a detail – it's the unsung hero behind how well energy probes work in predictive coding networks, accounting for up to 66% of the probe-softmax gap.
Cacioli (2026) showed that the K-way energy probe on standard discriminative predictive coding networks reduces approximately to a monotone function of the log-softmax margin. The reduction rests on five assumptions, including cross-entropy (CE) at the output and effectively feedforward inference dynamics. This pre-registered study tests the reduction's sensitivity to CE removal using two conditions: standard PC trained with MSE instead of CE, and bidirectional PC (bPC; Oliviers, Tang&Bogacz, 2025). Across 10 seeds on CIFAR-10 with a matched 2.1M-parameter backbone, we find three results. The negative result replicates on standard PC: the probe sits below softmax (Delta = -0.082, p<10^-6). On bPC the probe exceeds softmax across all 10 seeds (Delta = +0.008, p = 0.000027), though a pre-registered manipulation check shows that bPC does not produce materially greater latent movement than standard PC at this scale (ratio 1.6, threshold 10). Removing CE alone without changing inference dynamics halves the probe-softmax gap (Delta_MSE = -0.037 vs Delta_stdPC = -0.082). CE is a major empirically load-bearing component of the decomposition at this scale. CE training produces output logit norms approximately 15x larger than MSE or bPC training. A post-hoc temperature scaling ablation decomposes the probe-softmax gap into two components: approximately 66% is attributable to logit-scale effects removable by temperature rescaling, and approximately 34% reflects a scale-invariant ranking advantage of CE-trained representations. We use"metacognitive"operationally to denote Type-2 discrimination of a readout over its own Type-1 correctness, not to imply human-like introspective access.
