Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To determine whether identical downstream task performance implies internal representational alignment, the authors evaluated paired CNNs trained on reversed task sequences followed by up to 50,000 shared optimization updates. Despite reaching behavioral parity within a 0.18% accuracy gap and yielding equivalent linear probe separability, models persistently retained significant representational divergence and prediction disagreements. These findings reveal that early optimization history permanently shapes internal representations—partially mediated by activation plasticity—meaning that behavioral equivalence cannot guarantee mechanistic convergence.
Even after 50,000 shared optimization steps and near-identical accuracy, neural networks preserve permanent internal footprints of their earliest training curricula.
Neural networks trained toward the same final objective can reach similar predictive performance while retaining internal representations shaped by earlier training history. We study this effect using controlled sequential-training experiments in which paired convolutional networks start from identical weights, experience reversed task orders, and then receive the same deterministic common-relaxation distribution. Across 20 paired MNIST runs, 16 satisfy a predeclared behavioral-matching criterion, yet their matched representations retain a mean history score of 0.139 (95% bootstrap CI: 0.127-0.153) and approximately 3.1% prediction disagreement. Extending common relaxation to 50,000 optimizer updates does not erase the measured difference: across five paired seeds, the representation-history score remains 0.190 (95% bootstrap CI: 0.161-0.219) at the end of the measured horizon while the mean accuracy gap is only 0.18 percentage points. Fresh linear probes show that, with sufficient labeled data, the two histories retain practically equivalent linearly accessible class information. A same-label rotated-MNIST control reproduces the effect: all five paired seeds reach behavioral matching while retaining a mean representation-history score of 0.162. Finally, a matched-learning-rate ReLU-LeakyReLU control reduces the 50,000-update representation residue by 0.040 on average in all five paired seeds, providing directional evidence that activation-mediated plasticity contributes to the persistence of training-history effects. These results provide protocol-scoped evidence that behavioral convergence need not imply representational convergence and that optimization history can leave measurable internal traces after prolonged common training.