Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates per-sample forgetting dynamics during fine-tuning of ResNet-18 and DeiT-Small on imbalanced retinal OCT and CUB-200 datasets, fitting exponential decay curves to each sample's retention trace. The study reveals that CNNs and ViTs forget different samples, with ViT forgetting being more structured, and that per-sample forgetting is stochastic across random seeds. Furthermore, a sample's loss after head warmup predicts its long-term decay constant, although a spaced repetition sampler based on these constants does not outperform random sampling.
CNNs and ViTs exhibit surprisingly divergent forgetting patterns during fine-tuning, suggesting that ensembling models with different architectures could provide more robust retention coverage.
Fine-tuning pretrained image classifiers is standard practice, yet which individual samples are forgotten during this process, and whether forgetting patterns are stable or architecture dependent, remains unclear. Understanding these dynamics has direct implications for curriculum design, data pruning, and ensemble construction. We track per-sample correctness at every epoch during fine-tuning of ResNet-18 and DeiT-Small on a retinal OCT dataset (7 classes, 56:1 imbalance) and CUB-200-2011 (200 bird species), fitting Ebbinghaus-style exponential decay curves to each sample's retention trace. Five findings emerge. First, the two architectures forget fundamentally different samples: Jaccard overlap of the top 10 percent most-forgotten is 0.34 on OCTDL and 0.15 on CUB-200. Second, ViT forgetting is more structured (mean $R^2 = 0.74$) than CNN forgetting ($R^2 = 0.52$). Third, per-sample forgetting is stochastic across random seeds (Spearman $ρ\approx 0.01$), challenging the assumption that sample difficulty is an intrinsic property. Fourth, class-level forgetting is consistent and semantically interpretable: visually similar species are forgotten most, distinctive ones least. Fifth, a sample's loss after head warmup predicts its long-term decay constant ($ρ= 0.30$ to $0.50$, $p < 10^{-45}$). These findings suggest that architectural diversity in ensembles provides complementary retention coverage, and that curriculum or pruning methods based on per-sample difficulty may not generalize across runs. A spaced repetition sampler built on these decay constants does not outperform random sampling, indicating that static scheduling cannot exploit unstable per-sample signals.