Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors investigate why pixel-level masked image modeling consistently outperforms contrastive and joint-embedding predictive architectures in handwritten text recognition, evaluating six self-supervised objectives across six multilingual benchmarks under matched setups. They find that discriminative signal in handwriting resides almost entirely in high-variance pixel subspaces, meaning representations that discard fine-grained pixel details lose critical character identity information. Consequently, encoder alignment with this high-variance pixel subspace directly predicts character error rate (CER), allowing a frozen pixel-grounded vision encoder paired with an LLM decoder to match fully fine-tuned supervised baselines.
While natural vision largely abandoned pixel reconstruction for latent objectives, handwriting recognition fundamentally requires it because character identities reside almost exclusively in high-pixel-variance directions that contrastive methods and JEPAs discard.
In self-supervised pretraining for Handwritten Text Recognition (HTR), pixel reconstruction methods outperform contrastive methods, unlike in natural-image classification. We argue that this difference follows from where discriminative signal lies in pixel space: for HTR, it is concentrated in high-variance directions and largely absent from low-variance ones. This predicts that objectives preserving high-variance pixel content will transfer best. We test six SSL methods from three families (pixel-grounded MIM, JEPA, and contrastive) under matched encoder, data, and evaluation protocols on six handwriting benchmarks across five languages. With full labels, pixel-groundrounded SSL achieves the lowest CER on every benchmark and both frozen probes, exposes per-position character information that other families recover only through the readout, and is the only family to benefit from pretraining on real handwriting. Pixel-grounded representations are also more label efficient. Across datasets, encoder alignment with the high-variance pixel subspace predicts CER within every method. With a pretrained LLM decoder, a frozen pixel-grounded encoder is competitive with fully fine-tuned supervised baselines; full fine-tuning achieves the lowest mean CER and ranks first or second on every benchmark. These results show that the value of pixel reconstruction depends on where discriminative signal lies in the input.