Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces predictive residual inference for ordered representations (PRIOR), a novel framework that enhances the utility of ordered bottlenecks in visual representation learning by addressing the limitations of masking-based ordering pressure (MBOP). By employing log2-scaled levels and level-wise predictors, PRIOR effectively separates explained from unexplained information, leading to improved representation quality across varying budget constraints. Experimental results demonstrate that PRIOR consistently outperforms MBOP-TD and MBOP-ITD in both contrastive learning and image reconstruction tasks, particularly in discrete and quantized settings where MBOP struggles.
PRIOR transforms ordered representation learning by ensuring that even low-budget models deliver meaningful coarse descriptors while maintaining high performance across all budgets.
Ordered bottlenecks aim to provide utility at flexible budgets by assigning coarse information to early tokens and task-relevant detail to later ones. Prior work, including tail dropping (TD), typically enforces ordering by means of a masking-based ordering pressure (MBOP): Late tokens are masked more frequently than early tokens and are therefore encouraged to store less essential fine details. We introduce predictive residual inference for ordered representations (PRIOR), a framework designed to address inherent weaknesses of MBOP. MBOP is prone to weak late-token utility because it lacks an explicit refinement objective and uses gradient exposure as a proxy for importance. Furthermore, representations may become particularly brittle in optimization-sensitive settings, such as when using discrete or quantized token representations. PRIOR replaces activation-rate control with log2-scaled levels and level-wise predictors. These predictors separate already explained from unexplained information, focusing each level on residual error. We compare PRIOR against MBOP-TD and independent tail-biased dropout (MBOP-ITD) in contrastive learning and image reconstruction tasks. Unlike the baselines, PRIOR learns well-ordered representations across experiments: low budgets provide coarse descriptors, while high budgets add refinements. Simultaneously, full-budget performance with PRIOR is higher in all but one experimental setting, where performance remains comparable. MBOP baselines are severely limited in discrete and quantized settings, while PRIOR approaches the performance of continuous counterparts. Taken together, these findings establish PRIOR as an effective framework for ordered representation learning.