Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of costly and redundant feature acquisition at test time for multimodal foundation models by introducing ECHO-$k$, a task-agnostic, self-supervised principle for sequential modality selection. ECHO-$k$ leverages pretrained internal representations as proxy targets to guide an RL policy in acquiring informative modalities under budget constraints. Experimental results demonstrate consistent downstream performance improvements across various foundation models compared to existing baselines, offering a principled approach to cost-aware deployment.
Foundation models can learn to select the most informative modalities at test time without knowing the downstream task, significantly reducing measurement costs.
Recent progress in multimodal, high-dimensional learning has enabled foundation models to process heterogeneous, large-scale data. However, at test time, acquiring all features or modalities can be prohibitively costly and often redundant. Sequentially selecting informative modalities is therefore critical, yet challenging when the downstream task or prediction target is unknown. To this end, we introduce ECHO-$k$, a task-agnostic and self-supervised learning principle for modality acquisition: we use a deep model's internal pretrained representations (e.g., from a foundation model) as proxy targets that summarize cross-modal information. We provide theoretical guarantees in a stylized linear setting that motivate a reinforcement learning (RL) policy for sequential modality selection. Across task-agnostic and label-free acquisition baselines, ECHO-$k$ consistently improves budgeted downstream performance across diverse foundation-model backends. Our method provides a principled route to cost-aware test-time deployment, with implications for any multimodal system where measurements are expensive or time-constrained, and downstream tasks unknown a priori.