Search papers, labs, and topics across Lattice.
This paper investigates the challenges of multi-view fusion in foundation models, revealing that increasing the number of fused encoders can lead to non-monotonic downstream performance due to redundancy and task misalignment. The authors introduce KAGES, a label-aware method that selects encoders based on their marginal gain in kernel-target alignment without requiring downstream classifier training. Empirical results demonstrate that KAGES significantly outperforms traditional fusion methods, improving average AULC across various protocols while highlighting the importance of selecting a compact, task-aligned set of views.
Fusing more encoders can actually degrade performance, but KAGES reveals how to select the right ones for optimal results.
Foundation-model hubs turn multi-view fusion into a selection problem: from a large heterogeneous encoder pool, which views should be fused, and how many? We show that downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline. We formalise this setting as view-set composition and propose KAGES (Kernel-Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target alignment. KAGES requires no downstream classifier training during selection, evaluates each candidate in $\mathcal{O}(n^2)$ time independent of encoder dimension, and admits a conditional $(1-e^{-\gamma})$ prefix-wise guarantee under monotonicity and a positive submodularity ratio. Across five recognition regimes and low-shot, larger-pool, and full-data protocols, KAGES improves average AULC over full fusion by 3.9, 5.8, and 3.3 points, respectively, and exceeds DPP and facility-location selection in average AULC. Image retrieval exhibits later, task-dependent saturation along the KAGES ordering, while peak-then-decline reproduces in frozen-LLM fusion. These results show that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.
