Search papers, labs, and topics across Lattice.
This paper introduces SeRIn, a novel multimodal fusion architecture that separates the processes of refining modality-specific signals and modeling cross-modal interactions. By allowing modality-specific representations to evolve independently before integrating them in a final prediction step, the approach enhances performance without increasing model capacity. Experimental results demonstrate that SeRIn achieves state-of-the-art performance on CH-SIMS and CMU-MOSEI, with ablation studies indicating that structured interactions are key to its success.
SeRIn reveals that isolating modality refinement from cross-modal integration can significantly boost sentiment analysis performance, challenging conventional fusion methods.
Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose \textbf{SeRIn} (\textbf{Se}gregate, \textbf{R}efine, \textbf{In}tegrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.
