Search papers, labs, and topics across Lattice.
This paper addresses the problem of branch imbalance in shared-private architectures for multimodal sentiment analysis, where shared representations become redundant and private representations lose discriminative power. They propose a Dual-Branch Rebalancing Framework (DBR) with a Temporal-Structural Factorization module to reduce redundancy in the shared branch and an Anchor-Guided Private Routing module to preserve modality-specific patterns in the private branch. Experiments on three benchmark datasets demonstrate that DBR outperforms existing methods by mitigating branch imbalance and improving sentiment reasoning.
Multimodal sentiment analysis suffers from "branch imbalance," where shared representations become redundant and private representations lose discriminative power, but a new rebalancing framework can fix it.
Multimodal Sentiment Analysis (MSA) requires integrating language, acoustic, and visual signals without sacrificing modality-specific sentiment evidence. Existing methods mainly improve either shared-private decomposition or cross-modal interaction. Although effective, both ultimately depend on how shared and modality-specific evidence is organized before prediction. We observe that, under standard shared-private pipelines, modality heterogeneity often induces a branch-imbalance process: dominant shared patterns accumulate in the shared branch, yielding redundant and modality-biased evidence, while repeated interaction and rigid alignment gradually leak shared information into modality-specific channels and weaken discriminative private representations. As a result, the complementarity between shared and private representations is reduced, limiting robust sentiment reasoning. To address this issue, we propose the Dual-Branch Rebalancing Framework (DBR) on top of a standard multimodal decoupling stage. In the shared branch, a Temporal-Structural Factorization (TSF) module disentangles temporal evolution from structural dependencies and adaptively integrates them to reduce shared redundancy. In the private branch, an Anchor-Guided Private Routing (AGPR) module preserves discriminative modality-specific patterns while allowing controlled cross-modal borrowing. A Bidirectional Rebalancing Fusion (BRF) module then reunifies the two regularized branches in a context-aware manner for final prediction. Extensive experiments on CMU-MOSI, CMU-MOSEI, and MIntRec demonstrate that DBR consistently outperforms the compared baselines. Further analyses show that these improvements come from coordinated mitigation of branch imbalance.
