Search papers, labs, and topics across Lattice.
This paper introduces the Disentanglement-based Equivariant Learning (DEAL) framework, which addresses the limitations of current compositional visual question answering (VQA) methods by focusing on the disentanglement of underlying concepts without relying on additional training clues. By employing causality-inspired interventions within a re-encoding framework, DEAL enhances the model's ability to understand and reason about novel combinations of visual and textual inputs. Experimental results on the CLEVR-CoGenT and GQA-SGL datasets demonstrate that DEAL significantly outperforms existing state-of-the-art techniques in both visual and linguistic generalization tasks.
Ground-truth answers alone can drive a new era of compositional VQA, as shown by the DEAL framework's remarkable performance on challenging benchmarks.
Compositional visual question answering (VQA) represents a challenging yet fundamental task that requires models to comprehend novel combinations of previously learned concepts. The current methods often overlook the disentanglement of underlying concepts and are restricted in terms of their ability to effectively capture the compositional variation mechanism. Moreover, the state-of-the-art techniques depend on additional clues for training, which is not feasible in real-world VQA scenarios. To address these issues, in this paper, we introduce a novel Disentanglement-based EquivAriant Learning (DEAL) framework for compositional VQA, which is guided exclusively by ground-truth answers. In DEAL, we employ causality-inspired interventions to disentangle concepts derived from visual and textual inputs within a re-encoding framework. Based on the principle of equivariance, we subsequently perform a compositional transformation on the inference input and impose the equivariant constraint on the output to augment the compositional reasoning capacity of the model. Comprehensive experiments conducted on the benchmark CLEVR-CoGenT and GQA-SGL datasets validate the superiority of our proposed DEAL approach over the existing state-of-the-art methods for compositional VQA tasks in both visual and linguistic generalization settings.
