Search papers, labs, and topics across Lattice.
This paper addresses the challenge of effectively fusing artifact and semantic features in multimodal large language models (MLLMs) for AI-generated image (AIGI) detection. They identify an attention dilution problem caused by high intra-feature similarity in artifact features. To mitigate this, they propose TranX-Adapter, a lightweight fusion adapter incorporating Task-aware Optimal-Transport Fusion and X-Fusion to facilitate information transfer between artifact and semantic features. Experiments on AIGI detection benchmarks demonstrate consistent accuracy improvements (up to 6%) when using TranX-Adapter with advanced MLLMs.
A simple adapter that leverages optimal transport and cross-attention can significantly boost the ability of multimodal LLMs to detect AI-generated images by better fusing artifact and semantic features.
Rapid advances in AI-generated image (AIGI) technology enable highly realistic synthesis, threatening public information integrity and security. Recent studies have demonstrated that incorporating texture-level artifact features alongside semantic features into multimodal large language models (MLLMs) can enhance their AIGI detection capability. However, our preliminary analyses reveal that artifact features exhibit high intra-feature similarity, leading to an almost uniform attention map after the softmax operation. This phenomenon causes attention dilution, thereby hindering effective fusion between semantic and artifact features. To overcome this limitation, we propose a lightweight fusion adapter, TranX-Adapter, which integrates a Task-aware Optimal-Transport Fusion that leverages the Jensen-Shannon divergence between artifact and semantic prediction probabilities as a cost matrix to transfer artifact information into semantic features, and an X-Fusion that employs cross-attention to transfer semantic information into artifact features. Experiments on standard AIGI detection benchmarks upon several advanced MLLMs, show that our TranX-Adapter brings consistent and significant improvements (up to +6% accuracy).
