Search papers, labs, and topics across Lattice.
This paper introduces PFAdapter, a novel framework that employs hierarchical LoRA decomposition to enhance personalized federated fine-tuning for Multimodal Large Language Models (MLLMs) in distributed network environments. By separating adapter parameters into global-shared and local-private components, PFAdapter effectively balances the need for universal multimodal semantics with the adaptation to client-specific nuances, significantly reducing communication overhead by nearly 50%. Experimental results across multiple datasets show that PFAdapter outperforms existing methods, achieving accuracy gains of 2.4% to 4.8%, thereby advancing the deployment of agentic AI systems in resource-constrained settings.
PFAdapter cuts communication costs by nearly 50% while boosting accuracy in personalized federated learning for MLLMs.
Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents capable of collaborative learning while maintaining data privacy at network edges. Within distributed network environments, Multimodal Large Language Models (MLLMs) serve as cognitive engines for edge devices, yet federated fine-tuning faces substantial challenges in balancing global knowledge aggregation with local adaptation under heterogeneous network conditions. Conventional federated protocols typically rely on uniform parameter aggregation, which conflates domain-invariant features with client-specific nuances, thereby resulting in suboptimal personalization and excessive communication overhead. To address these challenges, we propose PFAdapter, a communication-efficient framework introducing hierarchical LoRA decomposition to explicitly separate adapter parameters into global-shared and local-private components. Query and key projections are assigned to global synchronization for capturing universal multimodal semantics across the network, while value and output projections remain localized for edge-specific adaptation. Additionally, orthogonality regularization based on the Frobenius norm enforces strict separation between these components, preventing redundant feature learning. Selective aggregation protocols synchronize only global-shared components across the federated network, preserving local expertise and reducing communication costs by nearly 50%. Extensive experiments on VQA-RAD, SLAKE, Hateful Memes, and CrisisMMD datasets demonstrate that PFAdapter consistently outperforms state-of-the-art baselines, achieving accuracy improvements ranging from 2.4% to 4.8% across diverse edge intelligence tasks. Consequently, our framework establishes an efficient solution for agentic AI deployment in resource-constrained communication networks.
Tsinghua AI