Search papers, labs, and topics across Lattice.
This paper introduces FedSEPT, a novel approach to federated prompt tuning that leverages Subspace-decomposed Expert Modeling (SEM) to effectively manage data heterogeneity while preserving privacy. By parameterizing multiple prompt experts with shared low-rank factors and private residuals, the method confines communication and differential privacy noise to a compact space, facilitating better aggregation on the server. Experimental results across 11 heterogeneous benchmarks demonstrate that FedSEPT significantly improves the balance between local adaptation and global generalization compared to existing methods under the same privacy constraints.
FedSEPT achieves superior local adaptation without sacrificing global generalization, even under stringent privacy conditions.
Federated prompt tuning (FPT) enables collaborative adaptation of vision--language models (VLMs) using lightweight prompts. Existing methods often address heterogeneity and privacy through a split-prompt design under local differential privacy (DP), combining a shared prompt for global transfer with private prompts for local adaptation. However, a single shared prompt may over-smooth diverse transferable knowledge, weakening the balance between personalization and generalization. Multi-expert prompts (MEPs) can better capture this diversity, but enlarge the communicated space, increasing DP noise and communication cost while making robust expert composition more difficult. We propose FedSEPT, a privacy-preserving Fed}erated Subspace-decomposed Expert Prompt Tuning. Specifically, we employ Subspace-decomposed Expert Modeling (SEM) to parameterize multiple prompt experts with shared low-rank factors, a fixed public basis, and private residuals, thereby confining communication and DP perturbation to a compact factor space while enabling direct server aggregation in a common coordinate system. We further design Instance-aware Expert Fusion (IEF), which adaptively combines semantically complementary experts via on-device routing and performs efficient logit-level fusion using cached expert-specific text features. Extensive experiments on 11 heterogeneous benchmarks show that, under the same privacy constraints, FedSEPT achieves a better trade-off between local adaptation and global generalization than strong baselines.
