Search papers, labs, and topics across Lattice.
This paper introduces AE-PSL, a novel framework that enhances Distributed Fine-Tuning (DFT) of large-scale Foundation Models (FMs) on edge devices by employing an AutoEncoder to compress intermediate activations and gradients. The method addresses the limitations of existing communication-compression techniques by implementing a two-stage alignment mechanism that ensures compatibility with pre-trained models, thus preventing feature-distribution misalignment. The results demonstrate that AE-PSL significantly reduces communication overhead while maintaining or improving DFT performance compared to traditional methods.
By integrating a lightweight AutoEncoder with a two-stage alignment mechanism, AE-PSL achieves efficient communication in Distributed Fine-Tuning without sacrificing model performance.
Distributed Fine-Tuning (DFT) of large-scale Foundation Models (FMs) on resource-constrained edge devices is limited by local compute constraints and communication overhead. Parallel Split Learning (PSL) reduces client-side computation by keeping few model layers on each client and offloading the remaining computation to the server; however, clients must exchange intermediate activations and gradients with the server at every training step. Existing SL communication-compression methods mainly rely on task-agnostic heuristics, such as sparsification and quantization. While learnable SL compressors can better adapt to intermediate representations, they require co-training with the target model. Therefore, directly inserting them into off-the-shelf FMs introduces feature-distribution misalignment and degrades DFT performance. To address this, we propose AE-PSL, a communication-efficient PSL framework that compresses intermediate activations and gradients using a lightweight AutoEncoder (AE) placed at the split layer. To ensure compatibility of AE compression with pre-trained FMs, AE-PSL introduces a novel two-stage alignment mechanism, which adapts the AE to the pre-trained model's feature manifold and client-specific feature distributions before DFT.
