Search papers, labs, and topics across Lattice.
Split learning allows resource-constrained clients to collaboratively fine-tune large language models, but transmitting intermediate autoregressive activations fundamentally leaks sensitive client inputs past traditional perturbation defenses. To resolve this privacy collapse, the authors implement a learned obfuscate-and-recover scheme that masks representations locally before server transmission while maintaining the ability to train an independently deployable server-side model. The approach demonstrates strong empirical defense against activation-inversion attacks with minimal downstream utility loss and computational overhead.
Intermediate activations in split LLM fine-tuning trivially leak raw training prompts past standard privacy defenses, but a learned obfuscation pipeline closes this leakage vector without destroying model utility.
Fine-tuning large language models (LLMs) on domain-specific data is essential for downstream adaptation. In many deployments, a participant cannot hold the complete model locally. This happens because the model owner keeps the full model proprietary, or because the participant lacks sufficient compute resources. Split Learning (SL) addresses this by partitioning the model between the participant and a server so that only a small portion runs locally. When the underlying data is additionally distributed across multiple institutions with privacy requirements, Federated Learning (FL) further enables collaborative training across participants by sharing only model updates instead of raw data. In this combined setting, each client transmits intermediate activations to the server, and for LLM fine-tuning, this exchange poses an inherent privacy paradox. The autoregressive nature of LLMs causes the transmitted activations to leak the input, and existing perturbation-based defenses are fundamentally ineffective in this setting. We address this leakage through a learned obfuscate-and-recover scheme that protects participants'private datasets while still allowing an independently deployable model to be trained on the server side. Experiments demonstrate that our approach achieves strong privacy protection with modest utility loss and system overhead, making split-based federated LLM fine-tuning practically viable.
