Search papers, labs, and topics across Lattice.
This paper investigates the Depth-Performance Dilemma in Split Federated Fine-tuning (SFF) of Large Language Models (LLMs), revealing that while deeper model partitions enhance throughput and privacy, they lead to significant performance degradation. Through rigorous evaluation across different model scales and benchmarks, the authors demonstrate that existing federated adapter aggregation methods fail to address the unique challenges posed by split architectures, resulting in catastrophic performance plateaus. The study identifies the underlying cause as the near-isometric topology of Transformers, which exacerbates aggregation noise and triggers Attention Collapse in server partitions.
Deeper partitions in federated fine-tuning may boost throughput and privacy, but they can also cause LLM performance to collapse catastrophically.
Split Federated Fine-tuning (SFF) is a promising paradigm for scaling Large Language Models (LLMs) by partitioning model depth between resource-constrained clients and a centralized server. While system incentives for throughput and privacy favor deep partitions, the impact of such configurations on model utility remains poorly understood. In this work, we identify and characterize the Depth-Performance Dilemma: the regime that maximizes system efficiency is precisely where fine-tuning quality collapses. Through a comprehensive audit across four model scales (GPT-2 to Llama-3-8B) and diverse benchmarks, we demonstrate that deeper partitions provide monotonic gains in throughput and privacy at the cost of catastrophic performance plateaus. We evaluate a suite of state-of-the-art federated adapter aggregation methods including AVG, STACK, SVD, and FREEZE, revealing that while these techniques are effective in standard Federated Learning, they fail to mitigate the artifacts unique to split architectures. Finally, we provide a mechanistic diagnosis for this failure, tracing the collapse to the near-isometric topology of Transformers, which allows aggregation noise to propagate without attenuation until it triggers Attention Collapse in the server partition. Our findings challenge the prevailing assumption that partition depth is a utility-neutral tuning knob and provide a structural foundation for stable distributed LLM fine-tuning.
