Search papers, labs, and topics across Lattice.
This paper introduces Gradient Mirage, a novel defense mechanism against gradient matching attacks (GMAs) in large language model split learning, which disrupts the critical gradient-objective consistency that attackers rely on to recover private labels. By inducing inconsistencies in objective, direction, and scale, the method ensures that the exposed gradients do not accurately reflect the client's full-label training objectives while maintaining optimization utility. Experimental results demonstrate that Gradient Mirage significantly enhances privacy protection compared to existing defenses, achieving a superior balance between privacy and model performance.
Gradient Mirage disrupts the gradient-objective consistency that underpins gradient matching attacks, offering a robust defense without sacrificing model performance.
Gradient matching attacks (GMAs) in LLM split learning (SL) rely on a critical yet underexplored assumption: the gradient exposed at the split interface is a faithful derivative of the client's full-label training objective. This gradient-objective consistency allows a curious server to recover private labels by searching for a sequence whose induced gradient explains the observation. We propose Gradient Mirage, a defense that breaks this consistency without discarding the optimization utility of the backward signal. Our key idea is to induce the adversary to solve a misspecified inverse problem, in which no plausible label sequence in the sequence space can explain the observed gradients. Concretely, Gradient Mirage achieves this by inducing inconsistency across three dimensions: objective, direction, and scale. Selective Autoregressive Supervision derives the exposed gradient from a masked surrogate loss rather than the full-label objective assumed by the attacker; Scale Blinding then applies randomized multiplicative rescaling, obscuring the gradient's natural magnitude; and Directional Privatization further randomizes the gradient direction while preserving its magnitude through the von Mises-Fisher (vMF) mechanism under a directional metric differential privacy guarantee. Crucially, utility is preserved: the Top segment still learns from all target tokens via Dual-Track Backpropagation, the exposed gradient remains informative since each supervised token retains its complete autoregressive context, and Bottom-Gradient Recovery restores the effective gradient for Bottom-segment optimization. Extensive experiments show that Gradient Mirage provides substantially stronger protection than existing defenses under comparable fine-tuning performance, achieving a better privacy-utility trade-off.
