Search papers, labs, and topics across Lattice.
To circumvent the prohibitive latency of full homomorphic inference, the authors analyze selective homomorphic inference, which evaluates only a sensitive region of interest (ROI) under FHE while running independent computations in plaintext without retraining. The speedup gained over full-input FHE is fundamentally constrained by how quickly encrypted dependency graphs expand through the network. Consequently, locality-preserving models achieve order-of-magnitude speedups for small ROIs, whereas architectures featuring early global mixing provide virtually no efficiency advantage.
Selective homomorphic encryption yields 10x inference speedups without model retraining, but early global mixing in modern vision backbones completely obliterates the savings.
Fully homomorphic encryption (FHE) enables inference on private data without revealing it to the server, but evaluating an entire input under FHE is expensive. We study \emph{selective homomorphic inference}, where only a sensitive region of interest (ROI) is encrypted, and computations independent of that region are performed in plaintext. Selective evaluation produces the same output as full FHE on the same model, without retraining. Its efficiency depends on how quickly encrypted dependencies spread through the network. For small encrypted ROIs, locality-preserving architectures can achieve order-of-magnitude homomorphic-evaluation speedups, whereas architectures with early global mixing provide essentially no speedup. These results identify locality as the key architectural property governing the benefit of selective homomorphic inference.
