Search papers, labs, and topics across Lattice.
This study evaluates the performance of large language model inference and training within a Trusted Execution Environment (TEE) on NVIDIA B200 GPUs, specifically using Intel's Trust Domain Extensions (TDX) and NVIDIA's Confidential Computing (CC). The findings reveal that when optimally configured, confidential inference incurs only a 1-3% throughput overhead, significantly lower than the 30-40% penalties observed with suboptimal configurations. The research also identifies the dual cost factors influencing performance, providing microbenchmarks and deployment guidance for efficient use of confidential computing in AI workloads.
Achieving just 1-3% overhead for confidential inference on Blackwell GPUs could redefine the viability of secure AI model training and deployment.
This paper measures the performance impact of running large language model inference and training inside a Trusted Execution Environment (TEE) on NVIDIA B200 GPUs, using Intel Trust Domain Extensions (TDX) confidential VMs together with NVIDIA Confidential Computing (CC) on Blackwell GPUs. The performance impact is derived from paired confidential versus non-confidential runs on a single physical host where the only variable is the GPU CC bit and the TDX guest object in the VM launch. The main result is that confidential inference on Blackwell achieves low single-digit throughput overhead when the stack is configured correctly, at about 1-3%. Stock inference stacks incur 30 to 40% penalties due to avoidable configurations rather than the achievable operating point. The cost is not fully represented by a single number because it is governed by two independent axes, a fixed per-host-operation cost that amortizes as batch size grows and a per-NVLink-traffic cost that tracks the share of the step spent in encrypted collectives, and which of the two dominates is set by the workload and the software. We localize each cost to a specific encrypted boundary, give a microbenchmark that predicts the serving penalty to within a submission count, and end with concrete deployment guidance. GPU compute, energy draw, and usable memory capacity are unaffected by CC.
