Search papers, labs, and topics across Lattice.
This paper introduces OpenPCC, a Confidential Cloud Inference Service (CIS) framework that utilizes commercially available Trusted Execution Environments (TEEs) to securely process user requests containing sensitive information. The authors analyze the fundamental requirements for a secure and open CIS, addressing the limitations of existing proprietary solutions that hinder broader adoption. Their implementation of an open-source prototype, evaluated on a Llama-3 8B vLLM workload, confirms both the feasibility and security of the proposed system, marking a significant advancement in privacy-preserving AI applications.
OpenPCC achieves secure LLM serving without proprietary hardware, paving the way for broader adoption of privacy-preserving AI solutions.
Generative AI applications such as personal AI agents, image generators, and chat assistants offer advanced capabilities to improve user experience. Behind the scenes, Large Language Models (LLMs) that power these services require a massive amount of computation and are usually deployed in the cloud, available as APIs, meaning that a user's request has to be sent to a Cloud Inference Service (CIS) for processing. However, the strong capabilities of LLM also mean that user's requests now contain much more personal sensitive or enterprise confidential information, demanding equally strong protection in CIS. While early industry efforts such as Apple Private Cloud Compute (PCC) and Google Private AI Compute have emerged to show the potential of secure CIS, they are not adoptable for deployment by others due to their reliance on proprietary hardware and closed ecosystem. In addition, they all suffer from their own design glitches that can undermine the ambitious goal of bringing in true privacy protection to end users. In this paper, we present our analysis of the fundamental requirements of building a secure yet open CIS. We then present OpenPCC, a Confidential CIS framework that does not rely on proprietary hardware but instead uses commercially available TEEs. We implement an open-source prototype and characterize it end-to-end on a Llama-3 8B vLLM workload, separating OpenPCC's own cost from the underlying TEE hardware. Our analysis and evaluation demonstrated the feasibility and security of the system.
