Search papers, labs, and topics across Lattice.
This paper introduces OTRO, an efficient oblivious tokenization path designed to mitigate side-channel leakage in large language model (LLM) serving while maintaining low latency. By leveraging square-root ORAM and implementing innovative strategies such as replicated ORAM instances and an epoch-based rotation policy, OTRO significantly reduces the time-to-first-token (TTFT) overhead to a maximum of 4.5%, while keeping tokenizer-induced latency under 10% of total TTFT. The method not only enhances security against prompt recovery attacks but also minimizes memory overhead, making it viable for production environments with stringent performance requirements.
OTRO reduces tokenizer latency to just 4.5% overhead while effectively safeguarding against side-channel attacks in LLMs.
The CPU-side large language model (LLM) tokenizer is a critical security gap in LLM serving through a confidential computing stack with CPU and GPU trusted execution environments (TEEs). Tokenizers converts the prompts through table-driven lookups, and the resulting memory access patterns are a powerful source of side-channel leakage. Recent work demonstrates end-to-end recovery of user prompts from tokenizer access pattern on production Intel TDX. However, a drop-in use of the popular tree-based Oblivious RAMs (e.g., PathORAM) to prevent access-pattern leakage introduces $\sim$13$\times$ tokenizer slowdown, resulting in 10-58% higher time-to-first-token (TTFT). In this paper, we present OTRO, an efficient, oblivious tokenization path tailored to latency-critical LLM serving. OTRO relies on square-root ORAM for fast single-access lookups, but avoids its prohibitive $O(N\log^2N$) rebuild cost every $\sqrt{N}$ accesses through three key innovations. First, OTRO provides a pool of replicated square-root ORAM instances that utilize the read-only nature of tokenizer table. Second, an epoch-based rotation policy decouples accesses from rebuilds and pads each epoch with dummy accesses to its boundaries, minimizing observable information. Lastly, chunked KV-cache-aware tokenization further overlaps rebuilds with GPU prefill and minimizes the instance count. Implemented as modules in HuggingFace Tokenizers and nano-vLLM, running within a TDX-enabled CVM with an NVIDIA H100 GPU, OTRO limits TTFT overhead to at most 4.5%, keeps tokenizer-induced latency under 10\% of total TTFT, and adds less than 0.5 GB of memory overhead while reducing the tokenizer's observable leakage across various model families and sizes.
