Search papers, labs, and topics across Lattice.
The paper investigates the vulnerability of per-token billing for LLMs to dishonest providers who can inflate token counts due to the opacity of model execution and tokenization. By analyzing three existing token auditing frameworks, the authors demonstrate that a provider can systematically overcharge users, achieving inflation rates of up to 1,469% in hidden reasoning scenarios and 50.85% even with visible reasoning strings. The findings highlight a fundamental "trust paradox" where audits rely on potentially manipulated evidence from the provider, necessitating verification methods based on external, uncontrollable evidence.
LLM providers can inflate your bill by over 14x without detection, thanks to unauditable token counts and a "trust paradox" at the heart of current auditing frameworks.
Per-token billing is now the standard pricing model for commercial large language models (LLMs), so the honesty of reported token counts directly affects what users pay. We show that this kind of billing is hard to audit by design: providers hide the model, the tokenizer, and the execution to protect their IP, mitigate jailbreaks, and preserve user privacy, which means an auditor can only inspect proofs the provider supplies. The audit therefore reduces to a consistency check on the provider's own reports. We call this a trust paradox: every audit must trust some artifact, but current frameworks trust exactly the ones a provider has the strongest reason to manipulate. We study three recent token auditing frameworks and show that a provider with ordinary commercial capabilities can systematically inflate billed token counts. In the most permissive setting, hidden reasoning usage can be inflated by 1,469% on average without detection. At current frontier reasoning prices, that turns a \$100 honest bill into roughly a \$1,569 bill on the same query. Even when the user can see the full reasoning string, tokenization ambiguity alone still allows 50.85% over-reporting below the detection threshold. These results suggest the problem is not in any specific auditor but in any audit whose evidence comes from the audited party. Restoring honest billing will require verification that ties reported token counts to evidence the provider does not control, such as trusted execution attestation, cryptographic proofs of inference, or third-party re-execution.
