Search papers, labs, and topics across Lattice.
This study introduces Black-Box Behavioral Fingerprinting (BBF), a method for detecting cloned outputs from large language models (LLMs) by leveraging behavioral signatures derived from model outputs. The authors conducted a large-scale analysis involving 288,000 responses across four model families and eight benchmarks, revealing that prompt choice accounts for 24.4% of output variance and achieving an AUC of 0.876 for same-model detection. Notably, BBF demonstrates resilience against non-adaptive prompt paraphrasing, although it faces challenges with style-invariant detection in structured outputs, highlighting a critical area for future research.
Cloned outputs can be detected with over 87% accuracy using behavioral signatures, revealing significant vulnerabilities in LLM deployment security.
System prompts can be extracted from commercial LLMs with over 80\% success and redeployed at zero cost, yet a prompt owner has no way to verify whether a suspected deployment is a clone. We propose Black-Box Behavioral Fingerprinting (BBF): the prompt owner registers a behavioral signature from model outputs and later tests whether a suspect deployment matches that signature more closely than an unrelated baseline. BBF requires only black-box API access. Through a large-scale study (4 model families, 8 benchmarks, 288{,}000 responses), we find that prompt choice explains 24.4\% of output variance and same-model detection reaches AUC 0.876. Cross-model performance is bounded by detector identity, with off-diagonal AUC ranging from 0.845 (Claude as detector) down to 0.665 (Qwen) and overall mean 0.725. BBF resists non-adaptive prompt paraphrasing (AUC $\geq 0.889$) and is robust to imperfect extraction, but a single-sentence formal-tone prefix can collapse detection on short structured outputs (MNLI 0.978 $\to$ 0.547), isolating style-invariant detection as the key open problem. Diagnostic Query Optimization, a zero-cost query selection rule, adds $+0.120$ to cross-model AUC.
