Search papers, labs, and topics across Lattice.
This paper introduces CPInj, a novel collaborative prompt injection attack targeting Textual Collaborative Prompt Optimization (TCPO), which allows multiple clients to enhance prompts for large language models while keeping their data local. The attack exploits the unique vulnerabilities of TCPO by injecting malicious instructions that persist through aggregation and benign prompt optimization, ultimately degrading performance across various tasks. Experimental results across multiple LLM families reveal that existing defenses are ineffective against CPInj, highlighting a critical security gap in TCPO and the need for improved mitigation strategies.
Malicious instructions can infiltrate and degrade collaborative prompt optimization processes, exposing a critical vulnerability in TCPO that current defenses fail to address.
Textual Collaborative Prompt Optimization (TCPO) extends Textgrad (Yuksekgonul et al., 2025) to a decentralized setting by allowing multiple clients to jointly improve prompts for large language models (LLMs) while keeping their data locally. Its reliance on free-form textual updating and aggregation introduces a new and largely unexplored attack surface, i.e., malicious instructions can be injected into local prompts and propagated through server-side prompt aggregation. Unlike conventional prompt injection attacks, attacking TCPO targets the collaborative optimization loop in TCPO. This setting is more challenging because malicious instructions must survive aggregation, persist through subsequent benign prompt optimization, and evade server-side defenses. To expose this risk, we propose CPInj, a collaborative prompt injection attack that contaminates the aggregated global prompt with malicious instructions, degrades downstream task performance, resists purification by prompt optimization on benign clients, and evades advanced detection-based defenses on the server. We find that current defense methods are ineffective against CPInj. To mitigate this attack, we further propose a defense-oriented aggregation method, i.e., APAgg, which purifies malicious instructions and partially recovers TCPO utility. We conduct extensive experiments across three LLM families and five reasoning tasks in math, logic, and medicine. The results demonstrate that our proposed attack reveals a critical vulnerability in TCPO. Although we take a first step toward mitigation, the attack remains highly effective and far from fully resolved, calling for more robust defense for TCPO.
