Researchers say LLM role tags can be abused to bypass prompt-injection defenses

Security researchers have reported a new way to mislead large language models into ignoring safety boundaries by taking advantage of how models interpret role-based prompts. The work suggests that current defenses against prompt injection remain fragile because models may not reliably tell the difference between trusted instructions and hostile text.

The study, by independent researchers Charles Ye and Jasmine Cui together with MIT associate professor Dylan Hadfield-Menell, is described in a paper titled Prompt Injection as Role Confusion, due to appear at ICML 2026. The authors argue that modern LLMs rely heavily on text labels such as system, user, assistant, tool, and think to separate instructions and manage behavior, but that these labels are not enforced securely inside the model itself.

According to the researchers, models often infer roles from superficial cues such as writing style rather than from trusted metadata. That opens the door to attacks that imitate internal reasoning or other privileged formats. In one demonstration, they used a technique they call Chain of Thought Forgery to make prompts appear as if the model had already reasoned through and approved the request.

The researchers say that, on benchmark tests, the method raised attack success rates dramatically compared with ordinary jailbreak attempts. They also claim the approach transferred across models more reliably than many existing prompt-injection tricks because it targets a structural weakness rather than relying on a single model’s quirks.

Why the issue matters

• Role markers are used as a core part of how LLMs separate trusted and untrusted text.
• Attackers may be able to inject misleading content through prompts or documents.
• Benchmark scores can overstate real-world resilience if they do not include adaptive human testing.
• The researchers say stronger protections may require a deeper redesign of how models understand roles and permissions.

The paper’s authors conclude that prompt injection is unlikely to disappear under the current security model. In their view, the industry may need new methods for making models recognize authority and boundaries in a more reliable way, rather than continuing to patch individual exploits as they emerge.