Two related attacks
- A jailbreak crafts a prompt that tricks the model into ignoring its safety rules, for example role play or obfuscation.
- A prompt injection hides malicious instructions inside content the model reads, such as a web page or document, hijacking its behavior.
Why they work
- The model treats all text in its context similarly, so it struggles to separate trusted instructions from untrusted data.
- Safety training is imperfect and attackers explore phrasings it never saw.
Layered defenses
- Instruction hierarchy: train the model to trust the system prompt over user text and untrusted content.
- Input and output filtering: classifiers scan for known attack patterns and unsafe outputs.
- Privilege separation: keep dangerous tools behind explicit confirmations, so injected text cannot silently act.
- Content provenance: clearly mark untrusted data and discount instructions found inside it.
A realistic stance
- No single defense is complete, so defenses are stacked.
- Treat the model like a confused deputy and never grant the raw output unchecked power over tools or data.
Key idea
Jailbreaks and prompt injections exploit the model's inability to separate trusted instructions from untrusted data, so defense layers an instruction hierarchy, filtering, and privilege separation rather than relying on any single barrier.