Prompt injection
Also called indirect prompt injection, LLM prompt attack
Prompt injection is an attack on a large language model in which text supplied by a user or hidden in content the model reads overrides its intended instructions. It can make a chatbot reveal confidential data, ignore guardrails or, in agentic systems, take actions such as sending emails or moving data without authority.
How it works
In direct prompt injection, a user types instructions designed to make the model disregard its system prompt or safety rules. In indirect prompt injection, the attacker plants instructions in content the model processes, such as a web page, email, document or customer message. Because the model treats everything in its context as text to follow, it may carry out the hidden instructions without the user knowing.
The risk grows when a model can call tools, read internal systems or act for users. Firms reduce it by limiting the model's permissions, separating trusted instructions from untrusted content, filtering inputs and outputs, requiring human approval for sensitive actions and logging activity. Red teaming tests these defences. No single control removes the risk, so firms layer controls and treat model output as untrusted.
The rules
Requires high-risk AI systems to be resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting vulnerabilities, including adversarial inputs.
Providers of high-risk AI systems, such as creditworthiness assessmentClassifies attacks on AI systems, including direct and indirect prompt injection against generative AI and agents, and the mitigations for each.
Any organisation, voluntary guidanceSets baseline security principles for AI systems and names indirect prompt injection among the AI-specific risks to design against.
Developers, system operators and data custodians of AI systems, voluntaryCommon questions
What is the difference between prompt injection and jailbreaking?
Jailbreaking is a user trying to make a model break its own safety rules. Prompt injection is broader: it includes instructions hidden in third-party content that hijack the model on someone else's behalf.
Why does prompt injection matter for banks?
Banks deploy chatbots and AI agents with access to customer data and internal systems. A successful injection can leak personal data, give customers wrong information or trigger unauthorised actions.
Can prompt injection be fully prevented?
No current technique removes it entirely, because models process instructions and data as the same text. Firms limit the damage with least privilege, human approval for key actions and monitoring.
