Red teaming
Also called AI red teaming, adversarial testing
AI red teaming is structured adversarial testing in which experts try to make an AI system fail, misbehave or cause harm before and after it goes live. Testers probe for jailbreaks, prompt injection, data leakage, biased or harmful outputs and unsafe actions, so weaknesses can be fixed and controls tested.
How it works
A red team agrees the scope, the harms that matter for the use case and the success criteria, then attacks the model or application. Manual testers write creative prompts and multi-step attacks, while automated tools generate large numbers of variations and score the outputs. For a bank chatbot, testers might try to extract other customers' data, get the model to give regulated advice or bypass its guardrails with role-play.
Findings are logged with severity, reproduced and passed to developers, who change prompts, filters, training data or system permissions, after which the red team retests. Results feed model validation, the model inventory and go-live decisions. Firms repeat red teaming when the model, data or connected tools change, because agentic systems that can take actions widen the range of possible failures.
The rules
Requires providers to evaluate their models using state-of-the-art protocols, including conducting and documenting adversarial testing, and to assess and mitigate systemic risks.
Providers of general-purpose AI models with systemic riskLists generative AI risks and recommends actions including AI red teaming to test for harmful outputs, information security weaknesses and misuse.
Voluntary for organisations developing or using generative AICommon questions
How is AI red teaming different from penetration testing?
Penetration testing looks for security weaknesses in networks and applications. AI red teaming also tests model behaviour, such as harmful or biased outputs, hallucinations and whether guardrails can be bypassed with crafted prompts.
Who should carry out AI red teaming?
A mix of internal testers independent of the build team, domain experts such as compliance staff, and external specialists. Automated tools add scale but do not replace human creativity.
Is AI red teaming required by law?
The EU AI Act requires adversarial testing for general-purpose AI models with systemic risk. For other systems, it is expected practice under model risk and AI governance frameworks.
