Guide
AI red teaming: assess model and agent risks systematically
AI red teaming tests how a model-powered system behaves under adversarial or unexpected inputs. For applications with tools, retrieval, or sensitive data, the assessment must cover the complete product and its integrations—not just model responses.
Define the system and threat model
Map the model, prompts, retrieval sources, plugins, tools, users, data stores, and external services. Identify which actions can change state and what data is sensitive. State the threats and business impacts you intend to assess.
Choose test accounts and data that let you observe risk without exposing real secrets or affecting customers. Record the model and application versions so a result can be repeated after changes.
Test high-impact failure modes
Probe for direct and indirect prompt injection, data exfiltration, insecure tool use, excessive permissions, unsafe code or command generation, refusal failures, and cross-user data exposure. Test whether untrusted documents or web content can alter the agent’s action plan.
For each behavior, capture the prompt or input, relevant context, tool calls, expected policy, observed result, and impact. Avoid publishing exploit details that would expose live secrets or enable harm.
Evaluate controls around the model
Test server-side authorization, tool allowlists, secret isolation, output validation, human approvals, rate limits, egress restrictions, logging, and emergency shutdown. These controls should work even when a model produces an unexpected answer.
Assess how incidents can be contained: who can revoke credentials, disable a tool, restore state, and notify affected users? Verify that audit records are useful without retaining unnecessary private data.
Prioritize fixes and retest
Rank issues by impact, likelihood, reachability, and the permissions available to an attacker. Assign each remediation to an owner and distinguish model behavior changes from application and integration fixes.
Retest the original case and related variants after changes. AI systems evolve with model, prompt, data, and tool updates, so incorporate targeted regression tests into release and change processes.