AI in red teaming: scope, working memory and autonomy levels

“Find a vulnerability here” is often the worst first instruction to give an assistant. Here might mean a cloned VM, a live client network, an administrator session or a test account. The model fills the gaps with a plausible story, and the operator may pursue a lead that fits that story rather than the target. The first rule is less dramatic: let it understand what exists, what we know and what is missing. This slows down the first few minutes and can save hours spent explaining the wrong result.

These are rules for making decisions during an engagement, not an agent architecture. Our note on offensive-agent scope guardrails already proposes a proxy, action classes, an audit log and an emergency stop. It remains a working note, not an account of a deployed system. The question here comes before each command: why this test, under which identity, and what result would change our mind?

Model the target before looking for a flaw

At the start of an engagement, I would give the assistant a short target sheet: observed products and versions with their sources, authorized hosts and paths, accounts and actual rights, testing window, fragile services and the exercise objective. I would add the unknowns. “The console appears reachable from the test VLAN” does not mean “the console is Internet-exposed.” “A log names SYSTEM” does not mean “our code ran with a SYSTEM token.” The wording of that sheet sets the quality of the reasoning that follows.

SEC565 puts intelligence, scenario, scope and rules of engagement before execution. The MITRE ATT&CK adversary emulation plans follow a related logic: chain behaviours drawn from public reports while preserving the limits of those reports. AI can help connect observations to a scenario. It cannot expand the mandate or make missing observations real.

Before asking for a lead, I would ask it to restate the identities and trust boundaries, then name three established facts, three unknowns and the source for each fact. If it cannot tell whether a command runs on the target VM or the operator’s workstation, the next action is a context question, not an exploit attempt. That looks slow only until a whoami result turns out to have come from the wrong host.

Test a hypothesis with a negative control

Suppose an application returns 200 to a request sent with a low-privilege account. A rushed reading says “missing access control.” Another says “same status, filtered data.” A third says “the browser supplied an administrator cookie.” The assistant should produce those competing readings before we choose the one we like. The useful test is not another payload variation: it is a clean session without the cookie, and a comparison of content or effects rather than HTTP status alone. This is a hypothetical example of method, not a rootzero vulnerability claim.

Each lead needs a control that could disprove it: request a nonexistent object, check the effective token or inspect the actual write path. Keep these control results with the original test. Apply the same checks to hypotheses from the operator and the model.

Keep a ledger of test results

An assistant’s context contains instructions, tool results and target pages. Reducing this history can discard details needed to verify a finding. Anthropic describes context as a finite resource to curate and refresh as work progresses. We propose keeping the following decision log separately:

FACT        Test account A: GET /records/17 -> 200, body has no owner field.
SOURCE      Request 0042 and response 0043; browser has no admin session.
HYPOTHESIS The filter applies to the body, perhaps not object existence.
REJECTED    “Full read of another user's record”: sensitive field absent.
NEXT        Compare /records/17 with a missing ID using the same account.

References point to traces preserved outside the prompt. The ledger stops “we proved access” from surviving a control that refuted it. At each step, the assistant can propose an update; the operator checks it against raw outputs. Read the tool calls and their arguments alongside the final summary. Target pages and documents remain untrusted data: they can contain instructions aimed at the model without becoming engagement rules.

Grant autonomy according to the cost of error

In an owned, isolated, restorable VM populated with dummy data, an agent can chain read operations, modify test scripts and rerun negative controls within a bounded budget. The point of that autonomy is to produce more verifiable experiments. It does not turn a lab artifact into a finding against a production version: we still need the exact version, prerequisites, a chain to impact and a contrary result.

On a live client system the cost changes. A call that sends email, modifies rights, affects availability or leaves scope does not become acceptable because the model calls it the “logical next step.” The operator must inspect the exact action, target, volume and expected effect before running it, under the rules of engagement. Account permissions and tool controls must limit what can happen even if the model or human makes a mistake. OWASP separates excess functionality, permissions and autonomy for this reason.

More “approve” dialogs are not enough; eventually people click without reading. Anthropic observed this fatigue in its own Claude Code permission prompts, not in red team operators generally. The transferable lesson is to narrow tool capabilities and isolate the environment, then spend human attention on decisions that warrant it. (Anthropic’s account)

Vulnerability research in a controlled lab and emulation on a client network therefore deserve different degrees of autonomy. In both settings, the assistant can be fast and inventive. It becomes useful when it can say “I don’t know,” when we are willing to retract an appealing conclusion, and when the next action is chosen to separate hypotheses rather than keep a story alive.