Cybersecurity AI agents and multi-agent systems: what to assess

A cybersecurity AI agent uses a model to choose security-related actions and execute them through tools within a defined scope. The label describes how a system works, rather than establishing its reliability. Summarising an alert, querying logs and sending test requests involve different permissions and consequences.

A multi-agent system coordinates several agents. Separate roles can help divide exploration, analysis and reporting. Agent count alone does not demonstrate better coverage or independent verification: several agents can repeat the same incorrect assumption.

This guide proposes an assessment framework. The example is educational; it does not describe a product’s internal architecture or report benchmark results.

Assistant, automation or agent?

Approach	Main decision	Useful question
Assistant	An operator selects actions and reviews suggestions	What must the operator check before using a suggestion?
Deterministic automation	Software follows a predefined rule or scenario	Which inputs and cases fall outside that rule?
AI agent	A system chooses actions using permitted tools	Which actions can it actually execute?
Multi-agent system	Several agents exchange tasks and results	Who may delegate, change memory and accept evidence?

A service may combine these approaches. Ask to see a specific workflow. Distinguish a suggested command, its execution and evidence of its outcome.

Three security uses with different controls

For monitoring, an agent can match publications against a product inventory. A useful result retains the vendor link, versions and date. A plausible summary without references is insufficient for a remediation decision. Our RSS monitoring guide describes a qualification process.

For investigation, a system can assemble events it is authorised to read. Separate this from isolating a host, revoking sessions or deleting files. Permission to read a log should not grant permission to modify the environment.

For AI pentesting, agents may test hypotheses against authorised applications. Assess access, exclusions and report evidence. Testing with AI differs from testing an AI system’s security. The SpectrAI guide explains the former and its limitations.

A worked example on a fictional API

Consider two test accounts in separate organisations. The task is to establish whether each account can only read its own documents.

1. An exploration role records workflows accessible to each account.
2. A testing role proposes a tenant-isolation hypothesis.
3. A verification role requests HTTP exchanges demonstrating the outcome.
4. A reporting role explains impact, preconditions and remediation.

The contract between these roles matters. “Cross-tenant access confirmed” is a conclusion, not evidence. Associate the account identity, requested object, expected owner and observed response. Keep an incomplete result unconfirmed.

HTTP Inspector, HTTP Diff and Evidence Sanitizer help prepare these artefacts locally in the browser. They do not execute this API test.

Controls to demonstrate in a pilot

OWASP addresses agentic and multi-agent systems through threat modelling. Its guides do not certify a supplier. For a pilot, record the following proposed checks and their observed outcomes:

Risk	Control to demonstrate	Expected evidence
Hostile instructions in retrieved content	Read data cannot expand permissions	Out-of-scope actions are refused and logged
Unsupported conclusions passed between agents	A recipient requires reproduction material	A finding stays unconfirmed when evidence is missing
Excessive account privileges	Permissions match the task and duration	Prohibited operations fail
Context shared between customers	Memory and attachments are isolated	No information crosses workspaces
An action loop grows without bounds	Duration, calls and stopping controls are enforced	Stopping works and completed actions are recorded

Run checks against an authorised demonstration environment. A retrieved page or document remains an input to assess, even when another internal agent supplies it.

Measure useful outcomes

Define expected tasks before starting. Retain the environment, access, blocked cases and verification procedure. Measure demonstrated outcomes, unconfirmed hypotheses, errors and the effort required to make results usable separately.

A detection rate requires a known set of expected cases and an explicit denominator. Comparing suppliers requires the same scope and conditions. Agent count, messages and requests cannot substitute for that comparison.

MSPs should also establish each customer’s authorisation, workspace isolation, evidence recipients and reporting responsibilities. For pentesting, examine the SpectrAI partner programme and request a proposal for the actual scope.

Sources and further reading

• OWASP: Agentic AI, Threats and Mitigations.
• OWASP: Multi-Agentic System Threat Modeling Guide.
• AI pentesting, manual testing and scanners.
• Browser security tools: choose by task.