Evaluating cybersecurity AI agents: tasks, evidence and denominators

A cybersecurity agent can sound convincing without completing its task. Start evaluation with an observable outcome: finding an authorised procedure, qualifying an advisory, demonstrating a laboratory flaw or refusing an unauthorised action. These objectives need different denominators.

Cybench defines CTF-style security tasks for studying capabilities and risks. Success there does not directly measure enterprise pentest coverage, customer-report quality or tenant isolation.

Define cases before execution

Record environment, access, objective, initial state and success criteria. Specify prohibited actions and reset procedures. Every run should restore comparable conditions rather than depend on state left by previous attempts.

In a fictional access-control check, the expected result is that account A cannot read B’s document. HTTP status alone is insufficient: identify the returned document and its owner. Demonstration data should make that association verifiable.

Score outcomes and evidence separately

A correct unsupported answer can be a useful hypothesis without proving observed behaviour. Genuine evidence can also be misinterpreted. Our proposed evaluation separates outcome, justification and scope compliance.

For a vulnerability finding, require preconditions, steps, observed result and impact. For abstention, require the blocking reason and information needed to resume. “Verified” should name an identifiable check.

Name the measured sets

Where positive and negative cases are known, precision is TP / (TP + FP) and recall is TP / (TP + FN). Incorrect alerts among findings differ from the false-positive rate across negative cases, FP / (FP + TN).

Fictional arithmetic example: an agent detects three of four positive cases and raises one incorrect alert among six negatives. Precision and recall are each three quarters; the false-positive rate is one sixth. These figures illustrate formulas rather than reporting a product experiment.

When an application’s complete set of flaws is unknown, a delivered report cannot establish total recall. No published finding does not prove the absence of vulnerabilities.

Include memory and recovery

Resumption must distinguish current facts from historical revisions. LongMemEval offers dimensions including temporal reasoning, updating and abstention. Adapt these to your service.

Add revoked permissions, unavailable tools, conflicting evidence and lost responses to completed external effects. Check that recovery does not repeat the effect and that blocked tasks are not presented as successful.

Keep comparisons reproducible

Single-agent and multi-agent comparisons require the same cases, access and explicit budgets. Retain model versions, parameters, tools and repetitions. Report variation, failures and excluded cases with reasons.

MAST’s multi-agent failure study motivates classifying coordination and verification errors. Aggregate rankings cannot replace this diagnosis. Model-based judging should also be checked against verified annotations on a suitable sample.

Our RAG selection demonstration publishes ten deterministic cases, expected and actual outcomes and the corpus hash. It generates no answers and executes no security tests. Its evidence concerns the selection rules it runs. The AI and security series connects evaluation to orchestration and RAG security.