Vulnerability validation cycle: from hypothesis to reproducible evidence

1. Problem

A security report exists to support decisions, and every line in it costs time: reading, triage, arbitration, remediation, verification. When it mixes demonstrated vulnerabilities with possible ones, it leaves the reader to tell them apart, even though the author was best placed to do so.

This mix is common. Automated analysis tools rely heavily on inference: a version banner, a signature recognised in a response, an unusual delay. These clues point to a possibility that still has to be established. The OWASP Benchmark project was created to measure, on a deliberately vulnerable application, the proportion of true and false positives these tools produce.

This note sets a rule and draws its consequences: a finding is published only if it is demonstrated. Other observations are leads for testing. They may appear in the report, in a separate section and under that name.

2. Definition of a demonstrated finding

We call a demonstrated finding one that meets the following five conditions.

1. It states an explicit hypothesis, with the suspected flaw and its mechanism: “the name parameter is interpreted by a server-side template engine”.
2. It rests on a minimal attempt, that is, a request or a short sequence of requests designed to validate the hypothesis with as little effect on the target as possible.
3. The observed response is discriminating. It differs from a reference response (baseline), and the difference can only reasonably be explained by the hypothesis.
4. A verifier separate from the agent that produced the attempt has replayed the request/response pair, with its baseline and a control, and confirmed the interpretation.
5. The attempt is recorded with its preconditions and can be replayed under comparable conditions, by the client to verify it or after remediation to confirm the fix.

The third condition is the most demanding. It rules out findings based solely on a version number, on a keyword in an error page or on a weak statistical signal.

3. Method: the five-step chain

Each step produces an artefact that is kept with the finding.

Step	Question	Artefact
Hypothesis	Which flaw, where, through which mechanism?	Statement, entry point, class (CWE)
Attempt	What is the smallest action that validates it?	Raw requests, timestamped
Observed response	What differs from the baseline?	Baseline response, test response, identified discriminator
Independent verification	Does the proof hold without the agent that produced it?	Replay by a separate verifier, confirmed discriminator, logged decision
Re-test	Does the proof replay, and does it disappear after the fix?	Replayable record, dated result

3.1 Canonical example: server-side template injection

The following example is fictional and the domain app.example.com is used for illustration only. It follows a classic detection of server-side template injection (SSTI), a technique described by James Kettle in 2015 in Server-Side Template Injection and included in the OWASP testing guide under WSTG-INJT-18.

The hypothesis concerns the /preview page, which would insert the name parameter into a template rendered on the server without treating it as plain data.

First, establish the baseline, meaning the normal behaviour:

GET /preview?name=rootzero HTTP/1.1
Host: app.example.com

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8

<p class="greeting">Hello rootzero</p>

The attempt then submits an arithmetic expression in the syntax of the most common engines, {{7*8}}, URL-encoded:

GET /preview?name=%7B%7B7*8%7D%7D HTTP/1.1
Host: app.example.com

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8

<p class="greeting">Hello 56</p>

The string {{7*8}} does not appear in the response. It has been replaced by 56, the result of the calculation, which shows that the server evaluated the input.

Three precautions make this discriminator reliable.

• The number 56 must not appear anywhere else in the baseline page, otherwise its presence would prove nothing. That is why an uncommon product is chosen rather than a round number.
• A second attempt with different operands, {{9*9}}, which should yield 81, rules out coincidence and confirms that a calculation took place.
• The expression {{7*'8'}} identifies the engine without escalating the test: Jinja2 repeats the string and returns 8888888, Twig returns 56. Knowing the engine is enough to qualify the potential impact, which is publicly documented for each of them.

The proof stops there. Pushing on to command execution on the server would add nothing to the demonstration of the flaw and would expose the target to additional risk. Escalation is considered only if the authorised scope explicitly provides for it, and it then stays minimal.

Finally, the verifier replays the baseline and both attempts, rules out competing explanations (cache, generic response, dynamic content that happens to contain the number), then records a decision.

3.2 The replayable record

The finding is stored in a format that allows it to be replayed mechanically, for example:

id: F-0007
title: Server-side template injection on /preview (name parameter)
cwe: CWE-1336
target: https://app.example.com       # fictional example
preconditions:
  auth: none
steps:
  - name: baseline
    request: { method: GET, path: "/preview", query: { name: "rootzero" } }
    expect: { status: 200, body_not_contains: "56" }
  - name: proof
    request: { method: GET, path: "/preview", query: { name: "{{7*8}}" } }
    expect: { status: 200, body_contains: "Hello 56", body_not_contains: "{{7*8}}" }
  - name: control
    request: { method: GET, path: "/preview", query: { name: "{{9*9}}" } }
    expect: { status: 200, body_contains: "Hello 81" }
validated_by: "independent verifier (identifier)"
validated_at: 2026-09-08

After remediation, the re-test runs the same steps. The report must distinguish three outcomes. If the baseline passes and the proof fails (the input is rendered literally or rejected), the flaw is no longer reproducible through that route; check whether the root cause was fixed or a filter merely blocks this payload. If the proof still passes, it is not. If the baseline itself fails because the entry point was removed, authentication changed or the page moved, the result is inconclusive, and the report must never present it as a fix.

4. Classes of false positives

Requiring a demonstration rules out, by construction, several families of false positives that are common in automated reports.

4.1 Inference from version

A banner announces a version affected by a CVE. Linux distributions, however, often fix vulnerabilities through backports without changing the upstream version number: the package looks vulnerable from its banner and is in fact patched. The reverse also happens, with hidden or spoofed banners.

4.2 Reflection mistaken for execution

The payload comes back unchanged in the response, in an error message or a pre-filled form field. Its presence does not prove that it was interpreted. For this reason, the discriminator in example 3.1 relies on the result (56) and on the absence of the original input.

4.3 Generic responses

Web application firewalls, uniform error pages and blanket redirects return the same response to any input deemed suspicious. That response can match a signature without revealing any flaw.

4.4 Noisy timing measurements

So-called blind injections often rely on a delay. Network jitter, server load or a cache produce delays of the same order. A timing proof therefore requires repeated measurements and a dose-response relationship: requested delays of 0, 3 and then 6 seconds must produce response times that follow that progression, reproducibly.

4.5 Spurious out-of-band interactions

A server-side request forgery (SSRF) test waits for the target to contact an address controlled by the tester. Other systems sometimes do it instead: link preview bots, email security gateways, scanners that visit the URLs they receive. The incoming connection must be attributed to the application before drawing a conclusion.

4.6 Caches and duplicates

A response served by a CDN or an intermediate cache may come from an earlier request, including one from another test. Duplicates, finally, count the same flaw several times when it is reached through several paths. They are not false positives in the strict sense, yet they still distort the perceived volume.

5. Why over-reporting happens

Over-reporting can start with a detection threshold tuned to find more leads at the cost of false positives. That choice is not itself a mistake; trouble starts when leads are published as findings. A generic tool knows nothing of the business logic, the effective configuration or compensating controls. Many detections compare a response with a signature instead of comparing it with a baseline response from the same application. And without an independent, replayed verification, nothing stops a wrong interpretation from reaching the report.

Machine learning does not solve these points on its own. A model can formulate a plausible hypothesis, and that plausibility still has to be demonstrated like any other. The chain described here applies whatever the nature of the tool that produces the attempt.

6. Limits

Some classes of vulnerability cannot be demonstrated without causing damage: denial of service, race conditions with lasting effects, business logic flaws that involve real transactions. For these, we recommend documenting the hypothesis and the clues as an undemonstrated lead, in a section separate from the findings, and proposing a test in a dedicated environment.

The absence of a demonstrated finding does not prove the absence of a vulnerability. A proof-based report must therefore state its coverage explicitly, indicating what was tested and what was not.

Independent verification has limits. It confirms that the proof replays and that the discriminator holds, without judging business impact, which the recipient of the report is best placed to assess. The report says so explicitly, which leaves risk qualification with its author and prioritisation with its reader.

The proof must stay minimal. Demonstrating access to data does not require extracting it. A proof that collects personal data beyond what is necessary raises a compliance issue under the GDPR (the EU General Data Protection Regulation, here on the site of CNIL, the French data protection authority), whatever its technical value. A record identifier, a row count or a masked value is usually enough.

Finally, no attempt is legitimate outside a scope authorised in writing. Note RZ-2026-2973 covers this point for autonomous agents.

7. An example from our research

In our test of pfSense CE 2.7.2 with ISC DHCP, a client sent a crafted hostname in DHCP option 12. The name was stored in a lease, copied into the ARP table and rendered without encoding by diag_arp.php; script ran when the page was opened. The Netgate pfSense-SA-26_03.webgui advisory confirms the flaw and its fix. It also says Kea sanitises the hostname before storage; our local test did not exercise Kea. That last statement comes from the vendor advisory, not from our experimental proof.

A report should keep those boundaries visible: tested build and configuration, observation, negative control and the source of claims about other versions. A strong proof says exactly where it stops.

References

• NIST, SP 800-115, Technical Guide to Information Security Testing and Assessment, 2008.
• OWASP, Web Security Testing Guide, in particular WSTG-INJT-18, Server-side Template Injection.
• OWASP, Benchmark Project.
• J. Kettle, Server-Side Template Injection, PortSwigger Research, 2015.
• ANSSI, Prestataires d’audit de la sécurité des systèmes d’information (PASSI), the French requirements framework for qualified security audit providers (in French).
• FIRST, CVSS v3.1 User Guide.
• CNIL, The General Data Protection Regulation (in French).