AI pentesting works when its output survives four checks: the same target run twice produces findings that match, your team's manual validation workload actually falls, a human sits on the escalation path for anything ambiguous or destructive, and the findings it surfaces get fixed, not filed. Score a vendor against those four before coverage claims or price enter the conversation.
The four-measure scorecard
| Measure | What it checks | Weak signal | Strong signal |
|---|---|---|---|
| Repeatability across runs | Same target, same scope, run twice: do the findings match? | Finding count or severity shifts materially between runs, with no explanation offered | Findings converge run over run, and any drift is explained (target changed, a patch landed) |
| Validation effort left with your team | Once a finding is flagged, how much manual confirmation does your team still have to do? | Every finding needs your own reproduction before anyone trusts it, so the workload has moved, not shrunk | Findings arrive with evidence, request and response, or reproduction steps, your team can check quickly rather than remake from scratch |
| Human escalation path | What happens when the tool is uncertain, or hits something destructive? | The tool proceeds on ambiguous or high-risk actions with no defined stop point | A named person or role is the escalation point, uncertain and high-risk actions pause for them, and that pause is logged |
| Proof findings reach remediation | Do flagged findings turn into tickets that get closed, or do they sit in a report? | Findings live in a PDF or dashboard with no link to a ticket or an owner | Each finding maps to a tracked ticket, an owner and a close date, and that mapping can be pulled back out later |
A worked example
Run the same external-facing test twice, two weeks apart, nothing else changed on your side. Fifteen findings the first run, eleven the second, four gone with no explanation. That is a repeatability fail: score it weak, and ask the vendor why before you look at anything else. Of the eleven that held, three needed a day of manual reproduction before your team trusted them, which means validation effort is not falling, it is moving from the tool to your analysts. One finding touched a production system directly: check whether a human paused it before it ran, and whether that pause is on record anywhere you can retrieve later. Two months on, pull the ticket tracker: if none of the eleven map to a closed ticket, the tool found things, it did not fix anything. Four measures, one afternoon of checking, and you know more about a vendor than any coverage claim in their deck tells you.
What the primary sources say about each measure
None of the four measures is new. They are the same tests the established guidance applies to human-run testing and to AI evaluation, carried over to a tool.
Repeatability. The NCSC penetration testing guidance says a test "can only validate that your organisation's IT systems are not vulnerable to known issues on the day of the test". That is a limit for a human team too, but a tool that runs continuously makes the day-to-day variance visible, so you have to measure it. The evaluators do exactly that: the UK AI Security Institute, when it tested frontier models on cyber tasks in July 2026, gave each model five attempts per task and reported results over ten runs with the minimum to maximum range shown. If a vendor cannot show you the same kind of spread across runs of its own tool, it has not measured repeatability, so it cannot claim it.
Validation effort. The NCSC guidance is blunt that "risk assessment and decisions on the application of fixes are your responsibility", not the tester's. The finding has to arrive in a form your team can check quickly: the request, the response, the reproduction steps. Otherwise the tool has produced a list, not evidence.
Human escalation. The OWASP Top 10 for LLM Applications entry on excessive agency gives the control in one line: "require a human to approve high-impact actions before they are taken". A tool that exploits a production system without a pause point is the textbook case.
Remediation. The NCSC guidance treats scoping and the handling of findings as the buyer's job, with risk owners, technical staff and testers agreeing the technical boundaries before the test starts. Agree the ticket owner at the same time, and the fourth measure has something to check against two months later.
SenseOn does not sell, resell or benchmark AI pentesting. Automated exploitation tooling and the AI that governs SOC detection and response decisions are different categories of product, and this article is not a way in to a pitch for either. The scorecard above is offered as a starting method: run it against any vendor you are evaluating, including ones that have nothing to do with SenseOn.
Sources
- NCSC, Penetration testing, published 8 August 2017, reviewed 10 January 2022, accessed 20 September 2026
- UK AI Security Institute, How far behind the frontier are leading open weight models on cyber?, 17 July 2026, accessed 20 September 2026
- OWASP GenAI Security Project, LLM06:2025 Excessive Agency, accessed 20 September 2026
Related reading:
How SenseOn writes, checks and corrects its content: editorial standards.
