AI vs. Human Pen Testers: What the Data Shows, and Why ITECS Hosts Its Own AI

Security leaders now ask a fair question about AI in penetration testing: can it find more vulnerabilities than a skilled human? Recent independent research gives a clear answer. On breadth, speed and cost, the best AI agents now rival most professional testers. On accuracy, depth and judgment, the top humans still lead. The strongest programs combine both, and how the AI is hosted decides whether you keep control of your data while it works.

What the research shows: AI agents versus human testers

What its authors call the first comprehensive evaluation of AI agents against human cybersecurity professionals comes from researchers at Stanford, Carnegie Mellon and Gray Swan AI, published at ICLR 2026. They tested ten cybersecurity professionals, six existing AI agents and a new multi-agent framework called ARTEMIS on a live university network of roughly 8,000 hosts across 12 subnets. Human participants had up to 10 hours of active testing, and the AI runs were scored on their first 10 hours (Lin et al., “Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing”).

Source: Lin et al., ICLR 2026 (arXiv:2512.09882). Cost figures are the authors’ estimates; the human rate is about $60 per hour, based on the average U.S. penetration tester salary the authors cite.

Two details matter as much as the headline. The ten human testers together found 49 unique validated vulnerabilities, far more than any single participant. And about 80% of the humans found a remote code execution vulnerability on a Windows machine reached through a web-based remote console, which ARTEMIS struggled to exploit because it required working through a graphical interface. The authors report that AI agents “exhibit higher false-positive rates and struggle with GUI-based tasks,” while top human testers were more likely to pivot and deepen a foothold instead of reporting a finding immediately.

Other evidence points the same way: AI adds remarkable speed and volume, and confirmed impact still needs expert validation.

Where AI wins, and where humans still win

Read together, the studies describe a division of labor rather than a replacement. The left column is where AI changes the economics of testing; the right column is why a human still signs every finding.

How ITECS combines private AI with human testers

ITECS runs two large language models on GPU servers in its own Dallas data center, each with a defined role in an engagement:

• GLM-5.3, reasoning and planning: correlates reconnaissance, ranks targets and reasons through multi-step attack paths across networks, cloud and identity.
• APEX, offensive operations: runs in-scope testing at machine speed and drafts reproducible proof of each potential finding for our testers.

The models take on the enumeration and triage that used to consume days, so testers spend their hours on business logic, chained attacks and impact. AI proposes; a human reproduces every finding by hand before it reaches your report, so your team never chases a hallucinated vulnerability. That is the answer to the false-positive problem the research describes.

The approach applies across ITECS security testing services:

• Penetration testing: external, internal, wireless and cloud attack surfaces, as a point-in-time test or continuous penetration testing as a service.
• Vulnerability assessment: broad discovery and risk-based prioritization, complemented by the same private models.
• Web application and API testing: code-level and business-logic protections across customer portals, APIs and integrations, mapped to the OWASP Top 10.
• AI and LLM application testing: the assistants, agent harnesses and integrations you deploy, including the tools, permissions and data they can reach.
• Remediation: findings delivered on the date set at scoping with a remediation workshop, critical findings reported as soon as they are confirmed, and retesting included within the window of your package.

Why we host our own models in Dallas

A penetration test handles some of the most sensitive information an organization has: network maps, credentials, unpatched weaknesses and proof of how to reach critical data. Sending that material to a public AI service means trusting a third party’s retention, logging and training policies with your attack surface. ITECS built its AI capability so you do not have to make that trade.

The same concern applies to the AI systems you deploy. The OWASP Top 10 for LLM Applications (2025) ranks prompt injection first and sensitive information disclosure second, and it lists excessive agency among the top risks. Testing an AI assistant means testing what it can be talked into revealing or doing, and doing that work on private infrastructure keeps the test itself from becoming an exposure.

Questions to ask any AI-assisted testing provider

AI-assisted testing is quickly becoming standard. Before you sign, ask these questions of any provider, including us:

The bottom line

The data no longer supports either extreme. AI agents are not a novelty: the best of them already outperform most individual testers in overall results. They are not a replacement either: the best humans still find more, report fewer false positives and understand business impact. The right model pairs AI coverage with human proof, on infrastructure you can trust with your attack surface.

Sources and methodology

• Lin et al. — Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing — Stanford, Carnegie Mellon and Gray Swan AI; ICLR 2026 (arXiv:2512.09882)
• DARPA — AI Cyber Challenge final competition results — August 8, 2025
• CSO Online — The top red teamer in the US is an AI bot — June 25, 2025
• Zhu et al. — CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities — arXiv:2503.17332, 2025
• HackerOne blog — analysis of the Lin et al. study — vendor perspective, December 17, 2025
• OWASP — Top 10 for LLM Applications 2025

Methodology: Comparison figures come from the independent studies above; ITECS has not yet published aggregate metrics from its own engagements. Descriptions of ITECS models and services reflect current ITECS operations. Prepared by the ITECS team with AI-assisted research on October 25, 2026.