As with any recent domain, I’ve heard the term “AI pentesting” more often than one would want to. From my perspective, it feels tiring to hear that artificial intelligence touches every conceivable subject.

While I am on the side of using it consciously, if at all, I will try to keep my biases in check.

We will first have to define what automated pentest means, what people want it to mean, and what I hope it will mean in the future.

So for now, automated pentesting has basically automation that runs repeatable security checks. With lately, of course, the addition of a language model that helps with the results: explanations, summaries, and reporting.

What people want from it

End‑to‑end penetration test, fully autonomous, taking decisions, basically an ethical hacker LLM.

How can we improve it:

• We can, of course, add an interface for the scanner with a dashboard.
• A report generator that sounds confident, and hopefully is also corect in generating critical decisions
• An “agent” that decides at every major step of the pentest process.

At the moment, as far as I’ve explored existing tools, automated pentest is more static than dynamic. Meaning even if the tool detects, for example, Apache as the server, it usually would try to run various explorations for Tomcat or JSP, for example.

So here is one of the real improvements AI can do with the autonomous pentest process, that is actually decide what tools to use, how relevant it is to continue reconnaissance if it already found all the info for next steps, and when to stop specific processes if they are not that useful.

How I want to see it in the future

Having been part of this industry for over 15 years, my bias is certainly evident. Because in the end I would want an autonomous software making my life easier, and me just sipping some lemonade on a tropical island, while I make sure the applications are secure. But the future might be a lot different, and possibly neither the best case scenario will happen.

So at this moment, an “AI” pentesting tool that tries to do more than summarise, explain, and check will eventually encounter a lot of issues along the way. And like it or not, this is just the LLM’s inherent weakness.

Where automated pentest tool can it fail from leveraging too much “AI”:

• Software changes (way) too often. We can have a process ran by the LLM, with a flag or output format that is not available anymore, or deprecated.
• A subtle difference, not a hallucination, that builds upon it, can end up with a broken analysis and though process. Eg: it hallucinate an IP that is not actually uncovered by nmap

• A critical step process that the LLM does not have access to.
• Security implications of running powerful commands on fully automated software.

Automation has been part of pentesting for years

Automation already covers a lot in the pentesting process. From reconnaissance, impact all the way up to exfiltration. Having certain scripts that can be repeated in different environments. Building reusable snippets so reports don’t become too time-consuming, especially in an environment where pentest is paid by the hour.

So none of this is new, not even in the tech industry. Everything explained above is just speeding certain tasks. Or when running a time-boxed penetration test it helps to even try running multiple processes in parallel with manual analysis. Whenever it’s a more complex reconnaissance or its just code analysis.

Actual threats are hard to find in an autonomous way, as well as just keeping a huge database of existing threats, or having a mix, or adding heuristic and non-heuristic analysis to not only match existing but similar or adjacent patterns. Too many fancy words to say it’s almost to catch new attack vectors, or innovative ways to write malware code.

Take shai-hulud for example. It was so successful and still is because it hid in plain sight. It used legitimate ethical hacking tools and development tools. When a hacker thinks like a developer or is a developer, the possibilities for malware are endless, and at this moment, no amount of automation and LLM can do the job of someone dedicated to this.

It’s really a paradox. If the LLMs (agentic or not) are as good as an ethical hacker, the bad news is that the same LLM can(will) be used to do harm as well.

I could even state that at this moment, threat actors are better leveraging AI than ethical hackers. Even if this statement is just based on my feeling and assumption.

Going back to a real pentest. These have constraints:

• Scope boundaries.
• Business context.
• Fragile systems.
• Access levels that change mid-week.
• The need to decide what matters today versus what can wait.

Scope boundaries are important because even if we can hope to automate everything and actually do so, there is still a lot to be done to make sure that we don’t cross any ethical, and legal boundaries. Will explain further.

Case study

Let’s say a client has an app, or server, or a device it wants security tested. The scope document says “test that device, domain or IP.” Seems simple.

But applications almost never exist in isolation. To list only a few possibilities, applications can:

• connect to a payment gateway,
• have their server pull data from a third-party API,
• connect to a vendor’s cloud infrastructure.

None of that is yours to test. Unless specified explicitly.

I’ve been in (and not in one) situations where careful execution and double and triple checking what I assessed and attacked was the only thing that kept me on the right side, or at least the legal side. During a sub-domain enumeration scan that I thought was targeting a client’s infrastructure, I discovered mid-process that some of the targeted subdomains were in fact hosted by a completely different organization. The client was not the owner. The positive news is that I was only conducting a brief reconnaissance, but I was quite close to crossing a boundary.

On another occasion, I traced a dependency chain while analyzing code. The trail led to an external public package repository. And one more automated request was needed, and I would have been probing infrastructure that, goes without saying it did not give consent to me for security testing.

And here’s the ugly part, where autonomous security testing gets really dangerous. An LLM deciding next steps won’t hesitate at that boundary, at least not without many, many layers and checks.

An LLM doesn’t feel the weight of “Wait, should I be here?”, or have that gut feeling. It follows the thread because following threads is what it was programmed to do.

What was white hat work becomes grey. Or worse.

Conclusion

The industry will keep pushing toward autonomy because that’s where the money is.

• Defence teams want security coverage.
• Vendors want scale.
• Everyone wants speed.

The uncomfortable question is: will AI get better or cost-effective to ask the least?

Photo by Kevin Ku on Unsplash