MAGE: A MEASURED RESPONSE TO GENERATIVE AI IN SECURITY ASSESSMENTS

Everyone seems to be building AI Thing™. In cybersecurity, this often means replacing hackers with hackbots. Sometimes this yields positive results. Most of the time it yields interminable slop. Thing is though, we’re hackers, and we like being hackers. We think hackers are still better than hackbots at hacking, but we are also much, much slower. So what if our AI Thing™ could turn our hackers into cyborgs and just make them super fast? The result of our experimentation is called MAGE, and this is how we built it, and why.

Signal, Noise and Slop

The cybersecurity industry is in a bit of a weird spot at the moment.

The AI hype in the space has grown significantly, a massive market has emerged around it, and there’s a lot of super interesting experimentation and development, and a large amount of unmitigated garbage.

We’ve been relatively well-positioned in our own experimentation as well as our engagements with customers in that we’ve had very broad exposure to many of the tools currently vying for air, while not ourselves being too invested in the AI space, allowing us to maintain some degree of objectivity.

Some of the industry’s biggest players have made some massive claims, with even more massive variance in degrees of veracity, with the resultant deluge of slop causing multiple critical software projects to lock down or entirely cancel their bug bounty programs.

Simultaneously, some historical bugs have been uncovered, especially in the last few months, with varying degrees of, but almost ever-present, involvement of LLM technologies.

There is much talk of bots so advanced they will solve cybersecurity, though with costs in the tens of thousands of dollars per bug discovery, and with these models being ever-more locked away from regular people and organizations, there is a question yet unanswered regarding accessibility.

Perhaps now more than ever, it is extremely difficult to tell the signal from the noise in cybersecurity. What is clear is that there is definitely something to be gained from LLMs, and somewhere in this space, there must be a space for them.

Spending significant time in environments of varying sizes from small 20-person outfits to multi-billion-dollar enterprises with a healthy degree of AI skepticism, we have come to a few observations:

1. False positives: LLM tooling has a significant propensity for false positives, and a significant lacking of contextual logical capability to help assess the relevance of any actual finding. Systems that lean more autonomously tend to save little in terms of human effort, they just shift the effort to validation of LLM outputs, sometimes being more time-expensive than just doing it by hand.
2. Security principles: The vast majority of LLM-based security assessment tooling completely ignores basic security fundamentals. We see a general disregard for what happens with data fed to the LLMs, where that data may end up, or what it is likely to contain. We also see a pervasive refusal for real evidence-handling; evidence is often stored as LLM-authored markdown artifacts with no actual hard-truth to back them. Hallucinations are rare but they do happen, and it is the standard in the agentic security field to treat LLM output as untrusted and adversarial, so why is it that our offensive tooling tends so often to treat these artifacts as canonical?
3. The human element: The majority of LLM-backed security tooling is centered around removing the human-backed security skill and knowledge from the equation, even while most evidence seems to point to this remaining the foundation of most of the really cool bugs found by really cool bug hunters using AI.
4. Offensive AI tooling is necessary: There is significant need for good offensive AI tooling. The amount of code being produced has increased by several orders of magnitude, the amount of bugs in that code, even more so. Cybersecurity as an industry has functionally been in a perpetual labour shortage since the industry began, but at this stage, a single AI-enabled engineer is capable of pushing out enough code in a single day to keep a dozen security researchers busy for a week. This profound gap in workable volume means more bad code gets shipped because there isn’t time to review it properly, and the overall security position of products that adopt AI worsens meaningfully as a result regardless of how effective the LLM is at producing that code.
5. AI makes you dumb: Much ink has been spilled over the cognitive offload and how it negatively affects people’s ability to do tasks themselves that they have been offloading to LLMs, as well as how the temptation for offload bleeds into other tasks, sapping these skills from tangential areas of knowledge as well. People get worse at things extremely rapidly the more they use AI.
6. The effect on our art: Finally, generative AI technologies have a propensity to destroy art. It’s not inherent and it’s not unavoidable, and there are a few very cool projects around that do use LLMs to create genuine art in all spaces, but the degree of cognitive and creative offload that LLMs incentivise and providers desire typically leads to this outcome. There is a moment in every good assessment where you are scrolling through thousands of logs and you see a weird error message, a stray field, or something. Your spider sense tingles. You poke at it, and you walk away with a cool finding. These instincts are the kind of thing that pure automation tools miss, and they’re the best part of the job. Hacking is our art, and we must preserve this moment, both for the sake of the craft, as well as for the sake of the customers who have paid for our time to help secure their code.

These are not unique takes. In fact, most of the industry, if they are operating in good faith, are currently trying to figure out “what does good LLM-enabled security tooling look like?”

We think it might look a little like MAGE.

Cyborg, Not Robot

MAGE is a harness we built to help out with security assessments, with the core principles:

1. Automation is not the enemy.
2. A tool is a tool, and should be treated like one, employed by skilled security researchers to make them much, much faster.
3. Cognitive offload is the purpose of every tool, whether that is sqlmap, joern, or a moleskine notebook. That offload, however, needs to be focused in service to human skill, rather than in replacement of it.
4. An LLM is a good tool for a wide variety of contexts, it is a terrible tool in many more, and in a vanishingly small number of cases, is the perfect tool for a given context. MAGE must always endeavour to place an LLM where the LLM is an at-least-good tool for the job.

MAGE must be built to enhance human capability, not replace it.

In short, to quote our favorite phrase, “Think Cyborg, Not Robot“.

MAGE’s architecture is not wholly unique, as it is functionally assembled, and also in some cases literally assembled, from various parts of other projects we did like, stitched together as a web of microservices, which looks a little like this:

• Frontend: An Electron application (we know), forked from an open source project, which houses GUI functionality, like chat windows, graph and timeline representations for findings, an integrated shared browser, etc. It’s supposed to be a nice user-friendly UX over all the capabilities and function as a generalized situation-monitoring console for the stack.
• LLM Comms: Also forked from an open source project, this part handles communications with the actual LLM providers, MCP, tool calling and whatnot. We forked this to make sure we could support whatever inference a customer is already using, or comfortable with using. We can piggyback off their Bedrock, use frontier models directly, or even run off a local GPU cluster if we want. This gives customers data privacy and an assurance that an assessment with MAGE does not increase their actual data footprint beyond what they’re comfortable with. We’ve stripped this down quite massively from the open source project we stole it from, to where it is now functionally just a router between the LLMs and the actual workhorse services.
• Backend: This is a Rust project with an embedded project-specific database, so there is no potential for bleed over from other projects. It also has a full text search index mainly for quick human search over the database as well as a semantic search layer in front of it for better agentic search capabilities, to make it a trustworthy central information store used collaboratively between humans and agents.
• Async Runner: The agent receives a list of available tools depending on a session profile (Web or Code at time of posting) where the tools are a curated list of normal tools we as hackers use in our normal engagements, but we, where possible, try to avoid giving agents access to broad-impact tools like bash which are difficult to control, and these are instead isolated to specific agent profiles which need more flexibility. All tool calls enter a work queue in the database, where they are picked up by an async runner service so they can be run parallelized, sanitized (to some degree), and monitored. The exact tool call inputs and outputs, like the command, args, stdout, stderr, etc. are recorded in the database layer without ever touching an LLM, so we know that the evidence therein is exactly representative of the facts of any specific tool run.
• Other situational microservices: There are some other microservices, like a local intercepting proxy, which, when enabled, will route network tool calls through the proxy via the async runner, as well as a Joern sidecar for code analysis.

These are stitched together into a workflow that incentivises tasks to be offloaded asynchronously as much as possible, so tool runs don’t block the main chat thread wherever it can be avoided. The early stages of information gathering and tools-heavy assessment are largely automated and left in the background, while the user’s curiosity gets to stay in the foreground, and when the assessment gets more interesting and in-depth, this allows everything to remain responsive to that same curiosity.

A centralized background findings flow handles a large amount of the spam we get from tools-based analysis and helps speed up analyst workflows. When a finding is logged, either by a tool, an agent, or a human analyst, it enters a work queue where it must be proven exploitable by one agent, and then a configurable number of agents must reach quorum on whether or not that exploiter’s work is valid or not. This helps significantly to reduce false positives, and also helps the analyst’s spider sense, since they can log a finding, and effectively get a second, third, fourth, and fifth pair of eyes on it to validate it before it goes in the report.

View of the findings and exploitation validation graph

Silicon Crawls, But It Can’t Tingle

After a ton of testing, in one of our first real assessments using MAGE, using a frontier model with cyber capabilities, something really cool happened. The researcher asked the LLM to do some basic discovery and see if anything interesting appeared. The agent didn’t find anything worth mentioning really, but the researcher, scrolling through the logs, did. Their spider sense tingled and they followed that curiosity, and in collaboration with the agent, were able to get a significant privilege escalation finding due to a logic error that the model completely missed. This was exactly what we were hoping for, and it was the moment we knew that we were at least along the right track towards what we wanted out of MAGE.

This isn’t perfect. We have not “solved cybersecurity”. There are some places where we’d definitely like to improve to ensure that MAGE fits our vision more tightly, but our real-world assessments conducted with it and with our partners have so far yielded genuinely positive results.

We’re not going to be publishing it publicly. MAGE is an internal workflow enhancement tool to be optionally included with our regular assessments should customers want an AI component to their testing, it is not a product or a service we are selling in-and-of itself. Maybe, you could even just take what we’ve written here and iterate on it, build your own internal AI Thing™. If you do, we’d love to see it, maybe you build something better that we can learn from, because our art is communal and iterative.

Alternatively, if you’d like to see MAGE in action, reach out for an assessment 🙂