Knowing what not to attack

Most talk about offensive security agents is about power. Can it find the bug? Can it chain the exploit? Can it own the box?

That’s the easy part. Wiring a model up to tools is normal engineering now, and raw capability improves on its own every time a new model ships. The parts that don’t improve on their own are measuring the thing and holding it back. Those stay hard and they’re where these projects actually fall over.

Everything below comes from how these systems work, not from any vendor’s roadmap. If you have seen this argument built on quotes from a job ad, ignore that part. Recruiting copy tells you what a company is hiring for. It doesn’t tell you what’s true.

“Knowing what not to attack” sounds like one skill. It’s two! and they need completely different plumbing.

The first is allowed or not allowed. Is this host in scope? Is this credential meant for this engagement? Is this class of action permitted at all? That’s a yes/no check, default deny, and it runs before the agent starts thinking about options. It belongs in infrastructure: allowlists, egress rules, scoped credentials, capabilities that simply aren’t there. Not in a prompt.

The second is what next. Out of the moves that are allowed, which one do we try? That’s a real decision problem with bad information, and it changes as evidence comes in.

Mixing these two is how an agent talks itself out of scope. If “blast radius” is just a number you subtract, then a good enough exploit path buys a big enough blast. That’s exactly the behaviour you were trying to stop. An out-of-scope target isn’t low priority. It’s impossible.

To be clear about where the line sits: inside the allowed set, you can absolutely weigh cost and impact and prefer the quieter option. That’s fine. What you can’t do is let a high enough score reopen a door that authorization already closed.

And scope enforcement fails too. Routing, DNS, proxy config, identity mixups. Test it like any other control.

You’ll see something like this passed around:

value * likelihood * information gain - cost - blast radius - risk

It looks precise. It isn’t.

Information gain is measured in bits. Value is measured in outcomes. Multiplying them means nothing. The thing you actually want is value of information: how much better will my next decision be if I learn this? A probe can be very informative about a question that doesn’t matter. And value of information should be added, not multiplied.

Blast radius is part of impact, and impact is part of risk. Subtracting both counts it twice. Likelihood probably gets counted twice too, though you can’t say for sure because nobody defines these terms. That’s the real problem: the words are undefined, so the arithmetic is decoration.

The deeper issue is that the numbers don’t exist. Nobody has shown well-calibrated success probabilities against a target they’ve never seen. Population level scoring like EPSS works fine as a prior across a lot of CVEs, but that’s not the same as “this exploit, this box, 0.7.” Assume your agent’s confidence numbers are uncalibrated until someone measures them. A madeup number that goes into a decision turns a guess into permission.

What works instead is coarser and less impressive. Fixed policies per action type. Hard budgets. Ordered heuristics. And a rule that the agent has to say what it thinks is true and what observation would prove it wrong, before it acts. Being able to read the reasoning beats fake maths.

None of this is smarter than what a good human red teamer does. It’s the same judgement. The difference is that with an agent you have to write it down, log it, and test it. That’s worth something, but it’s not a new capability.

One more thing. “Which single action next” comes from human economics, where every action costs a person’s time. Agent actions are often cheap and can run in parallel, so probing wide and pruning fast can beat committing to one path. Depends on how noisy you can afford to be.

“I built an autonomous security agent” is a demo. “I built an eval that shows where it breaks” is engineering. Here’s what makes the second one hard.

Ground truth. Seeded bugs tell you the agent finds the bug classes you thought to plant. They tell you nothing about the classes you didn’t think of, which is the miss that matters on a real assessment. No findings is not the same as no vulnerabilities.

Watch out for the reverse too: if your agent finds something real that nobody planted, a naive scoring script marks it as a false positive. Your adjudication needs room for that.

Contamination. Environments built from public CVEs, CTFs and famous vulnerable apps are probably in the training data. A high score there might be memory, not reasoning. Private purpose-built ranges fix that specific problem and nothing else. Report both, separately.

Variance. These systems are random. One good run is a story, not a result. Run it many times, report the spread, and say which rule you’re using  “worked at least once in 8 tries” and “works reliably across 8 tries” are two different products. Also be honest about power: with pass/fail outcomes and 5 or 10 runs, the error bars eat most of the differences people brag about.

Cost. A success rate with no cost attached can’t be read. Wall clock, compute, actions fired at the target, human review minutes. Two agents with the same success rate and a 20× cost gap are not the same tool.

False positives need a denominator. Per finding? Per run? Per claim inside a report? And who decides? A human decides, so rater agreement is part of your instrument. What you actually care about is triage cost per confirmed finding. Also decide how you handle duplicates and severity, or an agent can win by reporting the same thing nine ways.

Give credit for evidence. A reproducible, safe demonstration counts. “This may be vulnerable” does not.

Baselines. Compare against scanners, scripts, plain workflows, and people, with matched time, tools, and access. Human to human variance is huge and authorized testing is expensive, so show your tiny sample sizes instead of hiding them in an average.

Regression. Pin your images, snapshot, seed. Otherwise a “regression” might just be the environment moving under you.

Gaming, both directions. Agents do exploit harnesses. They read answer keys, attack the scoring service, escape the sandbox. That last one is funny in a bad way: the agent produced a real finding and destroyed your measurement at the same time. Defend structurally the agent gets no write access to the harness, ground truth, scoring logic, or logs. Grading criteria stay hidden. Held-out environments rotate. Auditors read full traces, not summaries. If the agent can move its own score, the score isn’t evidence.

The failure people talk about less is yours. Proxy metrics drift away from the goal, and iterating against the same suite overfits it. More findings is not more security. An agent that floods the triage queue makes the process worse. Treat your eval suite as something that decays.

This is the gap I see most often. Everyone lists safety controls and nobody counts them.

Put them in the same harness as capability. How many out of scope reaches? How many guardrail hits? How many approvals overridden? How many actions outside the allowed classes? If you skip this, you’ve shipped a fast car with an untested brake.

An offensive agent lives on untrusted text. Banners, error strings, file contents, source comments, HTTP responses. Models still don’t reliably tell instructions apart from data, so injection against your agent isn’t an edge case. It’s the normal working environment.

A baited target is talking to a process that holds credentials, network access, and a shell. So scope has to be enforced structurally, not in the system prompt, because the prompt is attack surface. And this belongs in the eval suite: can content from the target push the agent out of scope or past a guardrail? If you haven’t tested that, your suite isn’t finished. The agent’s own tools and dependencies are in the threat model too.

One thing that gets lost when you frame everything as “uncertainty”: a live defender adapts. WAF rules get tuned, credentials get rotated, IR contains you halfway through. That’s not a fixed environment with hidden state, it’s another player. It also breaks the pinned and reproducible assumption the moment you leave the range.

Related: “read-only is safe” is false. Reads create data-handling duties. Enumeration causes outages. Any action can trip detection and burn someone’s incident response budget. Deconfliction with the defenders is a control, not a courtesy.

And reversibility is often something you can’t know at the time. Writes propagate. Data, once copied, stays copied.

Stopping the agent must not depend on the agent agreeing to stop. Termination lives in the orchestrator, the network policy, and the credential system, not in a tool the model can decline to call.

Logs must be append-only and outside the agent’s write scope. Otherwise they’re notes, not an audit trail.

Human approval is a ratelimited resource, not a safety property. Under volume it becomes a rubber stamp. Pick a small, stable set of actions that need sign-off, make each request carry the intended action, expected side effects, supporting evidence and a rollback plan, and enforce everything else by mechanism.

“The model wasn’t smart enough” is where unexplained runs go to die.

The usual list is right: reasoning error, missing context, wrong tool picked, confusing tool output, lost state, bad memory, orchestration bug, made up evidence, a guardrail firing correctly, broken harness, wrong ground truth. Fine as far as it goes, but that’s the easy ten percent. The categories overlap, and a bad tool interface looks exactly like a reasoning failure from the outside.

Real attribution needs instruments. Full traces, replay, and ablations. Swap the tool schema and re-run. Swap the context and re-run. Swap the model and re-run. Without that you’re guessing with confidence.

Don’t pick an autonomy level up front. Start narrow and widen it only when your evals show the agent stays inside policy and produces findings that hold up. That’s a safety case, not a feature roadmap. The goal was never maximum autonomy it’s bounded autonomy with evidence behind it.

Worth saying out loud: when an agent goes out of scope, the liability is yours. Not the model’s, not the vendor’s. Giving it more room is a decision about who carries that. Your eval artifacts are sensitive too  a suite that proves an agent can chain X against Y is a useful document for the wrong reader.

Whether you’re hiring or building, the things that actually signal competence:

• Capability reported as a distribution, not a demo
• Controls measured as carefully as capability
• Failure attribution backed by traces and reruns, not a story
• Scope enforced in infrastructure, not in prompts
• Injection resistance tested, not assumed
• A policy that lets the agent do nothing, and evidence that when it did nothing, it was for the right reason

Here’s the one question I’d ask. Have them explain a single failed run in detail. A good answer separates model error from missing context, wrong tool, bad interface, lost state, broken orchestration, made up evidence, a guardrail working as intended, and a broken harness and points at the reruns that told them which one it was.
A weak answer blames the model.