“Why not just prompt Claude Code or Copilot to triage dependency alerts per our security policy?” is the right question. A coding agent can read your code, trace a call path, and reason about whether a vulnerable function is reachable. That is the hardest part of alert triage, and the agent is genuinely good at it.
We know because we built our triage system that way. The first working version took about a week: an agent, a rubric in a prompt, and a plan to score every alert on reachability and exploit evidence. It produced verdicts that looked reasonable.
Then we started measuring, and the measurements changed our architecture. Run after run against our own products and real open-source backlogs, with hand-labeled ground truth, we found failures that don’t announce themselves in the output and squeak through informal inspection. If you have tried agent-only triage on a real backlog, you probably hit at least one of them.
Here are the six problems that forced us to move work out of the prompt and into a server, with the numbers that convinced us. At the end you can check our results against your own workflow.
1. Sub-agents skip the method, and nothing in the output tells you
Point an agent at a large backlog and it fans the work out to sub-agents. We measured what those helpers actually do: across 6 large-target runs on Redash (272 alerts, Claude Sonnet in Claude Code, July 2026), 0 of 62 delegated helpers ever read the scoring procedure they were instructed to follow. They produced confident, well-formatted verdicts from memory or from the orchestrator’s paraphrase.
We tried to fix it with better prompting: an orchestrator paraphrase, a delegation template, a file-passing protocol. Three escalating interventions each measured zero: 0 of 7, then 0 of 14, then 0 of 15 helpers read the procedure. Instruction does not survive delegation.
The trap is that every one of those runs looked fine. The verdicts were formatted correctly and mostly landed in safe buckets, so any output-only check passes. We considered this an existential problem for the whole approach until we solved it, and the solution is not a prompt. The k9 MCP server now mints a fresh token each time an agent actually reads the analysis procedure, and the score_risk MCP tool rejects any call that arrives without a valid one. Each helper has to read the method itself, every session, or the call fails with instructions on how to fix it.
2. Threat evidence silently vanishes
Exploit intelligence is keyed by CVE. But many of your alerts arrive as GHSA, GO, PYSEC, and other identifiers. Somewhere in the workflow, something has to resolve those identifiers to CVEs, or every KEV and EPSS lookup quietly returns nothing and every alert scores “no known exploitation.”
With the resolution step written into the prompt, we measured it failing: 3 of 7 runs over the same 272-alert backlog never performed alias resolution at all. The skips did not correlate with run length. It is stochastic attention, not a wording problem, and the output gives no hint that the threat column is empty for a reason.
So we moved resolution server-side. Every score_risk call now resolves the advisory through a curated catalog and consults every CVE it carries, keeping the one that establishes the most threat. On one repo measured after the change (July 2026), 98 of 104 alerts resolved an alias; all of them would previously have scored with unknown threat.
The catalog work matters too. Across our 610,260-record vulnerability catalog, 91 advisories carry more than one CVE, and 29 of those differ in severity across their CVEs. On 2 of them, the canonical CVE is absent from KEV while a sibling CVE is listed. Reading only the id in the alert reports “not exploited” for an advisory that is being exploited.
An agent alone has none of this: no KEV feed, no EPSS refresh, no advisory catalog, and no way to know what its lookups missed.
3. “Undetermined” presented as analysis
Ask an agent to assess reachability across a backlog and the careful-sounding answer is often “couldn’t determine.” On one of our own 55-alert repos, an early rubric version came back undetermined on 47 of 55 alerts. The agent’s own summary then proposed “a one-time call-graph pass to retire the unknowns.” It knew the work was doable. But it skipped it. Early lunch?
Defaults are the same failure wearing a suit. In our measurements, alerts scored with no asset context at all stated the defaulted values as fact, and 7 of 9 observed undetermined alerts carried confidence: high. An alert nobody analyzed scored byte-identically to an alert proven safe.
Fixing this required procedure and enforcement together. A numbered basis-selection procedure with worked examples cut that same repo from 47 undetermined to 5, then to 0. The server now computes quality flags on every batch, so an agent that returns more than 10 percent undetermined or leans on defaulted context gets told, in the tool response, to go resolve it. And an alert with no threat evidence and no reachability conclusion on a critical asset does not slip into the DEFER pile: a dedicated policy floor routes it to REVIEW, because “we never looked” must not score the same as “we looked and it is safe.”
4. Reachability claims that need a schema to reject them
Some reasoning errors recur so reliably that prose instructions cannot stop them. Three we measured:
- Exposure argued as reachability. One early run marked 16 build-time dependencies “not reachable because they run on isolated build infrastructure.” That is a network exposure argument. The vulnerable code runs every build; reachability is about whether attacker-influenced input can reach it.
- “Grep found no imports, done.” Zero direct imports is necessary evidence, not sufficient. Flask’s CLI imports python-dotenv. Alembic imports mako. Your test runner executes packages your application never loads.
- Precondition hand-waving. Sometimes attacker input reaches the vulnerable function but the advisory’s precondition for harm is absent, say the vulnerable behavior sits inside an
ifblock the configuration never enables. Agents filed that shape under whatever basis fit the verdict they wanted. The claim was unauditable either way.
Our answer is a controlled vocabulary. Reachability is a structured claim: a tri-valued answer (reachable, not reachable, undetermined) plus a basis drawn from a small set of defined, mutually exclusive bases with stated precedence, like manually_traced_call_graph, no_attacker_controlled_input, or advisory_precondition_unmet with its precondition named. Cross-field validators reject the conflations outright. An agent cannot submit “unreachable because isolated infra” anymore; the score_risk tool returns an error instead of recording a wrong answer.
5. Same alert, different verdict, depending on the run
Run agent-only triage twice on the same repo and you can get materially different work queues. Across three identical Claude Sonnet runs on the JupyterLab stack, the REVIEW count went 9, 9, 2 and the SCHEDULE count went 0, 0, 8. A team reading any single run would have planned a different week than a team reading another.
We no longer describe this as the model being flaky. When we traced the unstable verdicts, the model was missing one of two things. Either it lacked the information to analyze the vulnerability consistently: the deployment context, the full advisory text, the dependency’s source. Or it lacked a clearly-defined vocabulary to express what it found. In both cases the model does what models do with a gap: it improvises, confabulates, or gives up, and which one you get varies by run.
Both gaps are closable. The customer authors a risk-context.yaml that pins the deployment facts. The server supplies full advisory bodies from its catalog, with no internet fetches to vary. The basis vocabulary gives the model exact words with defined precedence, so two runs that found the same thing say the same thing. And our evals verify the remaining analysis is right for the right reason, against hand-labeled ground truth, at N of at least 3, because a single good run is not a measurement.
6. The risk score is computed from memory, and then it evaporates
Even with correct reachability analysis, an agent-only workflow still improvises twice more.
First, the score itself. Ask a model to weigh reachability, exploitability, exposure, and asset impact into a risk score “per NIST SP 800-30” or your organization’s custom risk scoring function and it does that math from memory, differently each time. And risk is not reachability plus exposure alone: security teams apply policy.
A known-exploited vulnerability on a crown-jewel asset gets attention regardless of a clean reachability argument. We modeled those rules as explicit policy floors, five of them. And score_risk computes the floors and the SP 800-30 assessment deterministically. Same inputs, same verdict, every time, with every activated floor listed in the output.
Second, the record. A prompt scores an alert and throws the verdict away in a chat scrollback. Two months later a security reviewer asks why an alert was dismissed, and the answer is gone. Every score_risk verdict is captured: the evidence, the basis, the floors, and the rubric version that produced it, per alert, per execution context. (Your source code isn’t captured.) That corpus answers the two questions leadership actually asks: how many exploitable and reachable issues are in production right now, and how much scanner noise did scoring eliminate. Internal builds probably don’t get this far: the scorer ‘works’, the project stops, and nothing reports upstream.
Keep the agent. Give it the other half of the system.
None of this argues against your coding agent. Coding agents are fantastic but they can’t triage dependency alerts reliably without significant help. The division of labor that survives all six problems is the one we run: the agent analyzes reachability (with 3,000+ words of guidance), because it can read your code, and k9 scores, because scoring needs maintained evidence feeds, an enforced method, a controlled vocabulary, deterministic math, and a durable record. You keep Copilot. You keep Claude Code.
What changed our minds at each step was measurement, and that is our advice whichever way you go. If you build your own workflow, build an eval test suite that runs the workflow at least three times on the same project and alert backlog, then count what varies. Check whether your helpers read your procedure. Check what your KEV lookups returned for GHSA-identified alerts. If your version holds up, you built something good.
If you want to check against ours, the full triage reports for Redash, JupyterLab, and NetBox are published, with the risk-context.yaml for each, in the Reachable Risk Samples. Run your workflow over the same projects and compare verdicts. Or connect your agent to the k9 MCP server and score your own backlog: the Reachable Risk trial is 7 days, on us, at k9security.io/lp/reachable-risk.
Recent Comments