Guardix is an hours-long audit against a pinned commit. Agents map how the protocol works, hunt for bugs, try to prove or kill each claim, and write a report you can share. That run is the product.
We measure whether the pipeline is getting better on a public benchmark. The completed EVMBench board is 88.9% attributed recall, 104 of 117. This post is how we got a score like that without turning the benchmark into the product.
A swarm on one snapshot
A run is not one prompt over the repo. It is a swarm. Different agents own different slices of the same commit. They do not all do the same job.
- Understanding: how the system is supposed to work, what must stay true, who can do what.
- Detection: specialized passes over the code, looking for different kinds of failure.
- Investigation: follow a suspected bug until it is proven, disproven, or left open.
- Report: a verdict on each finding, tied to the commit.
What one agent is
Each agent is a model in a harness, with tools that beat guessing. The harness is the loop around the model: how it is prompted, what it can write, when it stops. Some agents use a short harness. Some need a long one.
One example is the protocol integration reviewer. It looks at how this repository calls an external protocol. It can run three tools: extract call sites, fetch the verified source, and look up a protocol card we already wrote.
An eval is a frozen case plus a check: did this agent, tool, or environment change catch the class it was built for, without wrecking the rest.
We also swap models. A model that writes useful architecture notes is not automatically the one that writes a protocol card. Each mix goes through that check. Shipping a new model without that check is how a worse mix reaches production.
How the system actually improved
EVMBench is expensive confirmation. You do not raise it by rerunning the full board after every change. The inner loop is recorded studies: real repos, real findings, the artifacts our pipeline already wrote.
We loaded a large set of those cases. We processed them with the same tools the product uses. We scored detections against the labels. Then we ran miss analysis on those cases. For each miss we asked where in the pipeline the bug dropped, and what class of miss that is.
From those runs we built evals.
Then we scaled that loop. The same work, in parallel, across agents, tools, and environment setup. The change is not one prompt edit. It is many recorded runs, miss analyses, and evals.
We kept that loop on a large set of repositories with labels. The result is the completed EVMBench board: 88.9% attributed recall.
This is how we improve Guardix between public boards. It is not a score we stamp on your repository, and it is not a reason to skip human review on a high-stakes launch.