How to build a self-improving agentic pipeline

Our smart contract audit pipeline finds 88.9% of the known bugs on the completed EVMBench board, scored by mechanism. We got there by running one loop many times. The loop runs the agents on cases with known answers, finds the stage where each miss happened, changes one thing, and ships the change only if it holds on cases it was not built from.

This post describes that loop as ten steps. Each step starts with what we did at Guardix, then gives the files and rules to build your own self-improving agentic pipeline. The steps apply to any agentic system whose output you can check against a known answer.

What self-improving means here

Nothing edits itself in production. Agents read misses and draft changes, and a gate decides which changes ship. Improving means that more of the known answers appear in the output, and the output gets no noisier. That is why the loop needs labeled cases before anything else.

Part 1: Build a pipeline you can measure

Step 1: Pick one task with a checkable answer

Write the task and the success test in one sentence each. Keep the task narrow. You can measure a pipeline that finds one kind of problem. You cannot measure a general assistant.

Task: Given a smart contract repository at a pinned commit, report the
bugs that could lose user funds.

Success test: a finding counts only if it describes the same mechanism
and code path as a bug a human auditor already confirmed.

Checkpoint. Someone who has not seen your pipeline can read the success test and score an output without asking you anything.

Step 2: Split the work into stages that write artifacts

Make each stage write a file that the next stage reads. Pin the input, and record the pipeline version and each agent's model in a manifest. If the input changes between runs, you cannot tell whether your change helped.

input/                  # pinned snapshot, never edited
01_understand/          # system_map.md, invariants.json
02_detect/              # candidates.json
03_investigate/         # verdicts.json
04_report/              # report.md
manifest.json           # input hash, pipeline version, model per agent

Checkpoint. You can run the whole pipeline twice on the same input and compare the two run folders file by file.

Step 3: Build the environment and tools for each agent

You do not build the agent loop. You build what it runs against, and that is where most misses get fixed.

• Give each agent one job, a workspace with only the inputs that job needs, and one output file with a schema.
• Add a tool when a program can compute what the model would otherwise guess.
• Make the agent validate its output against the schema before it stops. An agent that writes an empty file fails.
• Keep the model a per-agent setting, so you can swap it for one agent without touching the others.

Checkpoint. Every agent has a workspace, a tool list, and an output schema you can diff. Changing the model for one agent is a one-line change.

Part 2: Build the measurement

Step 4: Build a labeled case set

Collect real inputs where a person already confirmed the answers. Store the labels where the pipeline cannot read them.

{
  "case_id": "case-017",
  "split": "dev",
  "labels": [
    {
      "label_id": "case-017-L1",
      "mechanism": "One-time setup reverts if the pool already exists",
      "code_path": "Launcher.bootstrap"
    }
  ]
}

• Split the cases into a development set and a held-out set. Split by source, so no held-out case shares a repository with a case you tune on.
• Freeze the split before you change anything.
• Keep a few clean inputs with no known issues. A change that adds noise shows up there first.

Checkpoint. Every label has a mechanism written in plain language, and the held-out set is locked.

Step 5: Write the scorer

Score by mechanism. A finding in the same file about a different problem is not a hit. Save the judge's verdict for every label in every run.

You are scoring one expected finding against a report.

Mechanism: {label.mechanism}
Code path: {label.code_path}

Report findings:
{findings}

Answer detected only if a finding describes the same mechanism and the
same code path. A finding in the same file about a different mechanism
is not a match.

Return JSON: {"detected": bool, "finding_id": string | null, "reason": string}

Also read the report the user gets. A bug that was found and then dismissed counts as a hit in the run files and as a miss for the reader. Record both.

Checkpoint. One command runs the pipeline on a split and prints recall for it. Two runs agree closely enough that you can trust a small difference.

Part 3: Run the improvement loop

Step 6: Diagnose every miss

For each missed label, open the run folder and record the stage where the answer dropped. These four failure modes apply to most pipelines.

• Never reached. No agent looked at the relevant code.
• Reached, not suspected. An agent read the code and flagged nothing.
• Suspected, not proven. A candidate existed, and investigation dropped it or left it open.
• Found, then lost. A candidate survived investigation and was dismissed before the report.

The setup bug above was suspected, not proven. Its miss record looks like this.

{
  "label_id": "case-017-L1",
  "stage": "investigate",
  "failure_mode": "suspected_not_proven",
  "class": "defense-treated-as-working",
  "notes": "Agent called the revert front-running protection. Setup could never finish."
}

Then group misses across cases. One miss is an anecdote. A class that repeats across repositories is a capability you can build. We have agents read misses side by side and propose classes, and a person reviews them.

Checkpoint. Every miss has a stage and a failure mode, and the misses fall into a short list of named classes.

Step 7: Freeze each miss class as an eval

An eval is a frozen case plus a check. The check asks whether the change caught the class it was built for without breaking anything else. Keep the label hidden from the run, and never delete an eval.

Checkpoint. Each miss class has at least one eval that fails today, and one command reruns the whole eval set.

Step 8: Change one thing

Change one thing at a time and record it. If you change two things and the score moves, you cannot say which one moved it.

Step 9: Gate the change

Run the changed pipeline on the development set, the held-out set, and the clean inputs. Compare each result with a frozen control, the same cases on the unchanged pipeline. Then run an ablation with the change turned off.

def gate(change_type, dev, held_out, clean, ablation, control):
    if control.labels_lost(dev, held_out) or clean.new_findings > clean.noise_budget:
        return "do_not_ship"
    if held_out.recall < control.held_out_recall:
        return "do_not_ship"
    if dev.recall <= control.dev_recall:
        return "unproven"
    if ablation.recall >= dev.recall:
        return "unproven"
    if change_type == "prompt" and held_out.recall <= control.held_out_recall:
        return "unproven"
    return "ship"

Checkpoint. No change reaches production without a recorded gate result, and the gate can say no.

Step 10: Repeat, then confirm on a public board

Run your loop on your own labeled cases. Rerunning a public benchmark such as EVMBench after every change is expensive and tunes the pipeline to that benchmark. Run it at milestones to check that your gains hold outside your own cases.

What the loop produced

The completed EVMBench board is 88.9% attributed recall, 104 of 117 bugs across 40 audits. Thirty audits had every expected bug detected, and none scored zero. The 13 missed bugs feed the next round of miss analysis. An earlier public run scored 59.8% with whole-report scoring, so the two numbers do not compare directly.

The smart contract audit you can run today is this pipeline. How it works describes each stage.