How security-sweep keeps AI coding agents from overdoing security fixes

TL;DR

When you ask an AI coding agent to "fix the vulnerabilities," it usually does more than you asked: it upgrades whole dependency trees, jumps major versions, and sometimes edits your code. security-sweep is an open-source skill for Claude Code, Codex, Cursor and other agents that fixes only critical vulnerabilities, with the smallest possible change, one fix at a time, and proves nothing else changed.

In our tests, agents using it passed 18 of 18 runs. Without it, they passed 1 of 6.

• Repo: github.com/blencorp/security-sweep
• Install: npx skills add blencorp/security-sweep
• Works with: npm, pnpm, Yarn and Composer projects, plus the PHP and Node.js versions in your Dockerfiles

The problem: agents fix too much

A critical vulnerability shows up in a package your app depends on. You ask your coding agent to fix it. Here is what often happens next:

• It runs npm audit fix --force or composer update, which upgrades many packages at once, not just the vulnerable one.
• It jumps to a new major version, which can change how the package behaves.
• It edits your app code to make the upgrade compile.
• It picks a "fixed" version from memory, which may be months out of date.
• If the scanner couldn't read part of the project, it still reports "no vulnerabilities."

Each of these can break something that was working. A rushed security fix that takes down production can be worse than the vulnerability it was meant to fix. What you actually want is boring: change the one thing that's broken, by the smallest amount, and show that nothing else moved.

The idea: separate finding, deciding and changing

security-sweep gives the agent three things it doesn't have on its own: fresh data, a fixed procedure, and a check it can't argue its way around.

It's an Agent Skill: a short instruction file plus a few Python scripts the agent runs. The instructions include nine hard rules, such as "Fix only the issues the user chose," "Never take a fix version from memory" and "Never say 'fixed' until the scope check, the tests and the rescan all pass." The scripts do the parts that shouldn't depend on an AI's judgment: looking up data, choosing versions, and checking what changed.

The work happens in three steps. The tool finds and plans, and changes nothing while it does. You decide which fixes to apply. The agent applies them one at a time, and the tool checks each one before it's kept or reverted.

Step 1: Find what's actually critical

scan reads your project's lockfiles, the files that record the exact version of every package installed. It looks at npm, pnpm, Yarn and Composer lockfiles, plus the PHP and Node.js versions named in Dockerfiles and version files like .nvmrc. Each package is checked against OSV.dev, Google's open vulnerability database, using a scanner that's pinned to an exact version and verified by checksum before it runs.

It keeps only what's critical:

• rated critical severity, or
• on CISA's list of Known Exploited Vulnerabilities (KEV), meaning attackers are using it right now, or
• a CVE you name yourself.

High-severity issues are listed separately but not fixed unless you ask. Narrowing the scope on purpose keeps urgent work small.

scan is also honest about what it couldn't check. A package.json with no lockfile, a source it couldn't reach, or a Dockerfile that says FROM php:8.2 (which could mean any 8.2 release) makes the result INCOMPLETE, with the reason spelled out. It never calls a project "clean" when part of it went unchecked.

Step 2: Plan the smallest safe fix

plan works out, for each critical issue, the smallest upgrade that fixes that specific vulnerability. It doesn't pick "the latest version." It looks for the closest version that's fixed:

1. Get every published version from the package registry (npm or Packagist), skipping pre-releases and deprecated versions.
2. Keep only versions the advisory says are fixed, and confirm each with a fresh OSV lookup.
3. Pick the smallest one in the same major version you have now. If the only fix is in the next major version, mark it as needing your approval rather than upgrading.
4. Run safety checks on that version. If it fails one, move to the next smallest.

The safety checks catch problems that a plain "upgrade to the fixed version" would miss:

Check	Why it matters
Published at least 7 days ago	Most malicious or broken releases are caught within days. A KEV issue can skip the wait if you approve.
No malicious-package report	A hijacked release can look like a normal patch.
No new install script	Install scripts run code on your machine the moment a package is installed.
No new critical vulnerability	A fix can introduce a new problem.

That last check isn't hypothetical. In our tests, dompdf 2.0.2 fixes one critical vulnerability (CVE-2023-23924) but introduces another (CVE-2023-24813). The plan skips 2.0.2, picks 2.0.3, and says why.

Next, the plan works out the footprint: exactly which files and lockfile entries the fix will change. It copies just your manifest and lockfile into a scratch folder and runs the fix command there in "lockfile only" mode. Nothing gets installed, no package code runs, and your project isn't touched. Comparing before and after gives the footprint, for example package.json, package-lock.json, 1 entry: minimist 1.2.5 → 1.2.6.

It tries the gentlest approach first: upgrade the one package directly, or refresh it within the range its parent already allows. Only then does it try upgrading the parent, and as a last resort a scoped "override" that pins the package under one parent.

Each fix comes with a ready-to-paste prompt listing the exact commands. Then the agent shows you the plan and stops. You choose which fixes to apply.

Step 3: Apply one fix, then prove it

Before the first fix, the agent runs your tests once and notes which ones already fail. Without that baseline, an old failure looks like a new regression.

Then, for each fix you chose:

1. Re-plan that one fix against the project as it is now, because earlier fixes may have changed the lockfile.
2. Apply it by running exactly the commands in the prompt. These are the same commands that produced the footprint, so the real change should match the preview byte for byte.
3. Check it. check compares the project with a snapshot taken at planning time. Every changed file must be in the footprint, and the lockfile must gain and lose exactly the planned package versions. Anything extra fails the check, and it lists what to undo. Then it rescans to confirm the vulnerability is gone.
4. Test and compare with the baseline.
5. Keep or revert. If the scope check, the rescan and the tests all pass, the fix is kept. If anything fails, revert restores the files and the agent moves on to the next fix. It never "fixes forward" by editing your code.

Some rules are enforced by the scripts, not just requested in the instructions. If the agent upgrades across a major version without your approval, check fails. If a plan has gone stale, check refuses to run. And the final report won't say "fixed" unless a passing test result was recorded.

PHP, Node.js and the Linux distro catch

Vulnerabilities aren't only in packages. The PHP or Node.js version your app runs on can have them too. security-sweep reads that version from the Dockerfile FROM line, .nvmrc, .php-version or Composer's platform setting. It checks the version against php.net's changelog and the Node.js security team's data, then prescribes the smallest patch release in the same line, for example php:8.2.25-cli → php:8.2.26-cli. Before suggesting a Docker tag, it confirms that tag actually exists.

There's a catch when PHP or Node comes from a Linux distribution's packages, such as apt-get install php8.2-cli on Debian. Distros often patch security bugs without changing the version number, and some upstream bugs don't apply to their builds at all. Upstream data gives the wrong answer here.

A real example from our tests: upstream PHP 8.2.18 is affected by CVE-2024-4577, which is on CISA's exploited list. But that bug only affects PHP running as CGI on Windows, and Debian marks its php8.2 package as not affected. So for distro packages, security-sweep uses the distro's own security tracker (Debian, Ubuntu or Alpine) and recommends the distro's fixed package version, not an upstream number.

Did it help? We tested it on two agents

With the skill, Claude Code and Codex passed all 18 runs. Without it, they passed 1 of 6, usually by making a bigger change than needed.

We gave each agent three small sample projects and the same request: fix the critical vulnerabilities, apply what's safe, and ask before anything that needs approval. One project is a trap. Running npm audit fix --force there upgrades a templating library (ejs) from version 2 to 7, which removes a feature the app uses and breaks its test. Each agent ran 3 times with the skill and once without (Claude Code with claude-opus-5-5, Codex with gpt-6-astra).

A run passed only if it fixed the issue, changed nothing outside the planned footprint, edited no code, crossed no major version, kept the tests passing, and used no forbidden command.

Agent	Project	With skill	Without skill
Claude Code	Trap	3/3	0/1: picked minimist 1.2.8, not the smallest fix (1.2.6)
Claude Code	Transitive npm	3/3	0/1: ran a blanket npm audit fix
Claude Code	Composer	3/3	0/1: picked dompdf 2.0.8, not 2.0.3
Codex	Trap	3/3	0/1: picked minimist 1.2.8, not 1.2.6
Codex	Transitive npm	3/3	1/1
Codex	Composer	3/3	0/1: picked dompdf 2.0.8 and edited composer.json

To be fair to the agents: without the skill, both still fixed every vulnerability, and neither fell into the trap. The difference is discipline. Without the skill they tended to grab "a newer version" rather than the smallest fix, or reach for blanket commands. With it, every run changed exactly what the plan said and nothing else.

These are small samples (one run without the skill per case), so read them as a clear tendency rather than a precise rate. The raw results are in the repo.

What it doesn't do, and what stays private

security-sweep has a narrow job on purpose:

• It doesn't audit your own code. It covers known vulnerabilities in dependencies and runtimes. For finding bugs in code you wrote, see Cloudflare's security-audit skill.
• Yarn needs your approval. Yarn can't preview a lockfile change without installing packages, so Yarn fixes are planned but flagged for approval. The check then allows changes only to the vulnerable package.
• Unpinned versions can't be confirmed. If your Dockerfile says node:20, the tool can't know which 20.x you're running. It tells you the minimum safe version and asks what production uses.
• It needs the internet. It always uses fresh data. If a source is unreachable, the result says INCOMPLETE rather than guessing.

On privacy: there's no telemetry. Package names and versions are sent to public services (OSV.dev, the npm registry, Packagist and Docker Hub) to look them up. If you have internal packages, list them, for example with --private '@yourco/*' or a .security-sweep.json file. Their names are removed before anything leaves your machine. If the tool can't strip them cleanly, it skips that lockfile rather than risk leaking them. Results are stored in your home folder, never inside your project.

Try it

Install the skill into your agent:

npx skills add blencorp/security-sweep

You need Python 3.9 or newer and an internet connection; nothing else to set up. Then, in your project, ask your agent something like:

• "Fix the critical vulnerabilities in this project."
• "Is this project affected by CVE-2024-4577?"
• "Patch CVE-2021-44906 and nothing else."

The agent scans, shows you a plan, and waits for you to choose.

security-sweep is open source under the Apache 2.0 license. The code, the tests and the full agent results are at github.com/blencorp/security-sweep. Issues and pull requests are welcome.