Contacto
Gabs
Blog

Can an AI security scan replace a senior engineer's review?

No. The best AI scan found 30.7% of known vulnerabilities in Vercel's DeepsecBench. Scan every change and keep a senior review.

Gabriel EspinheiraSenior Full-Stack Engineer
  • Published
  • 9 minReading time
  • EngineeringCategory

On 16 November, a non-developer posted on r/TechLeader about a booking app they had built with Cursor and Claude. A friend who does backend work sat with them for an hour, opened devtools and found the OpenAI key, the live Stripe secret key and the key for Resend, their email service, in the JavaScript every visitor downloads. One of the first replies suggested an AI security scan: "It's as easy and deploying the site then asking an AI to pen test the website." (opens in a new tab) For those keys, the reply is right. A scanner flags a secret in a bundle easily. The harder question is what the scan misses once the obvious mistakes are gone, and whether anything replaces the senior engineer who catches those.

TL;DR: No. In Vercel's DeepsecBench launch tests, the best AI security scan found 30.7% of the known vulnerabilities, and AI-written code still failed about 44% of Veracode's security tasks. Run a scan on every change as the first pass, and keep a senior engineer for the threat model, the triage and a test for every real finding.

Why is security review harder now that agents write the code?

AI-written code compiles almost every time, and on security-sensitive tasks it comes out safe only about half the time. Veracode's 2026 GenAI Code Security Report (opens in a new tab), published in July, tested models on coding tasks built around known weakness types, the CWE list. Syntax was correct in nearly 100% of them, about 44% introduced a known vulnerability, and the average security pass rate was 56%. Cross-site scripting was the worst case: models wrote safe code for it 15% of the time, against 83% for SQL injection.

That code lands in a bigger pile of bugs. VulnCheck, which tracks vulnerabilities and their exploitation, counted 100,222 CVEs published in 2026 by 13 November, more than double 2025's total, The Register reported (opens in a new tab). And review is already the slow step. A Harvard working paper (opens in a new tab) by Fiona Chen and James Stratton, covering 718 firms, found that engineers using coding agents added about 30% more code, finished work didn't measurably rise, and getting code through review took 49% longer.

More code, more bugs and a review queue that is already behind: that is the case for handing review to a machine. A working app makes the case feel stronger than it is, because a feature that works looks finished.

What does an AI security scan actually catch?

The two tools I'd start with, both with open-source CLIs, point a frontier model at your repository and ask it to follow data through the code the way a reviewer would, by different routes. Codex Security (opens in a new tab), OpenAI's CLI, builds a threat model of the project, scans, validates candidate findings and drafts patches: cs scan . covers the repository and cs scan . --diff origin/main covers a branch. It's a research preview (opens in a new tab) for ChatGPT Pro, Business, Enterprise and Edu. deepsec (opens in a new tab), from Vercel, runs a free pattern scan to pick candidate code, hands the candidates to Claude Code or Codex for a review you pay for in model tokens, and has a revalidate step that re-checks findings to cut false positives.

These tools find real bugs. Google's Mandiant team reports (opens in a new tab) that its own agent pipeline found over 100 confirmed critical vulnerabilities in two days during one incident response. They also miss a lot. Vercel built DeepsecBench (opens in a new tab) on an open-source codebase at the commit just before a batch of security fixes, with 231 human-judged findings as the answer key. The best run found 30.7% of them, and 20 of 25 runs found under 20%.

Newer models do better. On Artificial Analysis's version of the benchmark (opens in a new tab), which runs its own harness, GPT-6.2 Astra now leads with an F2 score of 56%, a score that weights finding bugs above avoiding false alarms. That is a real jump. It is also a score against a fixed answer key of bugs someone already knew about, and the bugs in your app aren't on anyone's key.

The tools also create work. Vercel puts deepsec's false positive rate at roughly 10 to 20% in its own use. Flavio Copes ran deepsec on his site's repository (opens in a new tab): 729 files investigated and 504 potential findings, before he revalidated and triaged them. OpenAI's president Greg Brockman said on 12 November that companies will be secure "only to the extent of the frontier model that they use to defend themselves" (opens in a new tab). I read the benchmarks the other way round: the model sets a floor, and a person still decides which of those 504 findings matter.

What does a senior engineer check that a scanner can't?

A good reviewer starts from what the app must never allow and then reads the code. A scanner starts from the code and has to guess what was intended, and the bugs that cost the most tend to hide in that guess.

Take the Lovable research Crissy Field published (opens in a new tab) on 14 November. Across 34,545 Lovable apps backed by Supabase, the researchers made read-only requests without logging in and found records containing email addresses in 10,560 tables across 7,403 sites. The front-end code in those apps can be perfectly clean. The bug is a missing row-level security policy, the database rule that says who may read which rows, and only someone who knows which tables are meant to be private can call it a bug. Some tools flag a table with row-level security switched off, but a table with a policy that is too loose looks like a working query to anyone who hasn't been told who should see it.

Mandiant's pipeline is built around that knowledge. Its agents get asset inventories, architecture documents and threat intelligence as input, each confirmed finding goes to a human expert who replicates the exploit before it counts, and the team writes that "we still need to address false negatives". The senior engineer's version for a single app fits on one page:

  • Who can see what. For each route and table, which user may read or change it, with a test that tries as the wrong user.
  • Where the secrets live. Keys stay on the server, a variable with a public prefix like VITE_ is treated as public, and every paid API key has a hard spending cap.
  • What runs that you didn't write. Dependencies and third-party scripts. On 16 November, Report URI (opens in a new tab), which monitors the scripts sites load, found malicious JavaScript served through Optimizely's own CDN to visitors of 178 organisations, without any of those sites changing a line of their own code.
  • What changed since the last review. A full scan once a quarter misses the endpoint an agent added on Tuesday.

How do you review AI-written code for security on every change?

Write down what the app must never allow, scan every diff against it, and turn each real finding into a test. The threat model and the triage are the senior engineer's job; the diff scan and the test are where an agent saves you hours. For the Lovable case, the test that would have caught it is a few lines:

test("a signed-out visitor can't read bookings", async () => {
  // anonClient: a Supabase client created with the public anon key
  const { data } = await anonClient.from("bookings").select("*");
  expect(data).toEqual([]);
});

That test only exists if someone decided bookings are private. Here is the loop that puts it, and the scanner, in place:

  1. Write the threat model once. A page in the repository listing what is worth protecting (user data, payments, admin actions) and the rules that protect it: a user only reads their own bookings, keys never reach the browser. Codex Security can draft a SECURITY.md and save a threat model. The senior engineer edits it, because the scanner can't know your business.
  2. Scan the diff in CI. Run cs scan . --diff origin/main or deepsec process --diff on each pull request, and a deep scan of the whole repository on a schedule. Attackers move fast once code is public: CrowdStrike saw (opens in a new tab) 88% of exploitation that used public proof-of-concept code happen within 48 hours of the code's release, in the first half of 2026.
  3. Triage every finding. Each one gets a verdict and a reason: real, false or accepted risk. A list of 504 findings protects nobody until someone has read it.
  4. Write the failing test before the fix. A confirmed finding becomes a test that tries the attack and fails, and the patch makes it pass. Tests are the spec agents need, and security is where that pays most: the next agent that touches the code has to keep the hole closed, or the build goes red.

When is an AI security scan enough on its own?

A prototype that nobody signs in to and nobody pays through can live with an AI security scan alone, as long as it holds no paid API keys either. The r/TechLeader app shows why that last condition matters: the OpenAI key had no spending limit, so a prototype with a leaked key is still a bill waiting to happen. Once real people log in or pay, the question moves from "is this code wrong?" to "what happens if another account can read this?", and that one needs someone who knows what the app is for.

If you build alone and security isn't your field, borrow someone for an hour before launch. The r/TechLeader poster did: one hour, a browser and devtools, three live keys, and 212 requests for /.env already in the server logs from the two weeks the app had been online. Those requests come from bots that probe every site, so nobody had to know the app existed for it to be tested. After that hour, put a scanner on every change, so the review doesn't stop on launch day.

What to do this week

Pick one app you ship and run cs scan . --diff origin/main, or npx deepsec init --max-cost-usd 25 to cap the first run at 25 dollars. Before you open the findings, write down what that app must never allow. Read the findings against that list, and write a test for each real one.

I write about coding agents and how to ship with them on the gabs.app blog. I'm a senior full-stack engineer and my work is on gabs.app; if you're hiring, the contact form comes straight to me.