Why security scanners report their own rules as vulnerabilities

We pointed our own scanner at our own source code. It returned 128 findings. Almost none of them were vulnerabilities, and the reason is a problem every security tool has and very few talk about.

A scanner works by knowing what a vulnerability looks like. That knowledge has to live somewhere, and where it lives is a list of strings. Regular expressions for AWS keys. Keyword tuples for command injection. A table mapping rule ids to CWE numbers. The file that holds those strings is, from the scanner's point of view, the most dangerous file in the repository.

Point the scanner at it and it reads its own vocabulary back to you as findings.

What it actually found

The first self-scan produced 128 findings across our backend and frontend. Working through them, they fell into a few groups, and each group turned out to be a different mistake.

Its own pattern tables

Our rule registry is a list of dictionaries. Each one names a vulnerability class and the keywords that indicate it. Here is the entry for transport security:

{
    "id": "AXM-TLS-001",
    "category": "Transport Security",
    "keywords": ("tls", "certificate", "rejectunauthorized", "verify=false"),
    "cwe": "CWE-295",
}

The scanner reported that as a High severity TLS Verification Disabled. It saw the string verify=false and did what it was built to do. The tuple naming subprocess became command execution. The secret pattern table became a secrets finding.

Its own documentation

Detection engines document themselves by showing the bad code they look for. Ours did too:

def detect_dynamic_sql(code):
    """Find SQL assembled from user input.

    Safe:   db.execute("SELECT * FROM users WHERE id = %s", (user_id,))
    Unsafe: db.execute(f"SELECT * FROM users WHERE id = {user_id}")
    """

Prose cannot execute. A docstring showing an injection is not one. But a line-based rule does not know it is inside a docstring, so it matched and reported.

Its own test suite

This one was the largest by volume, and it is the one that affects customers rather than just us. On a real repository we scanned, 33 of 66 code findings came from a single Playwright spec:

const response = await page.request.get(`${mockBaseUrl}/__e2e/state`);

A test calling its own mock server is not server-side request forgery. Half that report described code that never reaches production, and it was burying the half that did.

Why the rules were wrong, not just unlucky

It would be comfortable to call these edge cases. They were not. Each one was a rule written more loosely than the thing it was trying to describe.

The clearest example was SQL injection. Our rule looked for a SQL keyword and any sign of dynamic construction on the same line. The keyword list included update. So this line, in the code that builds package-manager upgrade commands, was reported as High severity SQL injection:

return `cargo update -p ${packageName} --precise ${fixedVersion}`

There is no database on that line. There is no query. There is a shell subcommand that happens to share a name with a SQL verb, and an interpolation. update, select, delete and from are ordinary English words and ordinary shell words. Matching one of them is not evidence of anything.

The same fault appeared three more times once we knew to look for it. The cross-site-scripting rule matched its own evidence text — the sentence "raw HTML rendering through innerHTML, dangerouslySetInnerHTML, or document.write" contains all three sink names. The file-inclusion classifier carried the bare keyword include, so a complaint about a regular expression that "includes nested quantifiers" was filed as CWE-98. And a check for the f-string prefix had no word boundary before it, so the trailing f of 'nosniff' was read as an f-string containing SQL.

What we changed

Three ideas, and each one is smaller than the problem it solves.

A definition is not an instance

A compiled regular expression, or a member of a constant collection whose name says it holds patterns, is a rule. The whole literal is tracked, not just the line that opens the bracket, because the strings that trip a rule sit on the member lines. Docstrings and block comments are documentation, and prose cannot execute.

Secrets are the deliberate exception. A credential pasted into a comment is leaked exactly as thoroughly as one in an assignment, so the secret scanner skips pattern tables and reads comments like any other line.

Test code is not shipped code

Findings in a repository's own test suite go to an appendix rather than the headline. Not deleted — they are really in the repository and a reader is entitled to see them — but they no longer set the severity counts on the cover.

Credentials are the exception again, and for the same reason. A key committed to a test file is in your git history. It stays with the findings.

Rules ask for the shape, not the word

Every keyword now matches on word boundaries, so lfi cannot answer for any word containing those three letters. SQL wants a statement. The XSS rule reads the line with its string literals blanked out, because a sink is written as code and a rule that describes one writes its name inside a sentence.

Where that left the numbers

Four self-scans, each after one round of fixes:

128 findings   →   24   →   6   →   2   →   1
 60 pages      →   31   →  12   →   7   →   5

The one that remains is real: an outdated nanoid with a published advisory. We upgraded it.

What to check in the tool you use

You do not need our scanner to test this. Point whatever you use at a repository that contains security tooling, a WAF rule file, or a substantial test suite, and read what comes back:

• Does it report your test suite's HTTP calls as server-side request forgery?
• Does it report a file of detection patterns as containing the things it detects?
• Does a docstring showing unsafe code get reported as unsafe code?
• Do the severity counts on the summary include all of the above?

The last one matters most. A scanner that raises noise is annoying. A scanner that lets noise into the number at the top of the report is telling you something false about your codebase, and that number is the one people act on.

The argument underneath

A scanner is only worth what its report is worth. If the report highlights the wrong thing, a better detection engine behind it changes nothing — the finding that mattered is still on page 40, underneath thirty-three findings about a mock server.

Every fix described here made our scanner find less. That was the point.

Run it on a repository you know, and see whether what comes back is about your code. If a scanner cannot be trusted on a codebase you can check by hand, there is no reason to trust it on one you cannot.