I found 10 bugs in my own security scanner. Here's what they taught me about false positives.

I found 10 bugs in my own security scanner. Here's what they taught me about false positives.

I built a VS Code extension that scans code for leaked secrets, PII, and security vulnerabilities before you commit. A few weeks in, I sat down and did something most tool builders put off: I went looking for everywhere my own scanner was wrong.

Not "wrong" as in crashing. Wrong as in flagging things that weren't actually problems, and — worse — missing things that were.

For a security tool, false positives aren't a minor annoyance. They're the thing that gets a tool uninstalled. The first time a scanner cries wolf on loginToken because it contains the substring "log," a developer stops trusting every other finding it makes, including the real ones. So I spent a focused pass hunting for exactly this kind of bug. Found 10. Here they are, with the actual root cause for each — because "we fixed some bugs" isn't useful to anyone, but the specific ways pattern-matching security tools go wrong might be.

1. Substring matching instead of call-site matching

A rule meant to catch sensitive data in log statements was matching any variable name containing "log" — including loginToken, which has nothing to do with logging. The regex was checking for the substring, not for an actual logging function call. Fixed by requiring the pattern to match a real log-call shape (logger.info(...), console.log(...), etc.), not just nearby text.

2. A dependency-pinning rule that fired on every file

An "unpinned dependency" check was broad enough that it lit up on files that had nothing to do with dependencies at all. It needed to be scoped specifically to requirements/manifest-style files, not applied repo-wide.

3. Outdated key format for GCP credentials

The private key detector only recognized the old BEGIN RSA PRIVATE KEY PEM header. Real-world GCP service account keys use PKCS#8 format (BEGIN PRIVATE KEY), which the pattern didn't cover. This wasn't a false positive — it was a false negative, arguably worse, since it means real leaked keys were silently passing through.

4. Credit card numbers with no checksum

Any 16-digit number was getting flagged as a potential credit card. Any 16-digit number. Order IDs, invoice numbers, random test fixtures — all of it. Added Luhn checksum validation, the same algorithm real payment systems use to validate card numbers, so only mathematically plausible card numbers get flagged.

5. Passport numbers matching SKUs and license plates

An unanchored regex for passport number formats was broad enough to match product SKUs and vehicle plates. Now it requires a nearby label ("passport," "passport no.," etc.) before it fires.

6. IPv6 shorthand wasn't recognized

The IPv6 detector only matched the full 8-group form. Real-world IPv6 addresses are almost always written in compressed shorthand using ::, which the original pattern missed entirely. Another false negative.

7. Flagging every import random

A rule meant to catch insecure random number generation (using random instead of a cryptographically secure source for things like tokens or session IDs) was flagging the import itself, regardless of what it was actually used for. Scoped it down to the specific insecure call sites, not the import statement.

8. Path traversal rule flagged normal file reads

Any call to open(data) was getting flagged as a path traversal risk, even when data had nothing to do with user input. Scoped to cases where the path actually derives from request data.

9. SSH public keys flagged as critical secrets

This one's almost funny in hindsight: SSH public keys were being flagged as critical leaked secrets. They're called public keys because they're meant to be shared. Pulled them out of the secrets detector entirely.

10. The same finding reported twice

A JS eval() call was tripping both the Python-oriented rule and the JS-oriented rule for unsafe eval usage, so it showed up as two separate findings for one line of code. Language-scoped the rules so each finding only fires once.

The bigger lesson

Almost every bug on this list comes down to the same root cause: a pattern that matched text instead of matching meaning. "Contains the word log," "is 16 digits," "is inside a BEGIN...END block" — these are all proxies for the thing you actually care about, and proxies break in both directions. They flag things that aren't real problems (noise), and they miss things that are (the actually dangerous outcome for a security tool).

The fix, in almost every case, wasn't a smarter model or more data. It was going back and asking "what's the actual signal here?" — a checksum instead of a digit count, a call-site instead of a substring, a label instead of a bare pattern.

I also added a zero-dependency test suite so these specific regressions can't silently come back, and fixed a handful of consistency bugs where the editor and the commit-hook CLI were enforcing different rules on the same code — which is its own quiet source of false trust ("it didn't complain in my editor" ≠ "it won't block my commit").

None of this makes the scanner perfect. But I'd rather ship "we found 10 ways this was wrong and fixed them" than pretend it was right from day one. If you're evaluating any static analysis or secret-scanning tool, "have they published their false positive fixes" is a better trust signal than any accuracy percentage on a landing page.

Veilo is a free VS Code extension — detection runs 100% locally, nothing leaves your machine. If you want to see if it catches anything in your own repo: marketplace.visualstudio.com/items?itemName=veilo.veilo