A language model reading your code will quote it back accurately and tell you the wrong line number. Both of those are reliable. If your scanner uses AI anywhere and prints line numbers, this is worth ten minutes of checking.
We found it in our own report. A Deep Review told us there was a problem in static_analyzer.py at lines 218 to 229, and printed the code:
SQL_STATEMENT_RE = re.compile(
r"(?is)\b("
r"select\b.*?\bfrom\b"
r"|insert\s+into\b"
r"|update\b.*?\bset\b"
r")"
)The code is exactly right. That regular expression is in that file, character for character. It is at line 410.
Why the quote is right and the number is not
A model reproducing a passage it has read is doing the thing it is built for. Counting how many newlines preceded that passage is arithmetic over a long context, and it is not built for that at all. So you get an accurate quotation with an invented address, and the invention is usually plausible — 218 rather than 410, not 9,000.
Plausible is what makes it dangerous. A wildly wrong number gets noticed. A number in the right neighbourhood sends someone to the wrong function and lets them conclude the finding was nonsense.
The second failure, which we did not expect
The wrong line number also broke something downstream, and this is the part worth generalising.
We had just built a filter to stop the scanner reporting its own rule definitions as vulnerabilities — the subject of the first post here. That filter works by line number: it computes which lines in a file are pattern definitions, then drops findings that land on them.
The finding above should have been dropped. SQL_STATEMENT_RE is a compiled regular expression, which is the clearest case the filter handles. But the filter checked line 218, found ordinary code there, and let the finding through.
And it was misclassified too
The same finding arrived labelled File Inclusion, CWE-98. It was about regular expression denial of service. The classifier matched on the word include, which appeared in the description as "the pattern includes nested quantifiers".
So: right code, wrong line, wrong category, and it survived a filter designed to catch exactly it. One finding, four problems.
The fix: let the snippet decide
The quoted code is the reliable part, so it should be the authority. Every AI finding now gets anchored before anything reads its line number:
- Take the first non-empty line of the quoted snippet.
- Find it in the real file. If it appears more than once, take the occurrence nearest the model's guess — the guess is wrong but rarely wild, so it is still information.
- Set the range from there, using the snippet's own length.
- If the snippet is not in the file at all, clear the line numbers.
That last step is the one that takes a decision. A finding with no line number is inconvenient. A finding with a confident wrong one is worse than inconvenient, because a reader has no way to tell the difference between "this is at line 218" and "we believe this is at line 218" when both are printed the same way.
normalized = anchor_findings_to_source(normalized, source_code)
normalized = drop_rule_definition_findings(normalized, source_code)Order matters here. The definition filter is only as good as the numbers it is given, so the anchoring has to happen first. Once it did, the finding above was dropped the way it should have been in the first place.
How to check the tool you use
You do not need access to a scanner's internals for this. Take any AI-assisted finding from a recent report and do one thing: open the file at the line it names.
- Is the code there the code in the report? If not, the number is invented.
- Try a large file — the error grows with distance into the context.
- Check whether any finding in the report has no line number. If none ever does, the tool is guessing rather than declining to answer.
The third check is the most revealing. Every scanner sometimes cannot locate a finding precisely. A report where that never happens is not a report with better analysis behind it — it is one that has decided to always produce a number.
The general shape
Language models are good at recognition and bad at bookkeeping. Any pipeline that takes their output and treats a numeric field with the same confidence as a quoted one will end up printing arithmetic as fact.
The rule we ended up with: believe what the model quoted, verify where it says it came from, and print nothing when you cannot.
Run a scan on something you know well and check a line number by hand. It is a five-minute test and it tells you a lot about what a report is worth.