Skip to content
where-static-analysis-findings-live.html

Where static analysis findings actually live

← Back to Learn · By · Published · Updated

We scanned 53 well-regarded open-source projects, which produced 32,174 findings. About 58% of them (18,568) are in tests, documentation, examples, benchmarks or bundled third-party code rather than in the code that ships. For security findings the share is higher: 86% of the 1,258 are outside production code. If you read a report from any static analysis tool as one flat list, you will spend most of your time on the least important part of it.

How we measured

The 53 projects, the commit scanned in each and the finding counts are in the open data file. For this article we classified a finding as non-production when its path has a directory named test, tests, testing, spec, docs, examples, samples, benchmark, fuzz, fixtures, third_party or vendor, or when the file is named like a test (test_*.py, *Test.java, *.spec.ts and similar). The rule is deliberately simple and blunt, which has two consequences. Generated website assets still count as production, so Retrofit’s bundled documentation scripts are included in the production figure. And test directories with unusual names, such as celery’s t/, are missed. Both push the true non-production share up, not down.

Sorting a report by path yourself

The rule above is simple enough to apply in a few lines. The analyser writes its findings to a JSON file, and each entry in violations has a path, a ruleId, a severity and a category. This sketch splits them with the same directory and filename rules we used:

import json, re, collections

NONPROD = {"test", "tests", "testing", "spec", "docs", "examples",
           "samples", "benchmark", "fuzz", "fixtures",
           "third_party", "vendor"}
TESTFILE = re.compile(r"(test_.*\.py|.*_test\.py|.*Test\.java|.*\.(spec|test)\.(ts|js|tsx|jsx))$")

def non_production(path):
    parts = path.split("/")
    return (any(p.lower() in NONPROD for p in parts[:-1])
            or bool(TESTFILE.match(parts[-1])))

report = json.load(open("out.json"))
count = collections.Counter(
    "tests/other" if non_production(v["path"]) else "production"
    for v in report["violations"])
print(count)

We ran it on a scan of Flask (25 findings, the same count as in the data file). 13 findings, 52%, were in non-production paths and 12 were in production code. Of its 9 security findings, 6 were in non-production paths. Those 6 were all py-sec-hardcoded-secret. The 3 in production were two py-sec-eval-exec findings and one py-sec-weak-hash, which is a short list that takes a few minutes to read.

One thing to know about how the engine behaves: it does not discount tests. We wrote the same line, a string assigned to a variable named PASSWORD, into tests/test_x.py and into src/cfg.py. Both came back as the same py-sec-hardcoded-secret warning. As of the October 2026 build, separating the two is a step you take when you read the report, as this sketch does.

Findings by language

  • JavaScript (5 projects): 8,847 findings, 75% outside production.
  • Python (18 projects): 6,158 findings, 62% outside production.
  • TypeScript (1 projects): 2,600 findings, 58% outside production.
  • Java (13 projects): 7,726 findings, 47% outside production.
  • C++ (14 projects): 5,503 findings, 44% outside production.
  • C# (2 projects): 1,340 findings, 41% outside production.

The share depends on how a language’s projects lay out their tests and on which rules dominate, so it varies from 41% to 75%. Even the lowest, C#, puts about four in ten findings outside production code. Three of the six groups hold five projects or fewer, so read the per-language figures as indications and not as measurements.

A few rules produce most of the findings

The three most frequent rules produce 39% of all findings:

  • js-var-usage: 8,166 findings (25%), 70% outside production.
  • cpp-magic-number-literal: 2,175 findings (7%), 28% outside production.
  • cpp-raw-new-delete: 2,052 findings (6%), 62% outside production.

The first is a style rule, flagging the var keyword in JavaScript. Three projects — lodash, Express and jQuery — produce 83% of it, and all three began before ES2015 introduced let and const. It is a legitimate finding, and also a good example of why counts mislead: it says more about a project’s age than about its current quality.

Some rules are almost entirely about test code. java-empty-catch-block fired 1,512 times, 98% of them outside production, likely because tests often catch an exception on purpose to check that it is thrown. py-mutable-class-attribute fired 1,994 times, 81% outside production.

Security findings, in particular

Only 172 of the 1,258 security findings are in production paths. The largest group is py-sec-hardcoded-secret: 450 findings, of which 5 are outside tests. Most of these are in test files, where placeholder passwords and tokens are common, and a pattern matcher cannot tell a placeholder from a real credential. The next largest are py-sec-pickle-load (296), ts-sec-inner-html (128) and py-sec-eval-exec (103).

The 172 findings in production code are the ones worth reading first. They are led by js-sec-inner-html (61, of which 56 are in Retrofit’s generated website scripts), then py-sec-eval-exec, py-sec-pickle-load and py-sec-weak-hash. A short list of real candidates is far more useful than a long one that is mostly test fixtures.

Reading a security finding in three questions

Once a finding is in production code, three questions settle most of them:

  1. Can it be reached by anyone who should not control it? eval, exec and pickle.load are only dangerous when the input can come from outside. The same call on a constant string is a code smell, not a hole.
  2. Is the value real? For py-sec-hardcoded-secret, check whether the string is a working credential or a placeholder. If it is real, revoke it and move it to configuration, because deleting the line does not remove it from the repository history.
  3. Does the context change the risk? A weak hash used as a cache key or checksum is a different matter from one used to protect a password. The usedforsecurity=False flag in Python is the documented way to say so. For innerHTML, ask whether the string includes anything a user typed.

What scanning real code taught us about false positives

Preparing this data set led to a round of fixes in the engine, and every number above comes from the fixed version. The patterns are worth knowing, because other tools run into the same things:

  • A deleted constructor, Foo(const Foo&) = delete;, and an operator new declaration were being reported as raw new/delete.
  • hashlib.md5(data, usedforsecurity=False), the documented way to say a hash is only a checksum, was reported as a weak hash.
  • Method calls such as regex.exec(...) and obj.eval(...) were taken for shell execution or the built-in eval.
  • __slots__, __all__ and ALL_CAPS constants were reported as mutable class attributes.
  • A TypeScript test file containing a call such as it("x", () => {...}) without a trailing semicolon made the parser invent a 4,000-line “function” that ran to the end of the block.

The last one is a good reminder that long-function counts depend on the parser being right. It is also why we publish the data: anyone can rerun a project and check.

What these numbers do not mean

  • Non-production does not mean unimportant. A test that swallows an exception can hide a failing test, and broken test helpers cost time. The point is order, not dismissal.
  • The path rule is blunt. It misses directories with unusual names and counts some generated code as production. We described both effects above, and both make the true non-production share higher, not lower.
  • One production finding can matter more than a thousand test ones. A single injectable query outweighs any pile of style notes. The point of filtering is to find it sooner.
  • These are REPO-SIGHT’s rules. Another tool has other rules and would produce other counts, and a rule’s share of findings says more about the rule than about the code.

A four-pass order for reading a report

  1. Security findings in production paths. It is the shortest list, in our data 172 of 32,174.
  2. Complexity and long-function findings in production paths. They point at the code that is hardest to change safely.
  3. Style findings in production paths. Cheap to fix, low in urgency, and often the bulk of the count.
  4. Everything in tests, examples and vendored code. Read it for patterns, such as a test helper that hides failures, and ignore the rest.

What to do when you read a report

  • Filter by path first. Look at production code, then separate the test findings into their own pass.
  • Read the rule’s explanation before the count. A rule that fires thousands of times is usually about style or convention.
  • Treat security findings as a short list. Start with the few in production paths and check each for real input reaching it.

Frequently asked questions

Are findings in tests worthless?

No. A test that swallows an exception can hide a failing test, and test code needs to be maintained. They are lower priority, not irrelevant.

Does REPO-SIGHT separate production from test code?

The report lists every finding with its file path, so test and production findings are easy to tell apart. See the code security scanner for what the security checks cover and the benefits of static code analysis for the wider picture.

Can I try it on my project?

Paste a GitHub URL and read the findings by file.