False alarms: 288,031 lines we did not write
A detector that cries wolf is useless for an audit. We ran FALSEWORK on the Python standard library (560 files, test folders excluded): code written by others, which we assume is not fabricated. Every MEDIUM-or-worse finding there is a false alarm.
| Grades | Before | After |
|---|---|---|
| REAL | 459 | 506 |
| PARTIAL | 91 | 44 |
| STUB | 1 | 1 |
| UNKNOWN (did not parse on this Python) | 9 | 9 |
Findings by check
| Check | Severity | Before | After |
|---|---|---|---|
swallowed_import_error | LOW | 30 | 77 |
swallowed_import_error | HIGH | 70 | 23 |
placeholder_marker | MEDIUM | 65 | 18 |
placeholder_marker | LOW | 0 | 47 |
bare_except_pass | MEDIUM | 28 | 21 |
bare_except_pass | LOW | 4 | 11 |
secret_compare_not_constant_time | LOW | 7 | 0 |
rng_fabrication | LOW | 4 | 4 |
rng_fabrication | CRITICAL | 3 | 3 |
rng_fabrication | HIGH | 2 | 2 |
check_cannot_fail | MEDIUM | 1 | 1 |
fixed_result | MEDIUM | 1 | 1 |
What we changed, and what we left
- Fixed (now LOW): an
ImportErrorwhose try has anelse:; the C-accelerator override (try: from _module import … except ImportError: pass); an optional import whose name is tested withif/and/or;FIXME/XXXnotes;except: passwith a try/else; a "token" comparison outside an authentication context (lexer and cache tokens). - Left as they are:
TODOcomments (18), optional side-effect imports (23),uuid.py's random bits graded STUB (1), a server's defaultverify_requestreturning True (1). These are things a reviewer can reasonably be asked to look at, or would need rules too specific to be general. - The seven checks promoted to MEDIUM fire zero times on the standard library.
FIRME AI CORPORATION · CAIN-42 is a Byzantine trust fabric. Numbers marked "our own check" were run by us, not by an independent party. Nothing on these pages is a certification.