# AI Fabrication Audit — Sample Co (synthetic test project)
**Issued:** 2026-11-02T06:46:06Z
**Tier:** Standard
**Audit ID:** `fa-20261102-a0b1a6c9`
**Prepared by:** FIRME AI CORPORATION
**Signing key fingerprint:** `SHA256:2232b4add9b35d4016e5be270df4ed56e0dc9b39a0530ac69891073253aae90d`
---
## 1. Verdict
### `STUB`
At least one module contains code that presents a result it does not actually compute. This is the highest-risk outcome and should be treated as a production defect.
| Severity | Count |
|---|---|
| CRITICAL | 1 |
| HIGH | 0 |
| MEDIUM | 2 |
| LOW | 1 |
| **Total** | **4** |
Files scanned: **24** · Analysed: **24** · Not analysed: **0** · Skipped as tests: **0**
---
## 2. What was audited
The audit is bound to the exact bytes below. Any later change to the codebase produces a
different digest, so this report cannot be re-pointed at different code.
**Manifest digest (SHA-256):** `47aa7b920e68a9d00b7c70e583636828b5596e4ea2344de653c2aa1c63d06c00`
---
## 3. Method
Static analysis only, by FALSEWORK (CAIN-42's fabrication detector). Deterministic rules over each
file's syntax tree and comments: no model was asked for an opinion, and the same code always gets the
same findings. The checks fall into these groups:
- **Made-up numbers** — a random draw, or a typed-in constant, returned as a measurement, score,
pass count or result.
- **Checks that cannot fail** — a verify / validate / is_allowed function that returns success without
comparing anything; a pass flag that is always True; signatures counted without being verified.
- **Failure turned into success** — swallowed exceptions, an error path that grants access, a failed
attempt counted as done, a health check that never calls the dependency.
- **Stand-ins in shipped code** — mocks used as the real database, simulated charges, encryption that
only encodes, placeholder routes, TODO / STUB markers.
- **Security shortcuts** — secrets compared with `==` (timing leak), keys written to world-readable
files, tenant ids accepted by a route and then ignored.
- **Route → engine mismatch** — a route calling a method that does not exist on the engine class it
is bound to. This class of defect returns HTTP 200 and does nothing.
Severities: a new check starts at LOW (listed, not counted toward the grade) and is raised only when
measurements show it does not raise false alarms.
---
## 4. Scope and limitations
> Pattern-based static checks on the files supplied. REAL means none of these checks fired on the files analysed; it is not proof that every result is computed. UNKNOWN means nothing could be analysed.
Read that sentence literally. This audit finds code that is structurally fabricated or
silently non-functional. It does **not** prove that correct-looking code is correct, it does
not review business logic, and it does not replace a functional test suite. A `REAL` grade
means these specific checks did not fire — nothing more.
**Measured accuracy (FIRME AI CORPORATION's own measurements, 2026-11-02; not independent):** on
a 103-case benchmark built from real defects, 37 of 42 defects were caught at a grade-moving
severity with no false alarms on 61 clean cases; on 560 files of the Python standard library, 44
were graded PARTIAL (reviewable notes such as TODOs, not fabrication); on a fresh blind set of
realistic modules written after the last engine change, **3 of 12 fakes were caught**, with no
false alarms on 12 honest look-alikes. The engine is precise and narrow: what it flags is worth
reading; what it does not flag has not been shown to be honest. Data:
https://cainstudio.online/top25-falsework-evidence.html
---
## 5. Findings by blast radius
Ranked by severity × how close the file sits to a real action (routes, money, auth and
dispatch paths rank above utilities).
| # | File | Score | Critical | High | Medium | Low |
|---|---|---|---|---|---|---|
| 1 | `accuracy_random_fake.py` | 100 | 1 | 0 | 0 | 0 |
| 2 | `audit_log_dropped_fake.py` | 10 | 0 | 0 | 1 | 0 |
| 3 | `virus_scan_skip_fake.py` | 10 | 0 | 0 | 1 | 0 |
| 4 | `tenant_delete_unscoped_fake.py` | 2 | 0 | 0 | 0 | 1 |
---
## 6. Detail
### 1. `accuracy_random_fake.py`
- **[CRITICAL] rng_fabrication** — line 5
- Random number returned as the result of `evaluate_model` -- the value it presents as a computation or measurement is built from a random draw and constants only.
- ```
return {"examples": n, "accuracy": random.uniform(0.91, 0.97)}
```
### 2. `audit_log_dropped_fake.py`
- **[MEDIUM] bare_except_pass** — line 4
- Bare except that swallows all errors with no logging -- real failures become invisible.
- ```
except Exception:
```
### 3. `virus_scan_skip_fake.py`
- **[MEDIUM] placeholder_marker** — line 2
- Explicit placeholder/incomplete-work marker left in shipped code.
- ```
# TODO: call the antivirus engine
```
### 4. `tenant_delete_unscoped_fake.py`
- **[LOW] tenant_param_ignored** — line 5
- Route takes `org_id` but never reads it, and reads stored records (line 6): the caller's tenant is checked at the door and then ignored, so the records returned can be another tenant's.
- ```
def delete_user(org_id: str, user_id: str, db=None):
```
### Files with no findings (20)
None of the checks fired on these files. Per §4, that is not a statement that they are correct.
Show the list
- `accuracy_computed_honest.py`
- `audit_log_raises_honest.py`
- `backup_checksums_honest.py`
- `backup_verified_constant_fake.py`
- `cost_report_computed_honest.py`
- `cost_report_typed_fake.py`
- `encrypt_fernet_honest.py`
- `encrypt_reverse_fake.py`
- `mfa_any_code_fake.py`
- `mfa_pyotp_honest.py`
- `pii_redactor_passthrough_fake.py`
- `pii_redactor_real_honest.py`
- `rate_limit_noop_fake.py`
- `rate_limit_real_honest.py`
- `signature_len_check_fake.py`
- `signature_verified_honest.py`
- `tenant_delete_scoped_honest.py`
- `uptime_hardcoded_fake.py`
- `uptime_measured_honest.py`
- `virus_scan_real_honest.py`