A real sample

A real run of the tool on 2026-11-02, on a synthetic 24-file project (the 12 fake and 12 honest modules of the FALSEWORK blind set C). Nothing in it was edited by hand. It has no reviewer notes, and says so.

The full report

# AI Fabrication Audit — Sample Co (synthetic test project)

**Issued:** 2026-11-02T06:46:06Z  
**Tier:** Standard  
**Audit ID:** `fa-20261102-a0b1a6c9`  
**Prepared by:** FIRME AI CORPORATION  
**Signing key fingerprint:** `SHA256:2232b4add9b35d4016e5be270df4ed56e0dc9b39a0530ac69891073253aae90d`

---

## 1. Verdict

### `STUB`

At least one module contains code that presents a result it does not actually compute. This is the highest-risk outcome and should be treated as a production defect.

| Severity | Count |
|---|---|
| CRITICAL | 1 |
| HIGH | 0 |
| MEDIUM | 2 |
| LOW | 1 |
| **Total** | **4** |

Files scanned: **24** · Analysed: **24** · Not analysed: **0** · Skipped as tests: **0**

---

## 2. What was audited

The audit is bound to the exact bytes below. Any later change to the codebase produces a
different digest, so this report cannot be re-pointed at different code.

**Manifest digest (SHA-256):** `47aa7b920e68a9d00b7c70e583636828b5596e4ea2344de653c2aa1c63d06c00`

---

## 3. Method

Static analysis only, by FALSEWORK (CAIN-42's fabrication detector). Deterministic rules over each
file's syntax tree and comments: no model was asked for an opinion, and the same code always gets the
same findings. The checks fall into these groups:

- **Made-up numbers** — a random draw, or a typed-in constant, returned as a measurement, score,
  pass count or result.
- **Checks that cannot fail** — a verify / validate / is_allowed function that returns success without
  comparing anything; a pass flag that is always True; signatures counted without being verified.
- **Failure turned into success** — swallowed exceptions, an error path that grants access, a failed
  attempt counted as done, a health check that never calls the dependency.
- **Stand-ins in shipped code** — mocks used as the real database, simulated charges, encryption that
  only encodes, placeholder routes, TODO / STUB markers.
- **Security shortcuts** — secrets compared with `==` (timing leak), keys written to world-readable
  files, tenant ids accepted by a route and then ignored.
- **Route → engine mismatch** — a route calling a method that does not exist on the engine class it
  is bound to. This class of defect returns HTTP 200 and does nothing.

Severities: a new check starts at LOW (listed, not counted toward the grade) and is raised only when
measurements show it does not raise false alarms.

---

## 4. Scope and limitations

> Pattern-based static checks on the files supplied. REAL means none of these checks fired on the files analysed; it is not proof that every result is computed. UNKNOWN means nothing could be analysed.

Read that sentence literally. This audit finds code that is structurally fabricated or
silently non-functional. It does **not** prove that correct-looking code is correct, it does
not review business logic, and it does not replace a functional test suite. A `REAL` grade
means these specific checks did not fire — nothing more.

**Measured accuracy (FIRME AI CORPORATION's own measurements, 2026-11-02; not independent):** on
a 103-case benchmark built from real defects, 37 of 42 defects were caught at a grade-moving
severity with no false alarms on 61 clean cases; on 560 files of the Python standard library, 44
were graded PARTIAL (reviewable notes such as TODOs, not fabrication); on a fresh blind set of
realistic modules written after the last engine change, **3 of 12 fakes were caught**, with no
false alarms on 12 honest look-alikes. The engine is precise and narrow: what it flags is worth
reading; what it does not flag has not been shown to be honest. Data:
https://cainstudio.online/top25-falsework-evidence.html

---

## 5. Findings by blast radius

Ranked by severity × how close the file sits to a real action (routes, money, auth and
dispatch paths rank above utilities).

| # | File | Score | Critical | High | Medium | Low |
|---|---|---|---|---|---|---|
| 1 | `accuracy_random_fake.py` | 100 | 1 | 0 | 0 | 0 |
| 2 | `audit_log_dropped_fake.py` | 10 | 0 | 0 | 1 | 0 |
| 3 | `virus_scan_skip_fake.py` | 10 | 0 | 0 | 1 | 0 |
| 4 | `tenant_delete_unscoped_fake.py` | 2 | 0 | 0 | 0 | 1 |

---

## 6. Detail

### 1. `accuracy_random_fake.py`

- **[CRITICAL] rng_fabrication** — line 5
  - Random number returned as the result of `evaluate_model` -- the value it presents as a computation or measurement is built from a random draw and constants only.
  - ```
    return {"examples": n, "accuracy": random.uniform(0.91, 0.97)}
    ```

### 2. `audit_log_dropped_fake.py`

- **[MEDIUM] bare_except_pass** — line 4
  - Bare except that swallows all errors with no logging -- real failures become invisible.
  - ```
    except Exception:
    ```

### 3. `virus_scan_skip_fake.py`

- **[MEDIUM] placeholder_marker** — line 2
  - Explicit placeholder/incomplete-work marker left in shipped code.
  - ```
    # TODO: call the antivirus engine
    ```

### 4. `tenant_delete_unscoped_fake.py`

- **[LOW] tenant_param_ignored** — line 5
  - Route takes `org_id` but never reads it, and reads stored records (line 6): the caller's tenant is checked at the door and then ignored, so the records returned can be another tenant's.
  - ```
    def delete_user(org_id: str, user_id: str, db=None):
    ```

### Files with no findings (20)

None of the checks fired on these files. Per §4, that is not a statement that they are correct.

<details><summary>Show the list</summary>

- `accuracy_computed_honest.py`
- `audit_log_raises_honest.py`
- `backup_checksums_honest.py`
- `backup_verified_constant_fake.py`
- `cost_report_computed_honest.py`
- `cost_report_typed_fake.py`
- `encrypt_fernet_honest.py`
- `encrypt_reverse_fake.py`
- `mfa_any_code_fake.py`
- `mfa_pyotp_honest.py`
- `pii_redactor_passthrough_fake.py`
- `pii_redactor_real_honest.py`
- `rate_limit_noop_fake.py`
- `rate_limit_real_honest.py`
- `signature_len_check_fake.py`
- `signature_verified_honest.py`
- `tenant_delete_scoped_honest.py`
- `uptime_hardcoded_fake.py`
- `uptime_measured_honest.py`
- `virus_scan_real_honest.py`

</details>

---

## 7. Reviewer notes

**No reviewer notes were added to this report.** Everything above is the engine's output only;
no person has reviewed the flagged files or looked for fabrication the checks cannot see.

---

## 8. Remediation priorities

1. **Critical items first.** Any `CRITICAL` finding in a route, billing, auth or dispatch
   path is a customer-visible defect, not a code-quality note.
2. **Then partial implementations.** Every swallowed failure in §6 is a place where an
   outage will look like success.
3. **Then markers.** TODOs and `mock_*` variables in shipped paths are where the next
   fabricated feature gets built.

Fix + Prove is available per item: the defect is fixed, an independent test is run against
both the old and the new code, and the result is signed into the same evidence chain.

---

## 9. Evidence and verification

The attached bundle (`evidence.zip`) contains the signed record, this report, the full
findings, the issuer's public key, and a standalone verifier that imports nothing from our
codebase.

**Do not trust the key inside the zip.** Anyone who hands you a zip can put their own key in
it and re-sign everything. Pin the key from a source you trust:

```
python3 verify_keystone.py --pinned-pub <PUBLIC_KEY_B64>
```

Compare the result against fingerprint `SHA256:2232b4add9b35d4016e5be270df4ed56e0dc9b39a0530ac69891073253aae90d` obtained independently.

Exit code 0 means: the record content matches its hash, the signature verifies against the
pinned key, the chain link is intact, and the report digest matches the report in the zip.

---

_Generated 2026-11-02T06:46:06Z by FIRME AI CORPORATION._

FIRME AI CORPORATION · CAIN-42 is a Byzantine trust fabric. Numbers marked "our own check" were run by us, not by an independent party. Nothing on these pages is a certification.