Developer documentation

Benchmarks

Last reviewed 31 August 2026

Benchmarks

Every figure on this page was measured on the live hosted deployment with

bench/benchmark.py. Nothing is modelled, extrapolated, or rounded to something

friendlier. Where a mode could not be measured, it says so and reports nothing.

Reproduce it yourself:

CAIN_API_KEY=<your key> python3 bench/benchmark.py --samples 300 --concurrency 16

Latency

Measured 2026-08-17 against https://cainstudio.online, deployment in enforce

mode. 1200 samples per mode after 5 discarded warm-up requests.

Fabric overhead: 10.28 ms median, 16.42 ms p95.

ActionProof adds 70.48 ms median on top of that, when a plan is submitted.

p99 is reported here because 1200 samples is enough to place it. At 300 samples it

was withheld as n/a rather than printed — see below.

Throughput

1007 requests over 10 seconds, every one a 200. Note that raising concurrency from

16 to 24 did not raise throughput (108 rps → 99 rps): the decision path is not

concurrency-limited at this level, so adding load adds queueing rather than work.

That is worth knowing before sizing a deployment, and it is why the figure is

published with its concurrency attached rather than as a bare "rps". The successful-rps column is

reported separately from total rps on purpose: a throughput number with the

errors hidden is the most common way a benchmark lies, and 400 rps where a third

of responses are 429 is not 400 rps of useful work.

How to read these numbers honestly

OFF is not "CAIN uninstalled". It is a request to the same host on a route

that does not enter the decision path. That isolates fabric cost from network

latency to this box, which is the number you actually want when deciding whether

to adopt. It is *not* a measurement of your agent calling its own tools locally

with no CAIN at all — that number depends entirely on your network, not ours.

p99 is withheld when there are too few samples. The benchmark refuses to print

a percentile unless it has at least 5× the samples needed to place it, because a

"p99" from 300 observations is roughly the third-worst one wearing a formal name.

The figures above come from 1200 samples, which is why p99 appears. Run with

--samples 300 and it correctly reports n/a instead.

SHADOW is not measured, and not estimated. Shadow versus enforcing is a live

property of the deployment, set by a global switch. Measuring shadow would mean

flipping that switch on production traffic, so the benchmark reports

not_measured with the reason rather than printing a plausible number.

ActionProof is the expensive stage, and that is expected. It discharges real

SMT queries about a submitted plan. 68 ms to prove a two-step plan cannot violate

your constraints is a different kind of work from a policy lookup, and it only

runs when you submit a plan. If you never send one, you never pay for it — the

stage reports skipped.

Failure behaviour

The benchmark also measures what happens when a stage has nothing configured. A

verification stage with no profile returns not_configured promptly; it does not

hang waiting for a verifier that has no constraints to check. That matters

because a control which stalls under a missing configuration becomes an outage

rather than a refusal.

What is not benchmarked yet

• Sustained load over hours. These are seconds-long windows. Nothing here

says anything about behaviour under sustained pressure, memory growth, or

connection-pool exhaustion over a day.

• Concurrency beyond 24. Two levels measured (16 and 24); throughput did not

improve, so the useful ceiling is somewhere at or below 16 concurrent callers on

this hardware. Nothing here characterises 100+.

• Self-hosted MCPGate. Its performance depends on your hardware, so a number

measured on ours would be misleading rather than useful.

• Cold start. Warm-up requests are discarded, so these are steady-state

figures. The first request after a deploy pays TLS setup and is slower.

These gaps are listed rather than quietly omitted, and they are on

what CAIN does not do yet.