Measurement and Maturity¶

The harness turns a claim into evidence. A claim like "an LLM found N bypasses" is meaningless without stating the instrument, the environment, and the population of attempts. This page describes the discipline the record enforces.

Deterministic campaigns¶

An inputs campaign with a committed manifest is deterministic in the generator sense: replay produces the same diff hashes. Whether the verdicts are reproducible depends on the pinned (TrustSight version, environment) pair.

If the campaign declares environment.accumulate: true, verdicts become order-dependent even with deterministic inputs, because the database warms as the campaign runs. Such a campaign is classified as stochastic.

Stochastic campaigns¶

LLM and mutation campaigns produce different diffs on different runs. Their results are reported as binomial proportions with Wilson intervals, not as reproducible counts. Compare them only under the same (TrustSight version, environment, prompt hash) triple.

Lower-bound reporting¶

The behaviour validator is conservative: a false negative discards one attempt; a false positive certifies a dead payload. Because false positives are worse, the validator refuses when uncertain. Therefore every bypass count is a lower bound. The record states this explicitly in bypass_rate.note.

The other side¶

A bypass count alone rewards a tool that flags everything. The harness therefore also measures the false-positive rate: python -m harness benign scans a corpus of benign diffs through the stateless analysis and reports how many a healthy update causes TrustSight to flag, with the same Wilson interval. That path is separate from a campaign on purpose - a benign update has no attack chain for the behaviour validator to certify, and an unflagged benign diff is not a bypass for the exporter to file.

What a record cannot claim¶

A campaign record is valid only for itself. It cannot say:

• "TrustSight is robust"; only "this generator, this version, this many attempts, found N proven bypasses."
• "Version A is better than version B"; that is the regression gate's job, expressed as open/closed bypasses.
• "The true bypass rate is exactly X"; the Wilson interval gives a range, and the lower-bound note says the true count may be higher.

These limits are not accidents; they are enforced by the schema and by the Judge.