07 · proof
Capability and reliability are not the same metric.
pass@k measures whether a case passed at least once in k runs. pass^k measures whether it passed every time. The capability gate requires pass@k ≥ 0.90; the regression gate requires pass^k = 1.0; --gate both requires both. A suite that passes the first and fails the second is flaky, and the report says by how much.
python -m hpp eval run examples/reliable-coding/benchmark-suite.json -k 3 --gate both
python -m hpp benchmark -k 3 --json
Measured: HPP benchmark: pass@k=1.00 · pass^k=1.00 · gate=PASS over ten controls (manifest contract, policy enforcement, WorkGraph waves, lane collision, monitor freshness, context provenance, routing floor, event/evidence gate, graph determinism, evidence attestation), three runs each. The JSON report carries the suite's sha256 and the platform it ran on. Exit 0 when the gate passes, 1 when it does not.
A decision made elsewhere is measured, never trusted by default
New in 2.6.0
The harness calls no model. A small decision taken outside it — by a rule, a person, a local model or a hosted typed-decision model — can be recorded as hpp.decision/v1: always advisory; with raise-only it may raise a declared value and never lower it; abstention and instrument-failure are outcomes that change nothing. hpp decide eval runs the decider you name as a command against labelled cases and reports coverage, selective accuracy, confident errors, abstentions and failures separately, with a coverage curve by confidence floor. Version 1 measures choice questions only.
python -m hpp decide validate decision.json
python -m hpp decide eval examples/typed-decisions/gotcha-family-suite.json --decider-command '["python", "examples/typed-decisions/baseline_decider.py"]'
decide eval SUITE — --decider-command takes the decider's argv as a JSON array of strings, run without a shell; omit it to replay the records stored in the suite's cases, where a case without a record counts as an instrument failure. --timeout (default 10.0) is the number of seconds per case before that case counts as an instrument failure. The gate: --min-selective-accuracy (default 0.9), --max-confident-errors (default 0; a confident error is a wrong answer at or above the suite's confidence_floor, 0.6 when the suite does not set one) and --max-failures (default 0). Exit 0 the gate passed · 1 the gate failed, or nothing was decided and there is no accuracy to measure · 2 an invalid record, suite or argv.
decide validate RECORD — prints valid and the effective value a consumer may act on, with its action (raised, kept, advised or none). Exit 0 valid · 2 refused, naming the first broken rule.
Measured from a checkout of main: the baseline decider on the example suite → 15 cases, 12 decided, 3 abstained, 0 instrument failures, coverage 0.8, selective accuracy 1.0, every threshold held, exit 0. The same suite without --decider-command holds no records → 15 instrument failures, no decision was made, exit 1. A record with "authority": "binding" → hpp: authority must be 'advisory': …, exit 2. The 2.5.8 package has no decide command: hpp decide there is an invalid choice, exit 2.
See examples/typed-decisions for the adapter, its risks and how to measure it before trusting it.
Evidence bundles: the exit code of a criterion, measured outside the model
New in 2.6.0
hpp drives no browser and calls no model. hpp evidence run runs the criterion command you declare — an end-to-end spec, a pytest suite, any script — as argv without a shell, measures its exit code, hashes every artifact file you declared, and writes .hpp/evidence/<id>-<UTC>.json (hpp.evidence/v1): the command, the base commit, the exit code, the verdict, stdout and stderr as byte counts and sha256 (never the text), the artifacts and a self-hash. The verdict is passed only when the command exited 0 and every declared pattern matched a file this run wrote (a file untouched since before the run is listed as unchanged and does not count); otherwise it is failed, missing-artifacts, timeout or could-not-start. hpp evidence verify re-derives a record from the files on disk. The self-hash makes an edited record visible; it is not a signature — anyone who can write the file can rewrite it — so a checker that must not trust the maker re-runs the command from its own lane instead of relying on verify alone.
python -m hpp evidence run --id smoke-page --artifact out/report.html --artifact out/smoke.log -- python examples/evidence/smoke_page.py
python -m hpp evidence verify <record_path>
evidence run --id ID [--artifact GLOB]... -- COMMAND — --artifact is a glob relative to the workspace, repeatable; --timeout (default 600.0) is the number of seconds before the run counts as a timeout; --out is the record directory, relative to the workspace (default .hpp/evidence); --record-event appends evidence_recorded to the event log when, and only when, the bundle passed, and needs at least one --artifact. A secret-like argv, an artifact or --out path outside the workspace, a bad id or a non-positive timeout is refused before anything runs. Exit 0 passed · 1 not passed · 2 refused, or the event could not be appended.
evidence verify RECORD — the record_path that run printed. Exit 0 valid, an intact record of a run that passed · 1 not-evidence, an intact record of a run that did not pass · 2 blocked, the record was edited or contradicts itself, or an artifact changed or went missing.
Measured from a checkout of main, in a copy of the tree so that out/ and .hpp/ were written there: the demo → passed, exit 0, and verify on its record → valid, exit 0. With --break after the script, both artifacts were written and the verdict was failed, exit 1; verify on that record → not-evidence, exit 1, and on a copy edited to read passed → blocked, the record was edited after it was written, exit 2. The first record, verified again after the second run had rewritten out/ → blocked, 2 artifact(s) changed since the run, exit 2: end-to-end runners that clear their output directory at the start of each run do the same, so give artifacts that must stay verifiable a per-run directory. --record-event with no work_started in the log → exit 2, the event not appended; after event append --type work_started it moved the loop to evidenced. The 2.5.8 package has no evidence command: hpp evidence there is an invalid choice, exit 2.
See examples/evidence for the same pattern around a real end-to-end runner.
Retrieval ruler: the retriever, measured apart from the answer
New in 2.6.0
The harness runs no index and calls no model. A retriever is a command you declare: it reads {"query": …, "k": …} as JSON on stdin and prints ranked ids, best first. hpp retrieval eval scores the top k of each answer against the ids an hpp.retrieval-suite/v1 suite labels relevant — hit@k, recall@k, precision@k, MRR and nDCG@k — and counts instrument failures (could not start, timeout, non-zero exit, output that is not JSON, a duplicate id) apart: they are never scored, and the means cover measured cases only. With no measured case the metrics are null and the gate says why, never 0%. Whether a good answer can be generated from what was retrieved is a separate question, not asked here.
python -m hpp retrieval eval examples/retrieval/suite.json
python -m hpp retrieval eval examples/retrieval/suite.json --retriever-command '["python", "examples/retrieval/keyword_retriever.py"]'
retrieval eval SUITE — --retriever-command takes the retriever's argv as a JSON array of strings, run without a shell; omit it to replay the results stored in the suite's cases. -k overrides the suite's cut-off. --timeout (default 10.0) is the number of seconds per case before that case counts as an instrument failure. The gate: at least one measured case, --min-recall (default 0.8) for mean recall@k and --max-failures (default 0). Exit 0 the gate passed · 1 the gate failed · 2 a refused suite or argument.
Measured from a checkout of main: both commands → 7 cases measured, 0 instrument failures, mean recall@3 0.786, hit@3 0.857, MRR 0.786, nDCG@3 0.749, mean recall@k 0.786 < 0.8, exit 1 — the keyword baseline misses the paraphrased question entirely, a measured 0, and the default gate fails; with --min-recall 0.75 → every threshold held, exit 0. A retriever that exits 3 on every case → 7 instrument failures, metrics null, no case was measured (7 instrument failures); there is no recall to measure, exit 1. --retriever-command '"python"' → hpp: --retriever-command must be a JSON array of strings (argv, no shell), exit 2. The suite is synthetic and its baseline was written beside it: these numbers prove the ruler works, not that any retriever is good. The 2.5.8 package has no retrieval command: there it is an invalid choice, exit 2.
See examples/retrieval for the retriever contract and the suite format.
Citation check: every marker resolves, or the exit code says which does not
New in 2.6.0
An answer written from sources marks each claim with the id of the source it rests on — [ID:<id>] by default. hpp cite check reads the answer and the list of context items it was written from, and reports without a model: a marker whose id is not in the context (UNKNOWN_ID), a marker naming a range or a list (RANGE) and an empty marker (EMPTY_MARKER) block; more than --max-per-sentence markers in one sentence (TOO_MANY) and a sentence with a number, percentage, currency amount or date and no marker (UNCITED_CLAIM) warn. It proves that every marker resolves, not that the cited source supports the sentence: that judgement needs a reader. The sentence splitter and the number detector are heuristics, listed in the docstring of hpp/citations.py.
python -m hpp cite check --text examples/citations/answer.md --context examples/citations/context.json
cite check --text FILE --context FILE — --context is a JSON list of items with an id (and their text), or of bare ids; --max-per-sentence (default 4); --marker takes a regex with exactly one capture group, the id. Empty text, text that is only code, secret-like text or context and an unusable regex are refused. Exit 0 clean · 1 warn (TOO_MANY, UNCITED_CLAIM) · 2 block (UNKNOWN_ID, RANGE, EMPTY_MARKER) or refused input.
Measured from a checkout of main: the shipped answer → verdict ok, 7 sentences, 5 markers, 4 of the 5 context ids cited, exit 0. In copies of it: [ID:glossary] changed to [ID:glossary-v2] → UNKNOWN_ID, exit 2; to [ID:runbook-7,sla-2026] → RANGE, exit 2; the marker removed from the sentence with 4 engineers → UNCITED_CLAIM, exit 1. And the limit, measured: 42 minutes changed to 90 minutes, still cited to the source that says 42 → verdict ok, exit 0. The 2.5.8 package has no cite command: there it is an invalid choice, exit 2.
See examples/citations for another marker syntax and the report format.
Attestation binds an approval to bytes
hpp attest create records a verdict together with the spec hash, the base commit, a digest of the full repository snapshot (tracked and untracked), the maker, the checker and the session. Maker and checker must differ, or the record is refused. hpp attest verify re-derives every binding and blocks when any of them moved.
python -m hpp attest create --repo . --spec spec.md --output att.json --maker a --checker b --session s1 --verdict approved
python -m hpp attest verify att.json --repo .
Measured in a scratch repository: --maker a --checker a → hpp: maker and checker must be different non-empty actors, exit 2. With different actors the record is approved and verify returns valid, exit 0; after one line of the spec changed, verify returns blocked with mismatches: spec_sha256, snapshot_digest, exit 2. Attestation requires git.
See the claim matrix and the benchmark contract.