Skip to content
runlog_Guide
Menu
HomeRunLog
Ship-Check · Local release checks

Check recorded agent actions
before release.

Use Ship-Check to check tool calls from your own test harness against step, cost and tool-use assertions. Start locally without an account, API key or model call.

Start with the offline example

  1. Download the reviewed 0.3.1 source archive and verify its checksum using the installation guide.
  2. From the extracted ship-check-0.3.1 folder, record and check the example below. It executes a real in-memory lookup tool with zero provider cost.
  3. Replace the example with traces from your own agent tests before relying on the result.
python3 examples/harness.py --output /tmp/ship-check-demo
python3 runner.py --offline --scenarios /tmp/ship-check-demo/suite.json --trace /tmp/ship-check-demo/trace.json

Expected: LOCAL PASS and exit 0. The example demonstrates wiring; it does not evaluate your agent.

Verify that a failure stops the check

python3 examples/harness.py --output /tmp/ship-check-demo --fail
python3 runner.py --offline --scenarios /tmp/ship-check-demo/suite.json --trace /tmp/ship-check-demo/trace.json

Expected: exit 1 because the example exceeds its step ceiling. Offline success is a local assertion result, not signed release approval.

Download and install Ship-Check →

Python 3.12; no runtime dependencies. Read the versioned offline guide ↗

Use your own recorded test run

Capture actual tool calls and costs with a trusted harness. Missing or unknown cost must not be recorded as zero. The local checker evaluates supported assertions on those records; it does not run your agent, establish evaluator quality, or prove that a caller-provided trace is true.

Run the offline checker in your own CI and require its passing job before merging. Review your harness, assertions and workflow together. Keep credentials out of untrusted pull-request code.

Phase 2: hosted signed gates

Hosted signed gates, account provisioning and new paid access are deferred. Existing pilot integrations are preserved; they are not required for the local tool.

Version 0.3.1 supports the runlog.trace-gate.v3 contract. Signed mode needs a provisioned RunLog API key, trusted issuer public key and evidence binding your suite, workflow, report and repository commits. It signs the trace-loop-policy decision and supplied hashes; it does not independently execute your agent or attest evaluator provenance.

The CI starter distinguishes offline PR checks from signed verification on protected main. Its hosted section is for existing provisioned pilots. Contact support for existing-access issues.

Add Loop-Guard to check calls while your agent runs.