Auditor's runbook · Verify the fasten audit chain offline

The membrane UI has a fast tail-verify indicator good enough for a live health signal. This runbook is what an auditor uses to prove tamper-evidence against a snapshot, without trusting any endpoint the audited entity controls.

What this proves

Every fasten audit row carries prev_hash and hash, chained by SHA-256 over a versioned canonical form of the row. A verifier reads the rows in monotonic_seq order and confirms each row's prev_hash equals the previous row's hash. When the chain verifies clean, one of two things is true:

1. Nothing has been tampered with since the rows were written; or
2. The tampering party had access to a private hash-collision oracle for SHA-256, which they do not.

When it does not verify, first_break_at names the first offending monotonic_seq. Every row before that seq remains trustworthy; every row at or after must be re-ingested from the upstream source before it can be relied on.

Prerequisites

• A copy of the fasten audit database. Snapshot the file and verify against the snapshot, not the live DB.
• Python 3.10+ with fasten installed. pip install fasten or from source at github.com/nerdapplabs/fasten.

Recipes

Three recipes. Each is a collapsed card. Click to expand.

python -c 'import fasten; from fasten.store.sqlite import SQLiteStore; \
  s=SQLiteStore("./membrane-audit.db"); \
  rows=s.query(limit=10_000_000); \
  r=fasten.verify_chain(rows); \
  print({"ok": r.ok, "rows": r.total_rows, "first_break_at": r.first_break_at, "reason": r.reason})'

{'ok': True, 'rows': 1284, 'first_break_at': None, 'reason': None}

{'ok': False, 'rows': 1284, 'first_break_at': 512, 'reason': 'hash mismatch at seq 512'}

#!/usr/bin/env python3
"""Offline verifier for a fasten audit DB. Exit 0 clean, 2 broken, 3 error."""
import argparse, sys
import fasten
from fasten.store.sqlite import SQLiteStore

p = argparse.ArgumentParser()
p.add_argument("--db", required=True, help="path to the audit sqlite file")
p.add_argument("--limit", type=int, default=10_000_000)
args = p.parse_args()

try:
    store = SQLiteStore(args.db)
    rows = store.query(limit=args.limit)
    result = fasten.verify_chain(rows)
except Exception as exc:
    print(f"error: {exc.__class__.__name__}: {exc}", file=sys.stderr)
    sys.exit(3)

print(f"rows checked: {result.total_rows}")
if result.ok:
    print("chain: CLEAN")
    sys.exit(0)
print(f"chain: BROKEN")
print(f"first_break_at: {result.first_break_at}")
print(f"reason: {result.reason}")
sys.exit(2)

python verify_chain.py --db ./snapshots/2026-07-09/membrane-audit.db
# rows checked: 1284
# chain: CLEAN
echo $?
# 0

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.12"}
      - run: pip install fasten
      - name: Snapshot the audit DB from prod
        run: aws s3 cp s3://fasten-prod-audit/latest.db ./membrane-audit.db
      - name: Verify the chain
        run: python verify_chain.py --db ./membrane-audit.db

If the chain is broken

The recovery path depends on where the break is and what caused it.

1. Identify the seq. The verifier prints first_break_at. Rows with monotonic_seq less than this value remain trustworthy.
2. Preserve the tampered snapshot. Copy the current DB to a read-only location before any recovery step. The tampered state is evidence.
3. Re-ingest from the upstream source. Every fasten emitter is idempotent by (request_id, code, target), so replaying the source stream from first_break_at onward will re-seal the chain from a known-good prefix.
4. Report. A chain break is a governance event in its own right. Emit a chain-integrity alert row into the recovered chain with the seq range that was replayed, so the audit trail carries its own remediation record.

Scope · what this catches, what it does not

Beyond a linear chain · Merkle roots and transparency logs

A linear chain proves that no one has tampered with the rows the chain names. It does not prove that no one has replaced the whole DB with a fresh, self-consistent one. That threat is defeated by publishing a Merkle root of the audit window to a party outside the audited entity. The same construction Certificate Transparency uses.

fasten's signed evidence-pack export ships an RFC 6962 Merkle root and per-row inclusion proofs, so an auditor can hold a single 32-byte commitment and verify any individual row's presence without downloading the whole log. The linear chain in this runbook is the substrate under that. The evidence-pack API is the layer above.