docs/contract-execution-pinning.md

@@ -0,0 +1,90 @@
+# Contract execution pinning
+
+Which copy of a contract script actually ran, and how that is proven.
+
+## Why this exists
+
+Three times on 2026-09-25 a contract reported a verdict from a copy that was
+not the merged one:
+
+1. The ops lane's own clone sat on the merged feature branch
+   `fix/search-stack-multi-engine-20260925` at `8b2eba4` with no `pve_auth`
+   fix, while it executed the daily digest from a different clone. Nothing in
+   the workflow noticed.
+2. `scripts/search-stack-check.py` was deployed into the pinned runner clone
+   by hand rather than through git.
+3. A stale local `origin/master` ref made an ancestry check report
+   "unlanded work" for a branch that had in fact merged — the same staleness
+   would have passed a stale script as current.
+
+A contract verdict is only meaningful if it came from the merged copy. The
+control is `scripts/revision-preflight.sh`.
+
+## The rule
+
+**Every contract pins exactly one clone for execution: the clone that
+`scripts/contract-run.sh` itself lives in.**
+
+`contract-run.sh` derives that from its own location (`SCRIPTS_DIR`) and checks
+the script it is about to run against `origin/master` in the same clone. There
+is no second path to configure, and no contract may be executed from a
+hand-copied location.
+
+| Contract | Script | Pinned clone |
+| --- | --- | --- |
+| `infrastructure-monitoring` | `scripts/infra-monitoring.sh` | the clone containing `contract-run.sh` |
+| `proxmox-monitor` | `scripts/proxmox-monitor.sh` | same |
+| `zulip-health` | `scripts/zulip-monitor.sh` | same |
+| `agent-health-check` | `scripts/agent-health-check.py` | same |
+| `litellm-health` | `scripts/litellm-health-check.py` | same |
+| `disk-gc-threat-response` | `scripts/disk-gc-scan.py` | same |
+| `pm2-self-heal` | `scripts/pm2-self-heal.sh` | same |
+| `search-stack-visibility` | `scripts/search-stack-check.py` | same |
+
+### The deployed runner
+
+The scheduler on **CT 100 (abiba)** runs contracts from
+**`/opt/contract-runner`** via `/etc/cron.d/contract-runner`. That clone is the
+pinned execution copy for every scheduled contract, and it must be kept current
+with `master` by fast-forward. Its `origin` is a local path to the upstream
+working copy, not a network remote.
+
+`daily-health-digest` is **not** in the table above because it has no contract
+file and no mapping — it is dispatched by cron as
+`fm-send.sh ops "run contract: daily-health-digest"` and was, until
+2026-09-25, executed by hand from whichever clone the operator happened to be
+in. Creating its contract file and pinning it to a clone is an open follow-up.
+
+## How the check works
+
+`scripts/revision-preflight.sh <script-path> <clone-path>`:
+
+* resolves the **repo-relative** path of the executing script inside the clone;
+* **fetches** the remote first, so a stale local ref cannot make a stale script
+  look current;
+* compares the script's sha256 against `<ref>:<repo-relative-path>`;
+* **fails closed** — a path absent from the ref, an unresolvable ref, or a
+  failed fetch is a failure, never a warning.
+
+Exit `0` means verified match. Exit `1` means mismatch or unverifiable.
+
+## Modes in `contract-run.sh`
+
+| `CONTRACT_REVISION_PREFLIGHT` | Behaviour |
+| --- | --- |
+| unset / `enforce` (default) | withhold the verdict, alert, exit `2` |
+| `warn` | log the mismatch and continue |
+| `off` | skip the check entirely |
+
+`enforce` is the default deliberately: an unverifiable copy is indistinguishable
+from a stale or hand-edited one, and a verdict from it is worse than no verdict.
+
+## Operating notes
+
+* A stale pinned clone will now make contracts **withhold** rather than report.
+  That is the intended failure. Recover by fast-forwarding the pinned clone:
+  `git -C /opt/contract-runner pull --ff-only`.
+* When a contract legitimately changes, land it through the normal branch + PR
+  path and fast-forward the pinned clone. Do not copy files into it by hand.
+* `--no-fetch` exists for offline inspection; it prints that freshness is
+  assumed rather than verified, and it is not used by `contract-run.sh`.

scripts/contract-run.sh

@@ -18,6 +18,14 @@
 #   litellm-health            -> scripts/litellm-health-check.py
 #   disk-gc-threat-response   -> scripts/disk-gc-scan.py
 #   pm2-self-heal             -> scripts/pm2-self-heal.sh
+#   search-stack-visibility   -> scripts/search-stack-check.py
+#
+# Execution copy: every contract pins the clone this script lives in (see
+# docs/contract-execution-pinning.md). Before a contract runs, this wrapper
+# proves the script it is about to execute byte-matches origin/master:
+#   CONTRACT_REVISION_PREFLIGHT=enforce (default) refuse to report on mismatch
+#   CONTRACT_REVISION_PREFLIGHT=warn            log the mismatch and continue
+#   CONTRACT_REVISION_PREFLIGHT=off             skip the check entirely
 #
 # Exit codes:
 #   0 = contract passed
@@ -111,6 +119,35 @@ echo "Started: $(date -u '+%Y-%m-%d %H:%M:%S UTC')" | tee -a "$LOG_FILE"
 echo "Script: $SCRIPT_PATH" | tee -a "$LOG_FILE"
 echo "" | tee -a "$LOG_FILE"
 
+# ── Revision preflight ───────────────────────────────────────────────────────
+# A verdict is only meaningful if it came from the merged copy. Refuse to report
+# one from a mismatched or unverifiable copy; that is a probe failure (exit 2),
+# not a contract verdict, because the result would be untrustworthy.
+# See docs/contract-execution-pinning.md.
+REVISION_PREFLIGHT_MODE="${CONTRACT_REVISION_PREFLIGHT:-enforce}"
+REPO_ROOT="$(cd "${SCRIPTS_DIR}/.." && pwd)"
+if [ "$REVISION_PREFLIGHT_MODE" != "off" ] && [ -x "${SCRIPTS_DIR}/revision-preflight.sh" ]; then
+    if "${SCRIPTS_DIR}/revision-preflight.sh" "$SCRIPT_PATH" "$REPO_ROOT" 2>&1 | tee -a "$LOG_FILE"; then
+        :
+    elif [ "$REVISION_PREFLIGHT_MODE" = "warn" ]; then
+        echo "⚠️  revision preflight failed — continuing because CONTRACT_REVISION_PREFLIGHT=warn" | tee -a "$LOG_FILE"
+    else
+        echo "🚫 VERDICT WITHHELD: executing copy does not match the merged revision" | tee -a "$LOG_FILE"
+        ALERT_MSG="🔴 Contract $CONTRACT_NAME: revision mismatch — verdict withheld. Log: $LOG_FILE"
+        ZULIP_API_URL="${ZULIP_API_URL:-https://chat.sysloggh.net/api/v1}"
+        ZULIP_API_KEY="${ZULIP_API_KEY:-}"
+        ZULIP_USER="${ZULIP_USER:-abiba-bot@chat.sysloggh.net}"
+        if [ -n "$ZULIP_API_KEY" ] && command -v curl &> /dev/null; then
+            curl -sf -X POST "${ZULIP_API_URL}/messages" \
+                -u "${ZULIP_USER}:${ZULIP_API_KEY}" \
+                -d "type=private" \
+                -d "to=9" \
+                -d "content=${ALERT_MSG}" > /dev/null 2>&1 || true
+        fi
+        exit 2
+    fi
+fi
+
 # Use timeout to prevent hangs (10 minutes default)
 TIMEOUT=600
 timeout "$TIMEOUT" $INTERPRETER "$SCRIPT_PATH" 2>&1 | tee -a "$LOG_FILE"

scripts/revision-preflight.sh

@@ -0,0 +1,128 @@
+#!/usr/bin/env bash
+# revision-preflight.sh — prove the copy a contract is about to execute is the
+# copy that is merged.
+#
+# Usage:
+#   revision-preflight.sh [options] <script-path> <clone-path>
+#
+# Options:
+#   --ref <ref>   Ref to compare against (default: origin/master)
+#   --no-fetch    Do not refresh the ref first (see FRESHNESS below)
+#   --quiet       Print nothing on success
+#   -h, --help    Show this help
+#
+# Exit codes:
+#   0  the executing script byte-matches <ref>:<repo-relative-path>
+#   1  MISMATCH, or the revision could not be resolved (see FAIL CLOSED)
+#
+# FRESHNESS
+#   A guard is only as good as the ref it compares against. On 2026-09-25 a
+#   stale local origin/master made an ancestry check on this fleet report
+#   "unlanded work" for a branch that had in fact merged, and it would equally
+#   have passed a stale script as current. So by default this guard FETCHES the
+#   remote before comparing. With --no-fetch it compares against whatever the
+#   local ref points at and says so out loud; it never silently assumes
+#   freshness.
+#
+# FAIL CLOSED
+#   An unresolvable path or ref is a FAILURE, never a warning. "Cannot verify"
+#   is precisely the state a stale or hand-edited copy produces, so treating it
+#   as success would defeat the guard. The original draft of this script did
+#   exactly that: it resolved the master revision with
+#   `git show origin/master:$(basename "$SCRIPT")`, which drops the scripts/
+#   prefix, queries the repo root, fails, and exited 0 — passing a script that
+#   exists in no revision at all.
+#
+# WHICH CLONE
+#   Pass the clone the contract is actually executing from. See
+#   docs/contract-execution-pinning.md for which clone each contract pins.
+
+set -euo pipefail
+
+REF="origin/master"
+FETCH=1
+QUIET=0
+
+usage() {
+    sed -n '2,45p' "$0" | sed 's/^# \{0,1\}//'
+}
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --ref)
+            [[ $# -ge 2 ]] || { echo "revision-preflight: --ref needs a value" >&2; exit 1; }
+            REF="$2"; shift 2 ;;
+        --no-fetch) FETCH=0; shift ;;
+        --quiet) QUIET=1; shift ;;
+        -h|--help) usage; exit 0 ;;
+        --) shift; break ;;
+        -*) echo "revision-preflight: unknown option: $1" >&2; exit 1 ;;
+        *) break ;;
+    esac
+done
+
+if [[ $# -lt 2 ]]; then
+    usage >&2
+    exit 1
+fi
+
+SCRIPT="$1"
+CLONE="$2"
+
+say() { [[ $QUIET -eq 1 ]] || echo "$@" >&2; }
+fail() { echo "❌ revision-preflight: $*" >&2; exit 1; }
+
+# ── 1. inputs must exist ──────────────────────────────────────────────────────
+[[ -f "$SCRIPT" ]] || fail "executing script not found: $SCRIPT"
+[[ -d "$CLONE" ]] || fail "clone path is not a directory: $CLONE"
+git -C "$CLONE" rev-parse --git-dir >/dev/null 2>&1 \
+    || fail "not a git clone: $CLONE"
+
+# ── 2. resolve the repo-relative path (the original defect) ───────────────────
+CLONE_ABS=$(cd "$CLONE" && pwd)
+SCRIPT_ABS=$(cd "$(dirname "$SCRIPT")" && pwd)/$(basename "$SCRIPT")
+case "$SCRIPT_ABS" in
+    "$CLONE_ABS"/*) REL="${SCRIPT_ABS#"$CLONE_ABS"/}" ;;
+    *) fail "script is outside the clone: $SCRIPT_ABS is not under $CLONE_ABS" ;;
+esac
+
+# ── 3. refresh the ref so staleness cannot mask a stale script ────────────────
+if [[ $FETCH -eq 1 ]]; then
+    REMOTE="${REF%%/*}"
+    [[ "$REMOTE" == "$REF" ]] && REMOTE="origin"
+    if ! git -C "$CLONE" fetch --quiet "$REMOTE" 2>/dev/null; then
+        fail "cannot fetch '$REMOTE' in $CLONE — refusing to verify against a possibly stale '$REF'. Re-run with network access, or pass --no-fetch to compare against the local ref deliberately."
+    fi
+else
+    say "⚠️  revision-preflight: --no-fetch — comparing against the LOCAL '$REF'; freshness is assumed, not verified"
+fi
+
+# ── 4. resolve the merged revision; unresolvable is a failure ────────────────
+git -C "$CLONE" rev-parse --verify --quiet "$REF" >/dev/null \
+    || fail "ref '$REF' does not resolve in $CLONE"
+REF_COMMIT=$(git -C "$CLONE" rev-parse --short "$REF")
+
+TMPFILE=$(mktemp)
+trap 'rm -f "$TMPFILE"' EXIT
+
+if ! git -C "$CLONE" show "$REF:$REL" > "$TMPFILE" 2>/dev/null; then
+    fail "'$REL' does not exist in $REF ($REF_COMMIT) — cannot verify $SCRIPT. A path that is absent from $REF can never be a merged copy."
+fi
+
+# ── 5. compare ───────────────────────────────────────────────────────────────
+EXEC_SHA=$(sha256sum "$SCRIPT" | cut -d' ' -f1)
+MERGED_SHA=$(sha256sum "$TMPFILE" | cut -d' ' -f1)
+
+if [[ "$EXEC_SHA" != "$MERGED_SHA" ]]; then
+    {
+        echo "❌ revision-preflight: MISMATCH — refusing to report from this copy"
+        echo "   script:   $SCRIPT_ABS"
+        echo "   clone:    $CLONE_ABS"
+        echo "   executed: $EXEC_SHA"
+        echo "   merged:   $MERGED_SHA  ($REF:$REL @ $REF_COMMIT)"
+    } >&2
+    exit 1
+fi
+
+say "✅ revision-preflight: $REL matches $REF @ $REF_COMMIT ($EXEC_SHA)"
+exit 0

tests/fixtures/revision-preflight.prefix.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Revision preflight guard: verify the script being executed matches origin/master
+# Usage: revision-preflight.sh <script-path> <clone-path>
+# Returns 0 if match, 1 if mismatch (prints both revisions)
+
+set -euo pipefail
+
+SCRIPT="${1:?Usage: revision-preflight.sh <script-path> <clone-path>}"
+CLONE="${2:?Usage: revision-preflight.sh <script-path> <clone-path>}"
+
+# Compute sha256 of the script being executed
+EXEC_SHA=$(sha256sum "$SCRIPT" | cut -d' ' -f1)
+
+# Compute sha256 of the merged origin/master version
+# Extract to a temp file to avoid pipe issues
+TMPFILE=$(mktemp)
+trap 'rm -f "$TMPFILE"' EXIT
+
+# Try to extract the file from origin/master
+if git -C "$CLONE" show "origin/master:$(basename "$SCRIPT")" > "$TMPFILE" 2>/dev/null; then
+    MASTER_SHA=$(sha256sum "$TMPFILE" | cut -d' ' -f1)
+else
+    echo "⚠️  revision-preflight: could not resolve origin/master revision for $(basename "$SCRIPT")" >&2
+    exit 0  # Warn but don't block if git show fails
+fi
+
+if [[ -z "$MASTER_SHA" || "$MASTER_SHA" == "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" ]]; then
+    echo "⚠️  revision-preflight: could not resolve origin/master revision for $(basename "$SCRIPT")" >&2
+    exit 0  # Warn but don't block if git show fails
+fi
+
+if [[ "$EXEC_SHA" != "$MASTER_SHA" ]]; then
+    echo "⚠️  revision-preflight: MISMATCH detected" >&2
+    echo "   Executed: $EXEC_SHA ($(basename "$SCRIPT"))" >&2
+    echo "   Merged:   $MASTER_SHA (origin/master:$(basename "$SCRIPT"))" >&2
+    exit 1
+else
+    echo "✅ revision-preflight: $SCRIPT matches origin/master ($EXEC_SHA)" >&2
+    exit 0
+fi

tests/test_revision_preflight.sh

@@ -0,0 +1,182 @@
+#!/usr/bin/env bash
+# Behavioural tests for scripts/revision-preflight.sh
+#
+# Every case builds a throwaway clone with a real bare remote, so origin/master
+# is genuine and the guard's fetch path is exercised. Nothing outside mktemp is
+# touched.
+#
+# The pre-fix draft is kept at tests/fixtures/revision-preflight.prefix.sh and
+# is run against the SAME cases, to prove these tests bite: the pre-fix guard
+# exits 0 where the fixed guard exits 1.
+
+set -uo pipefail
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO="$(cd "$HERE/.." && pwd)"
+GUARD="$REPO/scripts/revision-preflight.sh"
+PREFIX_GUARD="$HERE/fixtures/revision-preflight.prefix.sh"
+
+PASS=0
+FAIL=0
+FAILED_CASES=()
+
+pass() { printf '  ✓ %s\n' "$1"; PASS=$((PASS + 1)); }
+fail() { printf '  ✗ %s\n' "$1"; FAIL=$((FAIL + 1)); FAILED_CASES+=("$1"); }
+
+# Build a clone with a real remote; echo the clone path.
+make_clone() {
+    local tmp
+    tmp="$(mktemp -d)"
+    git init --bare -q "$tmp/remote.git"
+    git init -q "$tmp/clone"
+    (
+        cd "$tmp/clone" || exit 1
+        git config user.email test@example.invalid
+        git config user.name test
+        mkdir -p scripts
+        printf '#!/bin/bash\necho hello\n' > scripts/demo.sh
+        chmod +x scripts/demo.sh
+        git add -A
+        git commit -qm init
+        git branch -M master
+        git remote add origin "$tmp/remote.git"
+        git push -q origin master
+        git fetch -q origin
+    )
+    echo "$tmp/clone"
+}
+
+echo "== revision-preflight behavioural tests =="
+
+# ── 1. match → exit 0 ────────────────────────────────────────────────────────
+echo "1. matching copy"
+C=$(make_clone)
+if out=$("$GUARD" "$C/scripts/demo.sh" "$C" 2>&1); then
+    pass "matching copy exits 0"
+else
+    fail "matching copy should exit 0 (got $?, output: $out)"
+fi
+if [[ -z "$( "$GUARD" --quiet "$C/scripts/demo.sh" "$C" 2>&1 )" ]]; then
+    pass "--quiet prints nothing on a match"
+else
+    fail "--quiet should print nothing on a match"
+fi
+rm -rf "$(dirname "$C")"
+
+# ── 2. mismatch → exit 1 and names both hashes ───────────────────────────────
+echo "2. mismatched copy"
+C=$(make_clone)
+printf '#!/bin/bash\necho TAMPERED\n' > "$C/scripts/demo.sh"
+out=$("$GUARD" "$C/scripts/demo.sh" "$C" 2>&1); rc=$?
+if [[ $rc -eq 1 ]]; then pass "mismatch exits 1"; else fail "mismatch should exit 1 (got $rc)"; fi
+if [[ "$out" == *"MISMATCH"* ]]; then pass "mismatch says MISMATCH"; else fail "mismatch should say MISMATCH"; fi
+if [[ "$out" == *"executed:"* && "$out" == *"merged:"* ]]; then
+    pass "mismatch prints both revisions"
+else
+    fail "mismatch should print both revisions"
+fi
+rm -rf "$(dirname "$C")"
+
+# ── 3. paths under scripts/ resolve (the original basename defect) ───────────
+echo "3. repo-relative path resolution"
+C=$(make_clone)
+if "$GUARD" --quiet "$C/scripts/demo.sh" "$C" >/dev/null 2>&1; then
+    pass "script under scripts/ resolves against origin/master"
+else
+    fail "script under scripts/ must resolve (basename defect)"
+fi
+rm -rf "$(dirname "$C")"
+
+# ── 4. script that exists in NO revision → must fail ─────────────────────────
+echo "4. untracked script present in no revision"
+C=$(make_clone)
+printf '#!/bin/bash\necho never committed\n' > "$C/scripts/ghost.sh"
+out=$("$GUARD" "$C/scripts/ghost.sh" "$C" 2>&1); rc=$?
+if [[ $rc -eq 1 ]]; then pass "ghost script exits 1"; else fail "ghost script must exit 1 (got $rc)"; fi
+if [[ "$out" == *"does not exist in"* ]]; then
+    pass "ghost script says it is absent from the ref"
+else
+    fail "ghost script should say it is absent from the ref"
+fi
+rm -rf "$(dirname "$C")"
+
+# ── 5. unresolvable ref → must fail ──────────────────────────────────────────
+echo "5. unresolvable ref"
+C=$(make_clone)
+out=$("$GUARD" --no-fetch --ref origin/nope "$C/scripts/demo.sh" "$C" 2>&1); rc=$?
+if [[ $rc -eq 1 ]]; then pass "unresolvable ref exits 1"; else fail "unresolvable ref must exit 1 (got $rc)"; fi
+rm -rf "$(dirname "$C")"
+
+# ── 6. missing script → must fail ────────────────────────────────────────────
+echo "6. missing script"
+C=$(make_clone)
+out=$("$GUARD" "$C/scripts/nope.sh" "$C" 2>&1); rc=$?
+if [[ $rc -eq 1 ]]; then pass "missing script exits 1"; else fail "missing script must exit 1 (got $rc)"; fi
+rm -rf "$(dirname "$C")"
+
+# ── 7. script outside the clone → must fail ──────────────────────────────────
+echo "7. script outside the clone"
+C=$(make_clone)
+OUTSIDE=$(mktemp)
+printf '#!/bin/bash\necho outside\n' > "$OUTSIDE"
+out=$("$GUARD" "$OUTSIDE" "$C" 2>&1); rc=$?
+if [[ $rc -eq 1 ]]; then pass "outside script exits 1"; else fail "outside script must exit 1 (got $rc)"; fi
+rm -f "$OUTSIDE"; rm -rf "$(dirname "$C")"
+
+# ── 8. --no-fetch states the freshness assumption ────────────────────────────
+echo "8. --no-fetch states its assumption"
+C=$(make_clone)
+out=$("$GUARD" --no-fetch "$C/scripts/demo.sh" "$C" 2>&1)
+if [[ "$out" == *"freshness is assumed"* ]]; then
+    pass "--no-fetch states the freshness assumption"
+else
+    fail "--no-fetch should state the freshness assumption"
+fi
+rm -rf "$(dirname "$C")"
+
+# ── 9. the pre-fix guard must FAIL these same cases (proves the tests bite) ──
+echo "9. pre-fix draft fails the same cases (bite proof)"
+if [[ ! -f "$PREFIX_GUARD" ]]; then
+    fail "pre-fix fixture missing: $PREFIX_GUARD"
+else
+    # 9a. repo-relative path: pre-fix drops scripts/ and cannot resolve
+    C=$(make_clone)
+    out=$("$PREFIX_GUARD" "$C/scripts/demo.sh" "$C" 2>&1); rc=$?
+    if [[ $rc -eq 0 && "$out" == *"could not resolve"* ]]; then
+        pass "pre-fix: exits 0 and cannot resolve scripts/demo.sh (defect confirmed)"
+    else
+        fail "pre-fix should exit 0 with 'could not resolve' (got rc=$rc)"
+    fi
+    rm -rf "$(dirname "$C")"
+
+    # 9b. ghost script: pre-fix passes a script that exists in no revision
+    C=$(make_clone)
+    printf '#!/bin/bash\necho never committed\n' > "$C/scripts/ghost.sh"
+    out=$("$PREFIX_GUARD" "$C/scripts/ghost.sh" "$C" 2>&1); rc=$?
+    if [[ $rc -eq 0 ]]; then
+        pass "pre-fix: PASSES a ghost script that exists in no revision (defect confirmed)"
+    else
+        fail "pre-fix was expected to wrongly pass the ghost script (got rc=$rc)"
+    fi
+    rm -rf "$(dirname "$C")"
+
+    # 9c. a file that DOES exist at the repo root still works pre-fix, showing
+    #     the defect is specific to nested paths
+    C=$(make_clone)
+    printf '#!/bin/bash\necho root\n' > "$C/rootlevel.sh"
+    ( cd "$C" && git add rootlevel.sh && git commit -qm root && git push -q origin master && git fetch -q origin )
+    if "$PREFIX_GUARD" "$C/rootlevel.sh" "$C" >/dev/null 2>&1; then
+        pass "pre-fix: root-level path resolves (so the defect is the basename, not git)"
+    else
+        fail "pre-fix should resolve a root-level tracked file"
+    fi
+    rm -rf "$(dirname "$C")"
+fi
+
+echo
+echo "  passed: $PASS   failed: $FAIL"
+if [[ $FAIL -gt 0 ]]; then
+    printf '  FAILED: %s\n' "${FAILED_CASES[@]}"
+    exit 1
+fi
+echo "All revision-preflight tests passed."