diff --git a/docs/contract-execution-pinning.md b/docs/contract-execution-pinning.md new file mode 100644 index 0000000..f27aeb2 --- /dev/null +++ b/docs/contract-execution-pinning.md @@ -0,0 +1,90 @@ +# Contract execution pinning + +Which copy of a contract script actually ran, and how that is proven. + +## Why this exists + +Three times on 2026-09-25 a contract reported a verdict from a copy that was +not the merged one: + +1. The ops lane's own clone sat on the merged feature branch + `fix/search-stack-multi-engine-20260925` at `8b2eba4` with no `pve_auth` + fix, while it executed the daily digest from a different clone. Nothing in + the workflow noticed. +2. `scripts/search-stack-check.py` was deployed into the pinned runner clone + by hand rather than through git. +3. A stale local `origin/master` ref made an ancestry check report + "unlanded work" for a branch that had in fact merged — the same staleness + would have passed a stale script as current. + +A contract verdict is only meaningful if it came from the merged copy. The +control is `scripts/revision-preflight.sh`. + +## The rule + +**Every contract pins exactly one clone for execution: the clone that +`scripts/contract-run.sh` itself lives in.** + +`contract-run.sh` derives that from its own location (`SCRIPTS_DIR`) and checks +the script it is about to run against `origin/master` in the same clone. There +is no second path to configure, and no contract may be executed from a +hand-copied location. + +| Contract | Script | Pinned clone | +| --- | --- | --- | +| `infrastructure-monitoring` | `scripts/infra-monitoring.sh` | the clone containing `contract-run.sh` | +| `proxmox-monitor` | `scripts/proxmox-monitor.sh` | same | +| `zulip-health` | `scripts/zulip-monitor.sh` | same | +| `agent-health-check` | `scripts/agent-health-check.py` | same | +| `litellm-health` | `scripts/litellm-health-check.py` | same | +| `disk-gc-threat-response` | `scripts/disk-gc-scan.py` | same | +| `pm2-self-heal` | `scripts/pm2-self-heal.sh` | same | +| `search-stack-visibility` | `scripts/search-stack-check.py` | same | + +### The deployed runner + +The scheduler on **CT 100 (abiba)** runs contracts from +**`/opt/contract-runner`** via `/etc/cron.d/contract-runner`. That clone is the +pinned execution copy for every scheduled contract, and it must be kept current +with `master` by fast-forward. Its `origin` is a local path to the upstream +working copy, not a network remote. + +`daily-health-digest` is **not** in the table above because it has no contract +file and no mapping — it is dispatched by cron as +`fm-send.sh ops "run contract: daily-health-digest"` and was, until +2026-09-25, executed by hand from whichever clone the operator happened to be +in. Creating its contract file and pinning it to a clone is an open follow-up. + +## How the check works + +`scripts/revision-preflight.sh `: + +* resolves the **repo-relative** path of the executing script inside the clone; +* **fetches** the remote first, so a stale local ref cannot make a stale script + look current; +* compares the script's sha256 against `:`; +* **fails closed** — a path absent from the ref, an unresolvable ref, or a + failed fetch is a failure, never a warning. + +Exit `0` means verified match. Exit `1` means mismatch or unverifiable. + +## Modes in `contract-run.sh` + +| `CONTRACT_REVISION_PREFLIGHT` | Behaviour | +| --- | --- | +| unset / `enforce` (default) | withhold the verdict, alert, exit `2` | +| `warn` | log the mismatch and continue | +| `off` | skip the check entirely | + +`enforce` is the default deliberately: an unverifiable copy is indistinguishable +from a stale or hand-edited one, and a verdict from it is worse than no verdict. + +## Operating notes + +* A stale pinned clone will now make contracts **withhold** rather than report. + That is the intended failure. Recover by fast-forwarding the pinned clone: + `git -C /opt/contract-runner pull --ff-only`. +* When a contract legitimately changes, land it through the normal branch + PR + path and fast-forward the pinned clone. Do not copy files into it by hand. +* `--no-fetch` exists for offline inspection; it prints that freshness is + assumed rather than verified, and it is not used by `contract-run.sh`. diff --git a/scripts/contract-run.sh b/scripts/contract-run.sh index 04643aa..e09b418 100755 --- a/scripts/contract-run.sh +++ b/scripts/contract-run.sh @@ -18,6 +18,14 @@ # litellm-health -> scripts/litellm-health-check.py # disk-gc-threat-response -> scripts/disk-gc-scan.py # pm2-self-heal -> scripts/pm2-self-heal.sh +# search-stack-visibility -> scripts/search-stack-check.py +# +# Execution copy: every contract pins the clone this script lives in (see +# docs/contract-execution-pinning.md). Before a contract runs, this wrapper +# proves the script it is about to execute byte-matches origin/master: +# CONTRACT_REVISION_PREFLIGHT=enforce (default) refuse to report on mismatch +# CONTRACT_REVISION_PREFLIGHT=warn log the mismatch and continue +# CONTRACT_REVISION_PREFLIGHT=off skip the check entirely # # Exit codes: # 0 = contract passed @@ -111,6 +119,35 @@ echo "Started: $(date -u '+%Y-%m-%d %H:%M:%S UTC')" | tee -a "$LOG_FILE" echo "Script: $SCRIPT_PATH" | tee -a "$LOG_FILE" echo "" | tee -a "$LOG_FILE" +# ── Revision preflight ─────────────────────────────────────────────────────── +# A verdict is only meaningful if it came from the merged copy. Refuse to report +# one from a mismatched or unverifiable copy; that is a probe failure (exit 2), +# not a contract verdict, because the result would be untrustworthy. +# See docs/contract-execution-pinning.md. +REVISION_PREFLIGHT_MODE="${CONTRACT_REVISION_PREFLIGHT:-enforce}" +REPO_ROOT="$(cd "${SCRIPTS_DIR}/.." && pwd)" +if [ "$REVISION_PREFLIGHT_MODE" != "off" ] && [ -x "${SCRIPTS_DIR}/revision-preflight.sh" ]; then + if "${SCRIPTS_DIR}/revision-preflight.sh" "$SCRIPT_PATH" "$REPO_ROOT" 2>&1 | tee -a "$LOG_FILE"; then + : + elif [ "$REVISION_PREFLIGHT_MODE" = "warn" ]; then + echo "⚠️ revision preflight failed — continuing because CONTRACT_REVISION_PREFLIGHT=warn" | tee -a "$LOG_FILE" + else + echo "🚫 VERDICT WITHHELD: executing copy does not match the merged revision" | tee -a "$LOG_FILE" + ALERT_MSG="🔴 Contract $CONTRACT_NAME: revision mismatch — verdict withheld. Log: $LOG_FILE" + ZULIP_API_URL="${ZULIP_API_URL:-https://chat.sysloggh.net/api/v1}" + ZULIP_API_KEY="${ZULIP_API_KEY:-}" + ZULIP_USER="${ZULIP_USER:-abiba-bot@chat.sysloggh.net}" + if [ -n "$ZULIP_API_KEY" ] && command -v curl &> /dev/null; then + curl -sf -X POST "${ZULIP_API_URL}/messages" \ + -u "${ZULIP_USER}:${ZULIP_API_KEY}" \ + -d "type=private" \ + -d "to=9" \ + -d "content=${ALERT_MSG}" > /dev/null 2>&1 || true + fi + exit 2 + fi +fi + # Use timeout to prevent hangs (10 minutes default) TIMEOUT=600 timeout "$TIMEOUT" $INTERPRETER "$SCRIPT_PATH" 2>&1 | tee -a "$LOG_FILE" diff --git a/scripts/revision-preflight.sh b/scripts/revision-preflight.sh new file mode 100755 index 0000000..2086f6b --- /dev/null +++ b/scripts/revision-preflight.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# revision-preflight.sh — prove the copy a contract is about to execute is the +# copy that is merged. +# +# Usage: +# revision-preflight.sh [options] +# +# Options: +# --ref Ref to compare against (default: origin/master) +# --no-fetch Do not refresh the ref first (see FRESHNESS below) +# --quiet Print nothing on success +# -h, --help Show this help +# +# Exit codes: +# 0 the executing script byte-matches : +# 1 MISMATCH, or the revision could not be resolved (see FAIL CLOSED) +# +# FRESHNESS +# A guard is only as good as the ref it compares against. On 2026-09-25 a +# stale local origin/master made an ancestry check on this fleet report +# "unlanded work" for a branch that had in fact merged, and it would equally +# have passed a stale script as current. So by default this guard FETCHES the +# remote before comparing. With --no-fetch it compares against whatever the +# local ref points at and says so out loud; it never silently assumes +# freshness. +# +# FAIL CLOSED +# An unresolvable path or ref is a FAILURE, never a warning. "Cannot verify" +# is precisely the state a stale or hand-edited copy produces, so treating it +# as success would defeat the guard. The original draft of this script did +# exactly that: it resolved the master revision with +# `git show origin/master:$(basename "$SCRIPT")`, which drops the scripts/ +# prefix, queries the repo root, fails, and exited 0 — passing a script that +# exists in no revision at all. +# +# WHICH CLONE +# Pass the clone the contract is actually executing from. See +# docs/contract-execution-pinning.md for which clone each contract pins. + +set -euo pipefail + +REF="origin/master" +FETCH=1 +QUIET=0 + +usage() { + sed -n '2,45p' "$0" | sed 's/^# \{0,1\}//' +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --ref) + [[ $# -ge 2 ]] || { echo "revision-preflight: --ref needs a value" >&2; exit 1; } + REF="$2"; shift 2 ;; + --no-fetch) FETCH=0; shift ;; + --quiet) QUIET=1; shift ;; + -h|--help) usage; exit 0 ;; + --) shift; break ;; + -*) echo "revision-preflight: unknown option: $1" >&2; exit 1 ;; + *) break ;; + esac +done + +if [[ $# -lt 2 ]]; then + usage >&2 + exit 1 +fi + +SCRIPT="$1" +CLONE="$2" + +say() { [[ $QUIET -eq 1 ]] || echo "$@" >&2; } +fail() { echo "❌ revision-preflight: $*" >&2; exit 1; } + +# ── 1. inputs must exist ────────────────────────────────────────────────────── +[[ -f "$SCRIPT" ]] || fail "executing script not found: $SCRIPT" +[[ -d "$CLONE" ]] || fail "clone path is not a directory: $CLONE" +git -C "$CLONE" rev-parse --git-dir >/dev/null 2>&1 \ + || fail "not a git clone: $CLONE" + +# ── 2. resolve the repo-relative path (the original defect) ─────────────────── +CLONE_ABS=$(cd "$CLONE" && pwd) +SCRIPT_ABS=$(cd "$(dirname "$SCRIPT")" && pwd)/$(basename "$SCRIPT") +case "$SCRIPT_ABS" in + "$CLONE_ABS"/*) REL="${SCRIPT_ABS#"$CLONE_ABS"/}" ;; + *) fail "script is outside the clone: $SCRIPT_ABS is not under $CLONE_ABS" ;; +esac + +# ── 3. refresh the ref so staleness cannot mask a stale script ──────────────── +if [[ $FETCH -eq 1 ]]; then + REMOTE="${REF%%/*}" + [[ "$REMOTE" == "$REF" ]] && REMOTE="origin" + if ! git -C "$CLONE" fetch --quiet "$REMOTE" 2>/dev/null; then + fail "cannot fetch '$REMOTE' in $CLONE — refusing to verify against a possibly stale '$REF'. Re-run with network access, or pass --no-fetch to compare against the local ref deliberately." + fi +else + say "⚠️ revision-preflight: --no-fetch — comparing against the LOCAL '$REF'; freshness is assumed, not verified" +fi + +# ── 4. resolve the merged revision; unresolvable is a failure ──────────────── +git -C "$CLONE" rev-parse --verify --quiet "$REF" >/dev/null \ + || fail "ref '$REF' does not resolve in $CLONE" +REF_COMMIT=$(git -C "$CLONE" rev-parse --short "$REF") + +TMPFILE=$(mktemp) +trap 'rm -f "$TMPFILE"' EXIT + +if ! git -C "$CLONE" show "$REF:$REL" > "$TMPFILE" 2>/dev/null; then + fail "'$REL' does not exist in $REF ($REF_COMMIT) — cannot verify $SCRIPT. A path that is absent from $REF can never be a merged copy." +fi + +# ── 5. compare ─────────────────────────────────────────────────────────────── +EXEC_SHA=$(sha256sum "$SCRIPT" | cut -d' ' -f1) +MERGED_SHA=$(sha256sum "$TMPFILE" | cut -d' ' -f1) + +if [[ "$EXEC_SHA" != "$MERGED_SHA" ]]; then + { + echo "❌ revision-preflight: MISMATCH — refusing to report from this copy" + echo " script: $SCRIPT_ABS" + echo " clone: $CLONE_ABS" + echo " executed: $EXEC_SHA" + echo " merged: $MERGED_SHA ($REF:$REL @ $REF_COMMIT)" + } >&2 + exit 1 +fi + +say "✅ revision-preflight: $REL matches $REF @ $REF_COMMIT ($EXEC_SHA)" +exit 0 diff --git a/tests/fixtures/revision-preflight.prefix.sh b/tests/fixtures/revision-preflight.prefix.sh new file mode 100755 index 0000000..0c5765d --- /dev/null +++ b/tests/fixtures/revision-preflight.prefix.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Revision preflight guard: verify the script being executed matches origin/master +# Usage: revision-preflight.sh +# Returns 0 if match, 1 if mismatch (prints both revisions) + +set -euo pipefail + +SCRIPT="${1:?Usage: revision-preflight.sh }" +CLONE="${2:?Usage: revision-preflight.sh }" + +# Compute sha256 of the script being executed +EXEC_SHA=$(sha256sum "$SCRIPT" | cut -d' ' -f1) + +# Compute sha256 of the merged origin/master version +# Extract to a temp file to avoid pipe issues +TMPFILE=$(mktemp) +trap 'rm -f "$TMPFILE"' EXIT + +# Try to extract the file from origin/master +if git -C "$CLONE" show "origin/master:$(basename "$SCRIPT")" > "$TMPFILE" 2>/dev/null; then + MASTER_SHA=$(sha256sum "$TMPFILE" | cut -d' ' -f1) +else + echo "⚠️ revision-preflight: could not resolve origin/master revision for $(basename "$SCRIPT")" >&2 + exit 0 # Warn but don't block if git show fails +fi + +if [[ -z "$MASTER_SHA" || "$MASTER_SHA" == "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" ]]; then + echo "⚠️ revision-preflight: could not resolve origin/master revision for $(basename "$SCRIPT")" >&2 + exit 0 # Warn but don't block if git show fails +fi + +if [[ "$EXEC_SHA" != "$MASTER_SHA" ]]; then + echo "⚠️ revision-preflight: MISMATCH detected" >&2 + echo " Executed: $EXEC_SHA ($(basename "$SCRIPT"))" >&2 + echo " Merged: $MASTER_SHA (origin/master:$(basename "$SCRIPT"))" >&2 + exit 1 +else + echo "✅ revision-preflight: $SCRIPT matches origin/master ($EXEC_SHA)" >&2 + exit 0 +fi diff --git a/tests/test_revision_preflight.sh b/tests/test_revision_preflight.sh new file mode 100755 index 0000000..02bdefe --- /dev/null +++ b/tests/test_revision_preflight.sh @@ -0,0 +1,182 @@ +#!/usr/bin/env bash +# Behavioural tests for scripts/revision-preflight.sh +# +# Every case builds a throwaway clone with a real bare remote, so origin/master +# is genuine and the guard's fetch path is exercised. Nothing outside mktemp is +# touched. +# +# The pre-fix draft is kept at tests/fixtures/revision-preflight.prefix.sh and +# is run against the SAME cases, to prove these tests bite: the pre-fix guard +# exits 0 where the fixed guard exits 1. + +set -uo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +REPO="$(cd "$HERE/.." && pwd)" +GUARD="$REPO/scripts/revision-preflight.sh" +PREFIX_GUARD="$HERE/fixtures/revision-preflight.prefix.sh" + +PASS=0 +FAIL=0 +FAILED_CASES=() + +pass() { printf ' ✓ %s\n' "$1"; PASS=$((PASS + 1)); } +fail() { printf ' ✗ %s\n' "$1"; FAIL=$((FAIL + 1)); FAILED_CASES+=("$1"); } + +# Build a clone with a real remote; echo the clone path. +make_clone() { + local tmp + tmp="$(mktemp -d)" + git init --bare -q "$tmp/remote.git" + git init -q "$tmp/clone" + ( + cd "$tmp/clone" || exit 1 + git config user.email test@example.invalid + git config user.name test + mkdir -p scripts + printf '#!/bin/bash\necho hello\n' > scripts/demo.sh + chmod +x scripts/demo.sh + git add -A + git commit -qm init + git branch -M master + git remote add origin "$tmp/remote.git" + git push -q origin master + git fetch -q origin + ) + echo "$tmp/clone" +} + +echo "== revision-preflight behavioural tests ==" + +# ── 1. match → exit 0 ──────────────────────────────────────────────────────── +echo "1. matching copy" +C=$(make_clone) +if out=$("$GUARD" "$C/scripts/demo.sh" "$C" 2>&1); then + pass "matching copy exits 0" +else + fail "matching copy should exit 0 (got $?, output: $out)" +fi +if [[ -z "$( "$GUARD" --quiet "$C/scripts/demo.sh" "$C" 2>&1 )" ]]; then + pass "--quiet prints nothing on a match" +else + fail "--quiet should print nothing on a match" +fi +rm -rf "$(dirname "$C")" + +# ── 2. mismatch → exit 1 and names both hashes ─────────────────────────────── +echo "2. mismatched copy" +C=$(make_clone) +printf '#!/bin/bash\necho TAMPERED\n' > "$C/scripts/demo.sh" +out=$("$GUARD" "$C/scripts/demo.sh" "$C" 2>&1); rc=$? +if [[ $rc -eq 1 ]]; then pass "mismatch exits 1"; else fail "mismatch should exit 1 (got $rc)"; fi +if [[ "$out" == *"MISMATCH"* ]]; then pass "mismatch says MISMATCH"; else fail "mismatch should say MISMATCH"; fi +if [[ "$out" == *"executed:"* && "$out" == *"merged:"* ]]; then + pass "mismatch prints both revisions" +else + fail "mismatch should print both revisions" +fi +rm -rf "$(dirname "$C")" + +# ── 3. paths under scripts/ resolve (the original basename defect) ─────────── +echo "3. repo-relative path resolution" +C=$(make_clone) +if "$GUARD" --quiet "$C/scripts/demo.sh" "$C" >/dev/null 2>&1; then + pass "script under scripts/ resolves against origin/master" +else + fail "script under scripts/ must resolve (basename defect)" +fi +rm -rf "$(dirname "$C")" + +# ── 4. script that exists in NO revision → must fail ───────────────────────── +echo "4. untracked script present in no revision" +C=$(make_clone) +printf '#!/bin/bash\necho never committed\n' > "$C/scripts/ghost.sh" +out=$("$GUARD" "$C/scripts/ghost.sh" "$C" 2>&1); rc=$? +if [[ $rc -eq 1 ]]; then pass "ghost script exits 1"; else fail "ghost script must exit 1 (got $rc)"; fi +if [[ "$out" == *"does not exist in"* ]]; then + pass "ghost script says it is absent from the ref" +else + fail "ghost script should say it is absent from the ref" +fi +rm -rf "$(dirname "$C")" + +# ── 5. unresolvable ref → must fail ────────────────────────────────────────── +echo "5. unresolvable ref" +C=$(make_clone) +out=$("$GUARD" --no-fetch --ref origin/nope "$C/scripts/demo.sh" "$C" 2>&1); rc=$? +if [[ $rc -eq 1 ]]; then pass "unresolvable ref exits 1"; else fail "unresolvable ref must exit 1 (got $rc)"; fi +rm -rf "$(dirname "$C")" + +# ── 6. missing script → must fail ──────────────────────────────────────────── +echo "6. missing script" +C=$(make_clone) +out=$("$GUARD" "$C/scripts/nope.sh" "$C" 2>&1); rc=$? +if [[ $rc -eq 1 ]]; then pass "missing script exits 1"; else fail "missing script must exit 1 (got $rc)"; fi +rm -rf "$(dirname "$C")" + +# ── 7. script outside the clone → must fail ────────────────────────────────── +echo "7. script outside the clone" +C=$(make_clone) +OUTSIDE=$(mktemp) +printf '#!/bin/bash\necho outside\n' > "$OUTSIDE" +out=$("$GUARD" "$OUTSIDE" "$C" 2>&1); rc=$? +if [[ $rc -eq 1 ]]; then pass "outside script exits 1"; else fail "outside script must exit 1 (got $rc)"; fi +rm -f "$OUTSIDE"; rm -rf "$(dirname "$C")" + +# ── 8. --no-fetch states the freshness assumption ──────────────────────────── +echo "8. --no-fetch states its assumption" +C=$(make_clone) +out=$("$GUARD" --no-fetch "$C/scripts/demo.sh" "$C" 2>&1) +if [[ "$out" == *"freshness is assumed"* ]]; then + pass "--no-fetch states the freshness assumption" +else + fail "--no-fetch should state the freshness assumption" +fi +rm -rf "$(dirname "$C")" + +# ── 9. the pre-fix guard must FAIL these same cases (proves the tests bite) ── +echo "9. pre-fix draft fails the same cases (bite proof)" +if [[ ! -f "$PREFIX_GUARD" ]]; then + fail "pre-fix fixture missing: $PREFIX_GUARD" +else + # 9a. repo-relative path: pre-fix drops scripts/ and cannot resolve + C=$(make_clone) + out=$("$PREFIX_GUARD" "$C/scripts/demo.sh" "$C" 2>&1); rc=$? + if [[ $rc -eq 0 && "$out" == *"could not resolve"* ]]; then + pass "pre-fix: exits 0 and cannot resolve scripts/demo.sh (defect confirmed)" + else + fail "pre-fix should exit 0 with 'could not resolve' (got rc=$rc)" + fi + rm -rf "$(dirname "$C")" + + # 9b. ghost script: pre-fix passes a script that exists in no revision + C=$(make_clone) + printf '#!/bin/bash\necho never committed\n' > "$C/scripts/ghost.sh" + out=$("$PREFIX_GUARD" "$C/scripts/ghost.sh" "$C" 2>&1); rc=$? + if [[ $rc -eq 0 ]]; then + pass "pre-fix: PASSES a ghost script that exists in no revision (defect confirmed)" + else + fail "pre-fix was expected to wrongly pass the ghost script (got rc=$rc)" + fi + rm -rf "$(dirname "$C")" + + # 9c. a file that DOES exist at the repo root still works pre-fix, showing + # the defect is specific to nested paths + C=$(make_clone) + printf '#!/bin/bash\necho root\n' > "$C/rootlevel.sh" + ( cd "$C" && git add rootlevel.sh && git commit -qm root && git push -q origin master && git fetch -q origin ) + if "$PREFIX_GUARD" "$C/rootlevel.sh" "$C" >/dev/null 2>&1; then + pass "pre-fix: root-level path resolves (so the defect is the basename, not git)" + else + fail "pre-fix should resolve a root-level tracked file" + fi + rm -rf "$(dirname "$C")" +fi + +echo +echo " passed: $PASS failed: $FAIL" +if [[ $FAIL -gt 0 ]]; then + printf ' FAILED: %s\n' "${FAILED_CASES[@]}" + exit 1 +fi +echo "All revision-preflight tests passed."