SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
778424acd4481bee73d6f18c52cf65a8e9dfb5b0
prose-contracts/scripts/contract-run.sh
T
root c5a6dcd42a
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 10s
Details
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 12s
Details
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Failing after 6s
Details
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
Details
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Details
fix(revision-preflight): default to warn, name the refusal class, bound the fetch 
Bounded correction round on PR #134 after a PASS-WITH-FINDINGS review whose
Finding 4 is High. The guard's purpose and its fail-closed fix stand; the
problem was that with 'enforce' as the default it gates EVERY scheduled
contract, and three legitimate states produce a refusal - a clone legitimately
ahead of origin/master mid-review, a detached HEAD, and an offline or failed
fetch - so any of them would turn the fleet's monitoring into withheld
verdicts. That risk outweighs the staleness the guard catches.

1. DEFAULT IS NOW 'warn'. 'enforce' remains available and documented. The
   criteria for flipping the default later are written into the doc as a
   decision with evidence - a sustained window (30 days / 200+ runs) with zero
   mismatch:* and zero cannot-verify:* refusals, no fetch blips, and a pinned
   clone demonstrably kept current - explicitly as its own change, not a silent
   flip.

2. 'COULD NOT CHECK' IS NOW DISTINGUISHABLE FROM 'THIS COPY IS WRONG'. Every
   non-zero exit prints a machine-readable REASON=<class> line:
     cannot-verify:fetch-failed | cannot-verify:ref-unresolvable   (exit 2)
     mismatch:path-absent | mismatch:content
     mismatch:detached-head | mismatch:clone-ahead                (exit 1)
   detached-head and clone-ahead are named separately because they are
   legitimate states, far less alarming than a hand-edited file. clone-ahead
   requires HEAD to be STRICTLY ahead; an uncommitted edit on a commit that IS
   the ref is a plain content mismatch (my own first cut got this wrong and the
   new test 7d caught it).

3. THE DEFAULT FETCH IS BOUNDED: --fetch-timeout, default 20s, 0 = unbounded,
   and a missing 'timeout' binary is itself a cannot-verify rather than an
   unbounded fetch inside a scheduled contract.

4. TEST COVERAGE ADDED for every new class: fetch failure, fetch timeout
   (asserted to return promptly under a 1s bound), unresolvable ref, detached
   HEAD, clone-ahead, genuine content mismatch, and the contract-run.sh default.
   The pre-fix draft fixture comparisons are kept: 31 passed, 0 failed.

5. MERGE-TIME SEQUENCE documented: fast-forward /opt/contract-runner, confirm
   clean, prove a contract runs and reports. Baseline recorded as of today -
   firstmate has already fast-forwarded it to 9faffe4 - with the note that an
   untracked file blocks a fast-forward even when byte-identical.

Live behaviour re-verified on the real runner path:
  default: REASON=mismatch:clone-ahead -> 'continuing because ...=warn' -> VERDICT: PASS, exit 0
  enforce: REASON=mismatch:clone-ahead -> 'VERDICT WITHHELD: mismatch:clone-ahead', exit 2

MANDATORY CHECKS (master went red once from a credential-SHAPED string, so
these are now run on every shippable branch):
  bash scripts/prose-lint.sh        -> LINT PASSED (18 warning(s))
  secret scan                       -> secret scan clean (tree; 34 allowlisted,
                                       24 inert value(s) ignored); No committed credentials
  shellcheck revision-preflight.sh  -> clean
  shellcheck test_revision_preflight.sh -> clean
  shellcheck contract-run.sh        -> SC2034 x1, SC2086 x2 - byte-identical on
                                       master, i.e. pre-existing, none introduced

tests/test_probe_drift.py::test_prose_lint_accepts_report_format_with_provenance
fails both before and after this branch (it runs prose-lint from a temp CWD and
cannot find its sibling secret-scan.sh). Pre-existing, unrelated, not fixed here.
2026-09-25 11:29:15 +00:00

221 lines
9.1 KiB
Bash
Executable File
Raw Blame History

#!/bin/bash
# contract-run.sh — Deterministic contract execution from machine scheduler
#
# Takes a contract name, resolves its script, runs it with timeout,
# logs output to $CONTRACT_RUN_LOG_DIR (default: /var/log/contract-runs/),
# and alerts on failure.
#
# Environment:
# CONTRACT_RUN_LOG_DIR Override the log directory (default: /var/log/contract-runs)
#
# Usage: bash scripts/contract-run.sh <contract-name>
#
# Contract names map to scripts as follows:
# infrastructure-monitoring -> scripts/infra-monitoring.sh
# proxmox-monitor -> scripts/proxmox-monitor.sh
# zulip-health -> scripts/zulip-monitor.sh
# agent-health-check -> scripts/agent-health-check.py
# litellm-health -> scripts/litellm-health-check.py
# disk-gc-threat-response -> scripts/disk-gc-scan.py
# pm2-self-heal -> scripts/pm2-self-heal.sh
# search-stack-visibility -> scripts/search-stack-check.py
#
# Execution copy: every contract pins the clone this script lives in (see
# docs/contract-execution-pinning.md). Before a contract runs, this wrapper
# proves the script it is about to execute byte-matches origin/master:
# CONTRACT_REVISION_PREFLIGHT=warn (default) log a refusal, still report
# CONTRACT_REVISION_PREFLIGHT=enforce refuse to report on a mismatch
# CONTRACT_REVISION_PREFLIGHT=off skip the check entirely
# A refusal names its class: cannot-verify:fetch-failed|ref-unresolvable,
# or mismatch:content|path-absent|detached-head|clone-ahead.
#
# Exit codes:
# 0 = contract passed
# 1 = contract failed (alert sent)
# 2 = probe failed (script missing, timeout, etc.)
set -uo pipefail
CONTRACT_NAME="$1"
SCRIPTS_DIR="$(cd "$(dirname "$0")" && pwd)"
LOG_DIR="${CONTRACT_RUN_LOG_DIR:-/var/log/contract-runs}"
TIMESTAMP=$(date -u '+%Y%m%d-%H%M%S')
LOG_FILE="${LOG_DIR}/${CONTRACT_NAME}-${TIMESTAMP}.log"
# Ensure log directory exists
mkdir -p "$LOG_DIR"
# Map contract name to script path
case "$CONTRACT_NAME" in
infrastructure-monitoring)
SCRIPT_PATH="${SCRIPTS_DIR}/infra-monitoring.sh"
INTERPRETER="bash"
;;
proxmox-monitor)
SCRIPT_PATH="${SCRIPTS_DIR}/proxmox-monitor.sh"
INTERPRETER="bash"
;;
zulip-health)
SCRIPT_PATH="${SCRIPTS_DIR}/zulip-monitor.sh"
INTERPRETER="bash"
;;
agent-health-check)
SCRIPT_PATH="${SCRIPTS_DIR}/agent-health-check.py"
INTERPRETER="python3"
;;
litellm-health)
SCRIPT_PATH="${SCRIPTS_DIR}/litellm-health-check.py"
INTERPRETER="python3"
;;
pm2-self-heal)
SCRIPT_PATH="${SCRIPTS_DIR}/pm2-self-heal.sh"
INTERPRETER="bash"
;;
disk-gc-threat-response)
SCRIPT_PATH="${SCRIPTS_DIR}/disk-gc-scan.py"
INTERPRETER="python3"
;;
search-stack-visibility)
SCRIPT_PATH="${SCRIPTS_DIR}/search-stack-check.py"
INTERPRETER="python3"
;;
*)
echo "Unknown contract: $CONTRACT_NAME" | tee -a "$LOG_FILE"
# Send alert for unknown contract
ALERT_MSG="🔴 Contract $CONTRACT_NAME: unknown contract name. Log: $LOG_FILE"
ZULIP_API_URL="${ZULIP_API_URL:-https://chat.sysloggh.net/api/v1}"
ZULIP_API_KEY="${ZULIP_API_KEY:-}"
ZULIP_USER="${ZULIP_USER:-abiba-bot@chat.sysloggh.net}"
if [ -n "$ZULIP_API_KEY" ] && command -v curl &> /dev/null; then
curl -sf -X POST "${ZULIP_API_URL}/messages" \
-u "${ZULIP_USER}:${ZULIP_API_KEY}" \
-d "type=private" \
-d "to=9" \
-d "content=${ALERT_MSG}" > /dev/null 2>&1 || true
fi
exit 2
;;
esac
# Check if script exists
if [ ! -f "$SCRIPT_PATH" ]; then
echo "Script not found: $SCRIPT_PATH" | tee -a "$LOG_FILE"
# Send alert for missing script
ALERT_MSG="🔴 Contract $CONTRACT_NAME: script not found at $SCRIPT_PATH. Log: $LOG_FILE"
ZULIP_API_URL="${ZULIP_API_URL:-https://chat.sysloggh.net/api/v1}"
ZULIP_API_KEY="${ZULIP_API_KEY:-}"
ZULIP_USER="${ZULIP_USER:-abiba-bot@chat.sysloggh.net}"
if [ -n "$ZULIP_API_KEY" ] && command -v curl &> /dev/null; then
curl -sf -X POST "${ZULIP_API_URL}/messages" \
-u "${ZULIP_USER}:${ZULIP_API_KEY}" \
-d "type=private" \
-d "to=9" \
-d "content=${ALERT_MSG}" > /dev/null 2>&1 || true
fi
exit 2
fi
# Run the script with timeout and capture output
echo "=== Contract: $CONTRACT_NAME ===" | tee "$LOG_FILE"
echo "Started: $(date -u '+%Y-%m-%d %H:%M:%S UTC')" | tee -a "$LOG_FILE"
echo "Script: $SCRIPT_PATH" | tee -a "$LOG_FILE"
echo "" | tee -a "$LOG_FILE"
# ── Revision preflight ───────────────────────────────────────────────────────
# A verdict is only meaningful if it came from the merged copy. This reports
# whether the executing copy matches, and distinguishes "could not check" from
# "this copy is wrong" so an operator can tell them apart.
# See docs/contract-execution-pinning.md.
#
# Default is WARN, not enforce: the guard gates every scheduled contract, and a
# legitimate state (branch mid-review, detached HEAD, briefly offline) would
# otherwise turn the whole fleet's monitoring into withheld verdicts. The
# criteria for flipping the default to enforce are written down in the doc.
REVISION_PREFLIGHT_MODE="${CONTRACT_REVISION_PREFLIGHT:-warn}"
REPO_ROOT="$(cd "${SCRIPTS_DIR}/.." && pwd)"
if [ "$REVISION_PREFLIGHT_MODE" != "off" ] && [ -x "${SCRIPTS_DIR}/revision-preflight.sh" ]; then
PREFLIGHT_OUT="$(mktemp)"
if "${SCRIPTS_DIR}/revision-preflight.sh" "$SCRIPT_PATH" "$REPO_ROOT" >"$PREFLIGHT_OUT" 2>&1; then
cat "$PREFLIGHT_OUT" | tee -a "$LOG_FILE"
else
cat "$PREFLIGHT_OUT" | tee -a "$LOG_FILE"
PREFLIGHT_REASON="$(grep -m1 '^REASON=' "$PREFLIGHT_OUT" | cut -d= -f2-)"
[ -n "$PREFLIGHT_REASON" ] || PREFLIGHT_REASON="unclassified"
if [ "$REVISION_PREFLIGHT_MODE" = "enforce" ]; then
echo "🚫 VERDICT WITHHELD: $PREFLIGHT_REASON" | tee -a "$LOG_FILE"
ALERT_MSG="🔴 Contract $CONTRACT_NAME: revision preflight REFUSED ($PREFLIGHT_REASON) — verdict withheld. Log: $LOG_FILE"
ZULIP_API_URL="${ZULIP_API_URL:-https://chat.sysloggh.net/api/v1}"
ZULIP_API_KEY="${ZULIP_API_KEY:-}"
ZULIP_USER="${ZULIP_USER:-abiba-bot@chat.sysloggh.net}"
if [ -n "$ZULIP_API_KEY" ] && command -v curl &> /dev/null; then
curl -sf -X POST "${ZULIP_API_URL}/messages" \
-u "${ZULIP_USER}:${ZULIP_API_KEY}" \
-d "type=private" \
-d "to=9" \
-d "content=${ALERT_MSG}" > /dev/null 2>&1 || true
fi
rm -f "$PREFLIGHT_OUT"
exit 2
fi
echo "⚠️ revision preflight: $PREFLIGHT_REASON — continuing because CONTRACT_REVISION_PREFLIGHT=$REVISION_PREFLIGHT_MODE" | tee -a "$LOG_FILE"
fi
rm -f "$PREFLIGHT_OUT"
fi
# Use timeout to prevent hangs (10 minutes default)
TIMEOUT=600
timeout "$TIMEOUT" $INTERPRETER "$SCRIPT_PATH" 2>&1 | tee -a "$LOG_FILE"
EXIT_CODE=${PIPESTATUS[0]}
# If timeout killed the process, EXIT_CODE will be 124
if [ $EXIT_CODE -eq 124 ]; then
echo "⏰ TIMEOUT: script exceeded ${TIMEOUT}s limit" | tee -a "$LOG_FILE"
fi
echo "" | tee -a "$LOG_FILE"
if [ $EXIT_CODE -eq 0 ]; then
echo "✅ VERDICT: PASS" | tee -a "$LOG_FILE"
exit 0
else
echo "🔴 VERDICT: FAIL (exit code $EXIT_CODE)" | tee -a "$LOG_FILE"
# Send alert (Zulip DM to user 9 + stream agent-hub topic alerts-infra)
# Using the same alert path as other monitors
ALERT_MSG="🔴 Contract $CONTRACT_NAME failed (exit $EXIT_CODE). Log: $LOG_FILE"
ALERT_SENT=false
# Take credentials from environment (ZULIP_API_KEY required)
ZULIP_API_URL="${ZULIP_API_URL:-https://chat.sysloggh.net/api/v1}"
ZULIP_API_KEY="${ZULIP_API_KEY:-}"
ZULIP_USER="${ZULIP_USER:-abiba-bot@chat.sysloggh.net}"
if [ -n "$ZULIP_API_KEY" ] && command -v curl &> /dev/null; then
# DM to user 9
DM_EXIT=0
curl -sf -X POST "${ZULIP_API_URL}/messages" \
-u "${ZULIP_USER}:${ZULIP_API_KEY}" \
-d "type=private" \
-d "to=9" \
-d "content=${ALERT_MSG}" > /dev/null 2>&1 || DM_EXIT=$?
# Stream agent-hub topic alerts-infra
STREAM_EXIT=0
curl -sf -X POST "${ZULIP_API_URL}/messages" \
-u "${ZULIP_USER}:${ZULIP_API_KEY}" \
-d "type=stream" \
-d "to=agent-hub" \
-d "topic=alerts-infra" \
-d "content=${ALERT_MSG}" > /dev/null 2>&1 || STREAM_EXIT=$?
if [ $DM_EXIT -eq 0 ] || [ $STREAM_EXIT -eq 0 ]; then
ALERT_SENT=true
else
echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') ALERT FAILURE: DM exit=$DM_EXIT, stream exit=$STREAM_EXIT" >> "$LOG_FILE"
fi
else
echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') ALERT SKIPPED: no ZULIP_API_KEY or curl" >> "$LOG_FILE"
fi
exit 1
fi
Reference in New Issue View Git Blame Copy Permalink