agent-health-check.prose.md

@@ -38,8 +38,7 @@ Runs every 4 hours (2, 6, 10, 14, 18, 22 UTC at :35) via cron (`35 2,6,10,14,18,
 - ct_liveness: map of CT → active status
 - config_integrity: map of config file → valid/invalid
 
-(## Execution
-)
+## Execution
 **Execution model**: This contract is executed by a host-scheduled cron job (see
 `scripts/contract-run.sh`). The cron job runs the monitoring script directly on the
 target host and appends the result to `/var/log/contract-runs/<contract>.log`. An

disk-gc-threat-response.prose.md

@@ -159,8 +159,7 @@ from the `report_only_guests` YAML block above.
 - `timeout`: 300 seconds per CT (GC may take time on large docker hosts)
 - `retry`: 2 attempts for SSH failures before marking a CT unreachable
 
-(## Execution
-)
+## Execution
 **Execution model**: This contract is executed by a host-scheduled cron job (see
 `scripts/contract-run.sh`). The cron job runs the monitoring script directly on the
 target host and appends the result to `/var/log/contract-runs/<contract>.log`. An

infrastructure-monitoring.prose.md

@@ -101,8 +101,7 @@ GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
 - LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
 - Stack persists across reboots (systemd for exporters, Docker restart policy)
 
-(## Execution
-)
+## Execution
 **Execution model**: This contract is executed by a host-scheduled cron job (see
 `scripts/contract-run.sh`). The cron job runs the monitoring script directly on the
 target host and appends the result to `/var/log/contract-runs/<contract>.log`. An

litellm-health.prose.md

@@ -110,8 +110,7 @@ contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu-
 | harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
 | harness-prometheus | prom/prometheus | :9090 | /-/healthy |
 
-(## Execution
-)
+## Execution
 **Execution model**: This contract is executed by a host-scheduled cron job (see
 `scripts/contract-run.sh`). The cron job runs the monitoring script directly on the
 target host and appends the result to `/var/log/contract-runs/<contract>.log`. An

pm2-self-heal.prose.md

@@ -50,8 +50,7 @@ description: >
   This was the root cause of the 18-restart accumulation. Threshold raised
   and PM2 counter reset on 2026-06-28.
 
-(## Execution
-)
+## Execution
 **Execution model**: This contract is executed by a host-scheduled cron job (see
 `scripts/contract-run.sh`). The cron job runs the monitoring script directly on the
 target host and appends the result to `/var/log/contract-runs/<contract>.log`. An

scripts/contract-run.sh

@@ -2,7 +2,11 @@
 # contract-run.sh — Deterministic contract execution from machine scheduler
 #
 # Takes a contract name, resolves its script, runs it with timeout,
-# logs output to /var/log/contract-runs/, and alerts on failure.
+# logs output to $CONTRACT_RUN_LOG_DIR (default: /var/log/contract-runs/),
+# and alerts on failure.
+#
+# Environment:
+#   CONTRACT_RUN_LOG_DIR  Override the log directory (default: /var/log/contract-runs)
 #
 # Usage: bash scripts/contract-run.sh <contract-name>
 #
@@ -24,7 +28,7 @@ set -uo pipefail
 
 CONTRACT_NAME="$1"
 SCRIPTS_DIR="$(cd "$(dirname "$0")" && pwd)"
-LOG_DIR="/var/log/contract-runs"
+LOG_DIR="${CONTRACT_RUN_LOG_DIR:-/var/log/contract-runs}"
 TIMESTAMP=$(date -u '+%Y%m%d-%H%M%S')
 LOG_FILE="${LOG_DIR}/${CONTRACT_NAME}-${TIMESTAMP}.log"
 
@@ -90,30 +94,38 @@ else
     # Send alert (Zulip DM to user 9 + stream agent-hub topic alerts-infra)
     # Using the same alert path as other monitors
     ALERT_MSG="🔴 Contract $CONTRACT_NAME failed (exit $EXIT_CODE). Log: $LOG_FILE"
+    ALERT_SENT=false
     
-    # Try to send via the existing alert mechanism
-    if command -v curl &> /dev/null; then
-        # Zulip DM to user 9
-        ZULIP_API_URL="http://192.168.68.117/api/v1"
-        ZULIP_API_KEY=$(cat /root/.config/zulip-api-key 2>/dev/null || echo "")
+    # Take credentials from environment (ZULIP_API_KEY required)
+    ZULIP_API_URL="${ZULIP_API_URL:-https://chat.sysloggh.net/api/v1}"
+    ZULIP_API_KEY="${ZULIP_API_KEY:-}"
+    ZULIP_USER="${ZULIP_USER:-abiba-bot@chat.sysloggh.net}"
     
-        if [ -n "$ZULIP_API_KEY" ]; then
+    if [ -n "$ZULIP_API_KEY" ] && command -v curl &> /dev/null; then
+        # DM to user 9
+        DM_EXIT=0
         curl -s -X POST "${ZULIP_API_URL}/messages" \
-                -u "user:${ZULIP_API_KEY}" \
+            -u "${ZULIP_USER}:${ZULIP_API_KEY}" \
             -d "type=private" \
             -d "to=9" \
-                -d "content=${ALERT_MSG}" > /dev/null 2>&1
-        fi
+            -d "content=${ALERT_MSG}" > /dev/null 2>&1 || DM_EXIT=$?
         
         # Stream agent-hub topic alerts-infra
-        if [ -n "$ZULIP_API_KEY" ]; then
+        STREAM_EXIT=0
         curl -s -X POST "${ZULIP_API_URL}/messages" \
-                -u "user:${ZULIP_API_KEY}" \
+            -u "${ZULIP_USER}:${ZULIP_API_KEY}" \
             -d "type=stream" \
             -d "to=agent-hub" \
             -d "topic=alerts-infra" \
-                -d "content=${ALERT_MSG}" > /dev/null 2>&1
+            -d "content=${ALERT_MSG}" > /dev/null 2>&1 || STREAM_EXIT=$?
+        
+        if [ $DM_EXIT -eq 0 ] || [ $STREAM_EXIT -eq 0 ]; then
+            ALERT_SENT=true
+        else
+            echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') ALERT FAILURE: DM exit=$DM_EXIT, stream exit=$STREAM_EXIT" >> "$LOG_FILE"
         fi
+    else
+        echo "$(date -u '+%Y-%m-%dT%H:%M:%SZ') ALERT SKIPPED: no ZULIP_API_KEY or curl" >> "$LOG_FILE"
     fi
     
     exit 1

zulip-health.prose.md

@@ -125,8 +125,7 @@ grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
 - **On `zulip-status` command**: Run on-demand and report to user
 - **On critical alert**: Escalate to relay message immediately, don't wait for schedule
 
-(## Execution
-)
+## Execution
 **Execution model**: This contract is executed by a host-scheduled cron job (see
 `scripts/contract-run.sh`). The cron job runs the monitoring script directly on the
 target host and appends the result to `/var/log/contract-runs/<contract>.log`. An