SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implement PBS GC four-state logic and update contracts
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
Details
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
Details
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
Details
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
Details
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
Details

Browse Source 
- Implement four-state PBS GC logic in proxmox-monitor.sh:
  - probe-failed: unparseable JSON, store not found, or empty body → FAIL
  - running: collection in progress (last-run-endtime absent, upid present) → DO NOT FAIL
  - stale: no completed run within 48h → FAIL, naming last completed run age
  - healthy: completed within 48h → PASS, naming endtime and pending bytes

- Add tests/test_pbs_gc_states.sh covering all four states
  - Proves the test bites on the pre-fix version (5/6 tests fail)
  - All 6 tests pass against the fixed version

- Update contract-run.sh to map disk-gc-threat-response -> scripts/disk-gc-scan.py

- Update Execution sections of host-scheduled contracts:
  infrastructure-monitoring, zulip-health, litellm-health, agent-health-check,
  disk-gc-threat-response, pm2-self-heal
  Adding note that execution is host-scheduled via cron, not agent session ack.
This commit is contained in:
root
2026-09-24 04:53:46 +00:00
parent c07aa5e382
commit c666d3e15c
9 changed files with 179 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/contract-run.sh
+1 -1
View File
@@ -12,7 +12,7 @@
# zulip-health -> scripts/zulip-monitor.sh
# agent-health-check -> scripts/agent-health-check.py
# litellm-health -> scripts/litellm-health-check.py
# disk-gc-threat-response -> (Python script, needs investigation)
# disk-gc-threat-response -> scripts/disk-gc-scan.py
# pm2-self-heal -> scripts/pm2-self-heal.sh
#
# Exit codes:
scripts/proxmox-monitor.sh
+43 -17
View File
@@ -79,8 +79,11 @@ if [ "$PBS_GC_OUTPUT" = "000" ]; then
echo " 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (expected JSON, got 000)"
FAILED+=("pbs-gc")
else
# Parse the JSON to get storepve-datastore's state
# last-run-endtime: epoch timestamp of last completion (0 when not run or in progress)
# Parse the JSON to get storepve-datastore's state with four distinct outcomes:
# 1. probe-failed: non-zero ssh status / empty / unparseable JSON
# 2. running: collection in progress (last-run-endtime absent or 0, but upid present)
# 3. stale: no completed run within 48h
# 4. healthy: completed within 48h
PBS_GC_RESULT=$(echo "$PBS_GC_OUTPUT" | python3 -c "
import sys, json
try:
@@ -88,41 +91,64 @@ try:
for store in data:
if store['store'] == 'storepve-datastore':
endtime = store.get('last-run-endtime')
upid = store.get('upid')
pending = store.get('pending-bytes', 0)
# State 2: Running (collection in progress) — last-run-endtime absent while run is in progress
if (endtime is None or endtime == 0) and upid is not None:
print(f'running|{pending}')
break
# State 3: No completed run (never-run or stale)
if endtime is None or endtime == 0:
print('never-run')
else:
print(f'{endtime}|{pending}')
print(f'no-completed-run|{pending}')
break
# States 3 & 4: Completed (has endtime)
print(f'completed|{endtime}|{pending}')
break
else:
print('absent')
print(f'absent|0')
except json.JSONDecodeError:
print('unparseable')
print(f'unparseable|0')
" 2>/dev/null)
if [ -z "$PBS_GC_RESULT" ] || [ "$PBS_GC_RESULT" = "unparseable" ]; then
# Parse the state|endtime|pending format
PBS_GC_STATE=$(echo "$PBS_GC_RESULT" | cut -d'|' -f1)
if [ "$PBS_GC_STATE" = "unparseable" ]; then
# State 1: probe-failed (unparseable JSON)
echo " 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (unparseable JSON)"
FAILED+=("pbs-gc")
elif [ "$PBS_GC_RESULT" = "absent" ]; then
echo " 🔴 PBS GC: never-run (storepve-datastore not found in GC list)"
elif [ "$PBS_GC_STATE" = "absent" ]; then
# State 1: probe-failed (store not found)
echo " 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (storepve-datastore not found)"
FAILED+=("pbs-gc")
elif [ "$PBS_GC_RESULT" = "never-run" ]; then
echo " 🔴 PBS GC: never-run (storepve-datastore has no last-run-endtime)"
elif [ "$PBS_GC_STATE" = "running" ]; then
# State 2: collection in progress — do NOT fail
PENDING_BYTES=$(echo "$PBS_GC_RESULT" | cut -d'|' -f2)
echo " ⏳ PBS GC: running (started: in-progress, pending-bytes: ${PENDING_BYTES} B)"
elif [ "$PBS_GC_STATE" = "no-completed-run" ]; then
# State 3: no completed run within 48h
PENDING_BYTES=$(echo "$PBS_GC_RESULT" | cut -d'|' -f2)
echo " 🔴 PBS GC: no completed run within 48h (pending-bytes: ${PENDING_BYTES} B)"
FAILED+=("pbs-gc")
else
# Parse the endtime|pending format
LAST_RUN_ENDTIME=$(echo "$PBS_GC_RESULT" | cut -d'|' -f1)
PENDING_BYTES=$(echo "$PBS_GC_RESULT" | cut -d'|' -f2)
# States 3 & 4: completed (has endtime)
LAST_RUN_ENDTIME=$(echo "$PBS_GC_RESULT" | cut -d'|' -f2)
PENDING_BYTES=$(echo "$PBS_GC_RESULT" | cut -d'|' -f3)
# Convert epoch to age in hours
NOW_EPOCH=$(date -u +%s)
AGE_HOURS=$(( (NOW_EPOCH - LAST_RUN_ENDTIME) / 3600 ))
if [ $AGE_HOURS -gt 48 ]; then
echo " 🔴 PBS GC: stale (last run ${AGE_HOURS}h ago, pending-bytes: ${PENDING_BYTES} B)"
# State 3: stale (no completed run within 48h)
echo " 🔴 PBS GC: stale — last completed run was ${AGE_HOURS}h ago (pending-bytes: ${PENDING_BYTES} B)"
FAILED+=("pbs-gc")
else
echo " ✅ PBS GC: healthy (last run ${AGE_HOURS}h ago, pending-bytes: ${PENDING_BYTES} B)"
# State 4: healthy (completed within 48h)
echo " ✅ PBS GC: healthy — last completed run ${AGE_HOURS}h ago (pending-bytes: ${PENDING_BYTES} B)"
fi
fi
fi