Fix PBS GC monitor false positive and add contract-run.sh wrapper #131

Merged

abiba-bot merged 6 commits from fix/contract-run-pbs-gc-20260924 into master 2026-09-24 06:07:10 +00:00

Conversation 0 Commits 6 Files Changed 10

+440 -17

abiba-bot commented 2026-09-24 04:10:18 +00:00

Owner

Copy Link

Copy Source

Part of Task: contract-execution-host-scheduler-20260924

Part of Task: contract-execution-host-scheduler-20260924

abiba-bot added 1 commit 2026-09-24 04:10:19 +00:00

Add contract-run.sh for machine scheduler execution and fix PBS GC leg ... c07aa5e382

- Add scripts/contract-run.sh: resolves contract name to script, runs with timeout,
  logs to /var/log/contract-runs/, alerts on failure via Zulip
- Add tests/test_contract_run.sh: proves passing and failing contract behavior
- Fix PBS GC leg in proxmox-monitor.sh: simplify logic to check last-run-endtime,
  use absolute paths for pct and proxmox-backup-manager to avoid PATH issues

Part of Task: contract-execution-host-scheduler-20260924

abiba-bot added 1 commit 2026-09-24 04:53:55 +00:00

feat: implement PBS GC four-state logic and update contracts ... c666d3e15c

- Implement four-state PBS GC logic in proxmox-monitor.sh:
  - probe-failed: unparseable JSON, store not found, or empty body → FAIL
  - running: collection in progress (last-run-endtime absent, upid present) → DO NOT FAIL
  - stale: no completed run within 48h → FAIL, naming last completed run age
  - healthy: completed within 48h → PASS, naming endtime and pending bytes

- Add tests/test_pbs_gc_states.sh covering all four states
  - Proves the test bites on the pre-fix version (5/6 tests fail)
  - All 6 tests pass against the fixed version

- Update contract-run.sh to map disk-gc-threat-response -> scripts/disk-gc-scan.py

- Update Execution sections of host-scheduled contracts:
  infrastructure-monitoring, zulip-health, litellm-health, agent-health-check,
  disk-gc-threat-response, pm2-self-heal
  Adding note that execution is host-scheduled via cron, not agent session ack.

abiba-bot added 1 commit 2026-09-24 05:12:48 +00:00

fix: make test_pbs_gc_states.sh self-contained with inline SSH replacement f16a890d0e

abiba-bot added 1 commit 2026-09-24 05:17:53 +00:00

fix: correct execution headings, variableize LOG_DIR, fix dead alert path ... 748ea389be

1. Replace '(## Execution' + ')' with '## Execution' in 6 contract files
2. Make LOG_DIR honor CONTRACT_RUN_LOG_DIR env var (default: /var/log/contract-runs)
3. Fix Zulip alert: correct URL (https://chat.sysloggh.net/api/v1), user
   (abiba-bot@chat.sysloggh.net), take ZULIP_API_KEY from environment, and
   write alert failures to run log

abiba-bot added 1 commit 2026-09-24 05:27:34 +00:00

fix: add -f to curl to fail on HTTP >= 400 ... fda6c844ff

Without -f, a rejected credential (HTTP 401) returns curl exit 0, making
a failed alert indistinguishable from a successful one. With -f, curl
exits non-zero on HTTP >= 400, so DM_EXIT and STREAM_EXIT correctly
capture the transmission failure and the run log records it.

abiba-bot added 1 commit 2026-09-24 06:01:05 +00:00

fix: three wrapper holes in contract-run.sh ... c0d04a2c02

1. Add disk-gc-threat-response to case statement (was only in header comment,
   hit *) branch and exited 2 silently). Mapped to scripts/disk-gc-scan.py per
   contract's Execution section.

2. Apply timeout to script invocation (was defined as TIMEOUT=600 but never used,
   so a hung check blocked the cron slot forever). Now wrapped with timeout, and
   exit 124 (timeout kill) logs a TIMEOUT line before the FAIL verdict.

3. Send alert on exit-2 paths (unknown contract and missing script). Both paths
   previously just echoed and exited, so a typo'd name or absent script was a
   silent monitoring loss. Now they send the same Zulip DM as a failed check.

Proved all four paths with raw output:
- unknown contract: curl -sf attempted, exit 22 on HTTP 401
- missing script: curl -sf attempted, exit 2
- disk-gc-threat-response: resolves to disk-gc-scan.py, runs, PASS
- stub sleep > timeout: TIMEOUT line logged, exit 1, alert failure recorded

abiba-bot merged commit 483a66b7b4 into master 2026-09-24 06:07:10 +00:00

abiba-bot deleted branch fix/contract-run-pbs-gc-20260924 2026-09-24 06:07:15 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

abiba-bot jerome (Jerome Kwame Tabiri) kagentz-bot kwame (Kwame Tabiri) mumuni-bot (Mumuni 🦅 (Syslog Falcon)) tanko-bot theo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#131