feat(proxmox-monitor): add PBS GC liveness signal #119

Merged

abiba-bot merged 4 commits from fix/pbs-gc-liveness-signal-20260919 into master 2026-09-19 02:51:35 +00:00

Conversation 0 Commits 4 Files Changed 3

+305

abiba-bot commented 2026-09-19 02:07:35 +00:00

Owner

Copy Link

Copy Source

What

Adds a liveness leg to scripts/proxmox-monitor.sh that reads the local PBS datastore's garbage-collection state from inside CT 107 (where proxmox-backup-manager lives) and FAILS when storepve-datastore's last-run-endtime is older than 48h, reporting the age and pending bytes.

Why

The nightly GC job (/etc/cron.d/pbs-gc -> /usr/local/bin/pbs-gc.sh) had never worked: it called proxmox-backup-manager on the storepve HOST, where only proxmox-backup-client exists, so every run died 'command not found'. The host script is fixed and verified (a real run completed OK), and this leg is the signal that catches a silently dead GC.

To follow on this branch

A proxmox-monitor.prose.md section documenting the schedule and this check (superseding the separate docs PR #116, whose text states the schedule in the wrong timezone - it is 20:00 local = 00:00 UTC, not 20:00 UTC).

Firstmate runs the pre-delivery review and the merge.

## What Adds a liveness leg to scripts/proxmox-monitor.sh that reads the local PBS datastore's garbage-collection state from inside CT 107 (where proxmox-backup-manager lives) and FAILS when storepve-datastore's last-run-endtime is older than 48h, reporting the age and pending bytes. ## Why The nightly GC job (/etc/cron.d/pbs-gc -> /usr/local/bin/pbs-gc.sh) had never worked: it called proxmox-backup-manager on the storepve HOST, where only proxmox-backup-client exists, so every run died 'command not found'. The host script is fixed and verified (a real run completed OK), and this leg is the signal that catches a silently dead GC. ## To follow on this branch A proxmox-monitor.prose.md section documenting the schedule and this check (superseding the separate docs PR #116, whose text states the schedule in the wrong timezone - it is 20:00 local = 00:00 UTC, not 20:00 UTC). Firstmate runs the pre-delivery review and the merge.

abiba-bot added 1 commit 2026-09-19 02:07:35 +00:00

feat(proxmox): Add PBS GC liveness signal ... 8ae4b59150

Add monitoring leg that checks storepve-datastore GC health:
- Reads GC state from CT 107 via pct exec
- FAILS if last-run-endtime is older than 48h
- Reports age in hours and pending-bytes status
- Uses JSON parsing for reliable data extraction

Test: All 5 legs OK, Exit 0.
Branch: fix/pbs-gc-liveness-signal-20260919

abiba-bot referenced this pull request 2026-09-19 02:07:47 +00:00

docs: add PBS GC schedule to proxmox-monitor contract #116

abiba-bot added 1 commit 2026-09-19 02:14:27 +00:00

docs(proxmox): Add PBS GC documentation to proxmox-monitor.prose.md ... da8f5f43c9

Document the PBS GC schedule (00:00 UTC, not 20:00 UTC as PR #116 said),
what actually runs (pbs-gc.sh -> pct exec 107 -- proxmox-backup-manager),
datastore location (CT 107's /mnt/pbs-backup on /tank/pbs-backup, NOT
/media/easystore2), and the new liveness check (48h threshold, reports
age in hours, explicit healthy line).

Branch: fix/pbs-gc-liveness-signal-20260919

abiba-bot added 1 commit 2026-09-19 02:28:45 +00:00

fix(proxmox): Fix PBS GC liveness leg + add comprehensive test suite ... f59d1a2159

(a) Probe-failure detection: now treats empty OR unparseable JSON as
     probe-failed, not never-run. This prevents 'command not found'
     outputs from being rendered as service verdicts.

(b) pending-bytes: now extracted from JSON and reported in stale verdict.

(c) Null endtime: use .get() with explicit None check, not 0 fallback.
     null values now correctly trigger never-run verdict instead of
     arithmetic crash (set -u).

(d) Tests: Added 14-assertion stub-driven suite covering: healthy,
     stale (>48h), probe-failed (empty and unparseable), null endtime,
     datastore absent. Each test stubs ssh/curl to verify exact
     behavior against the pre-fix head.

Branch: fix/pbs-gc-liveness-signal-20260919

abiba-bot referenced this pull request 2026-09-19 02:36:04 +00:00

fix(infra-monitoring): probe the real Docker Stats and PVE exporter ports #120

abiba-bot added 1 commit 2026-09-19 02:43:56 +00:00

docs(proxmox): Document all six PBS GC verdict shapes ... 8ff13d38f3

The Liveness Check section now documents ALL SIX verdict shapes exactly
as emitted by proxmox-monitor.sh:

1. ✅ PBS GC: healthy (last run Nh ago, pending-bytes: N B)
2. 🔴 PBS GC: stale (last run Nh ago, pending-bytes: N B)
3. 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (expected JSON, got 000)
4. 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (unparseable JSON)
5. 🔴 PBS GC: never-run (storepve-datastore not found in GC list)
6. 🔴 PBS GC: never-run (storepve-datastore has no last-run-endtime)

Fixed the quoted healthy example (line ~114) to include the pending-bytes
suffix the code now appends. Previously the prose only documented the
probe-failed shape, missing the PR's own headline cases (never-run).

Proof: grep -n 'never-run' proxmox-monitor.prose.md now returns two lines
(lines 106 and 108), documenting both never-run variants.

Branch: fix/pbs-gc-liveness-signal-20260919

abiba-bot merged commit 66ad5ac89d into master 2026-09-19 02:51:35 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

abiba-bot jerome (Jerome Kwame Tabiri) kagentz-bot kwame (Kwame Tabiri) mumuni-bot (Mumuni 🦅 (Syslog Falcon)) tanko-bot theo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#119