feat(contracts): add infrastructure-maintenance contract
New responsibility contract consolidating host-level system maintenance and Docker image lifecycle management, filling the gap left by infrastructure-update (which owns cluster-wide apt waves). Scope: - OS package updates on primary host with pre-update snapshot/backup check - Docker image pulls for LiteLLM, SearXNG, and other running containers - Container restarts with per-stack health verification - Post-update verification: LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways - Rollback on failure (image/apt/config restore) with circuit breaker Owner: ops (firstmate secondmate). Trigger: weekly Sunday 2am ET. Escalation: warning/critical->abiba+mumuni, fatal->abiba+mumuni+kwame. circuit_breaker: max_retries 2, window 7200, trip_action escalate_to_fatal. depends_on: infrastructure-monitoring (pre-update health baseline). Registry: - Add infrastructure-maintenance to by_category.maintenance, by_domain.infrastructure, by_owner.ops (new), by_trigger.scheduled, by_sensitivity.high - Add 'ops' to owners list - Move infrastructure-update owner abiba -> ops (in contracts entry + by_owner index) Also adds '## Maintaining this file' section to AGENTS.md per fm-ensure-agents-md.
This commit is contained in:
root
@@ -0,0 +1,195 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: infrastructure-maintenance
|
||||
description: >
|
||||
Weekly system-level maintenance for the Syslog inference fleet: OS package
|
||||
updates on the primary host, Docker image pulls for LiteLLM/SearXNG and other
|
||||
running containers, container restarts with health verification, post-update
|
||||
verification of every critical service (LiteLLM proxy, SearXNG, Zulip, Gitea,
|
||||
PM2 processes, Hermes gateways), and rollback on failure. Consolidates the
|
||||
raw shell scripts that previously did this piecemeal and closes the Docker
|
||||
image management gap that infrastructure-update left open (infra-update owns
|
||||
cluster-wide apt waves; this contract owns the host-level maintenance loop
|
||||
on the primary host + Docker image lifecycle). Runs Sunday 2am ET. Owner:
|
||||
ops (firstmate secondmate). Blast radius: an unverified image pull can break
|
||||
LiteLLM (all agents lose inference) or SearXNG (search-stack down); a bad apt
|
||||
upgrade can leave the host in a half-upgraded state. Pre-update backup check
|
||||
and rollback are mandatory for this reason.
|
||||
agent: ops
|
||||
triggers:
|
||||
- weekly (Sunday 02:00 ET) via cron
|
||||
- on demand when ops/abiba triggers "infra maintenance"
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
## Maintains
|
||||
|
||||
- maintenance-status: { phase: idle|preflight|apt|images|restarts|verify|rollback|done|failed, host, step, result, timestamp }
|
||||
- image-baseline: { service, current_tag, pulled_tag, digest, updated_at } — last known-good image per container
|
||||
- apt-state: { upgradable_before, upgradable_after, held_broken, kernel_reboot_required }
|
||||
- health-baseline: snapshot of critical-service health captured pre-update (used for regression check post-update)
|
||||
- rollback-snapshot: { backup_path, configs, image_digests, timestamp } — restore point created in preflight
|
||||
- maintenance-history: array of past runs with phase results and any escalations
|
||||
|
||||
## Scope
|
||||
|
||||
Primary host is the maintenance host where apt updates apply. Docker image pulls
|
||||
span the two Docker ecosystems that run critical services. Topology, CT IDs, and
|
||||
IPs are live-state fields — verify against `infrastructure-control.prose.md`
|
||||
(the source of truth) and the live system before mutating.
|
||||
|
||||
| Host | IP | Role | Trust |
|
||||
|------|----|------|-------|
|
||||
| CT 116 (syslog-api) | 192.168.68.116 | LiteLLM proxy + Grafana + Prometheus (inference harness) | ⚠️ VERIFY-BEFORE-USE |
|
||||
| VM 109 (docker-vm) | 192.168.68.7 | SearXNG + Firecrawl + home stack (Docker host) | ⚠️ VERIFY-BEFORE-USE |
|
||||
| CT 117 (zulip) | 192.168.68.19 | Zulip (storepve bridge IP .19) | ⚠️ VERIFY-BEFORE-USE |
|
||||
| Gitea | https://git.sysloggh.net | Prose-contracts + agent configs source control | ⚠️ VERIFY-BEFORE-USE |
|
||||
| CT 100 (abiba/pi) | 192.168.68.24 | PM2 processes (pi agent harness) | ⚠️ VERIFY-BEFORE-USE |
|
||||
|
||||
> "Primary host" for the apt phase is the host the ops agent runs maintenance
|
||||
> from. Confirm which host that is against infrastructure-control before
|
||||
> running; do not assume. If the ops agent is containerized/CT-based, apt runs
|
||||
> inside that CT.
|
||||
|
||||
## Requires
|
||||
|
||||
- SSH/exec access to CT 116 (.116) and VM 109 (.7) for Docker operations
|
||||
- `apt`, `docker`, `docker compose` available on target hosts
|
||||
- LiteLLM master key available (Infisical vault, `LITELLM_API_KEY`) for health verification
|
||||
- `infrastructure-monitoring` run completed within the last 30 minutes — provides the pre-update health baseline used by the regression check
|
||||
- Writable backup directory `/tmp/infra-maintenance-backup-<date>/` on each mutated host
|
||||
- Proxmox snapshot of the primary host available (or confirmed not required) before apt phase
|
||||
|
||||
## Continuity
|
||||
|
||||
- Self-driven: weekly cron `0 2 * * 0` (Sunday 02:00 ET)
|
||||
- Also wakes on: explicit "infra maintenance" trigger from ops/abiba
|
||||
- Depends on `infrastructure-monitoring` for the pre-update health baseline — do not run if the last monitoring run is stale (>30 min) or RED; abort and escalate instead
|
||||
|
||||
## Execution
|
||||
|
||||
### Phase 0 — Preflight (snapshot/backup check + health baseline)
|
||||
|
||||
1. **Capture health baseline** — run the `infrastructure-monitoring` postcondition checks (LiteLLM, Zulip, Gitea, SearXNG, Proxmox API) and record results as `health-baseline`. If any critical service is already down, **abort**: maintenance must not run on a degraded fleet.
|
||||
2. **Backup check** — confirm a Proxmox snapshot of the primary host exists OR `/tmp/infra-maintenance-backup-<date>/` was created this run. Snapshot critical config files into the backup dir:
|
||||
- `/opt/inference-harness/docker-compose.yml`, `/opt/inference-harness/litellm_config.yaml` (CT 116)
|
||||
- `/opt/search-stack/searxng/docker-compose.yml`, `/opt/search-stack/firecrawl-source/docker-compose.yaml` (VM 109)
|
||||
3. **Record image baseline** — `docker inspect --format '{{.Image}} {{.Config.Image}}' <container>` for every running container on .116 and .7; store digests in `image-baseline` so rollback can restore them.
|
||||
4. **Disk check** — `df -h` on each mutated host; abort if free space <20% (apt upgrade + image pulls need headroom).
|
||||
|
||||
### Phase 1 — OS package updates (primary host)
|
||||
|
||||
```bash
|
||||
# On the primary host only (VERIFY host against infrastructure-control first)
|
||||
apt update
|
||||
apt upgrade -y
|
||||
```
|
||||
|
||||
- Capture `apt list --upgradable` before and after → store in `apt-state`.
|
||||
- If apt reports held/broken packages (`apt-get -s upgrade | grep -i broken`, or non-zero exit), **stop** — do not force. Record `held_broken` and go to rollback/escalate.
|
||||
- If `/var/run/reboot-required` exists after upgrade, flag `kernel_reboot_required: true` in `apt-state` but **do not reboot automatically** — that's a separate coordinated action (see infra-update Wave 4). Note it in the report.
|
||||
|
||||
### Phase 2 — Docker image pulls
|
||||
|
||||
Pull latest stable tags for every running container. Do NOT pin to `:main`/`:nightly` — use stable tags where the compose file specifies them; otherwise `latest`.
|
||||
|
||||
```bash
|
||||
# CT 116 (.116) — inference harness
|
||||
cd /opt/inference-harness && docker compose pull
|
||||
|
||||
# VM 109 (.7) — search + home stacks
|
||||
cd /opt/search-stack/searxng && docker compose pull
|
||||
cd /opt/search-stack/firecrawl-source && docker compose pull
|
||||
# any other running stacks on .7 (home stack, audiobookshelf) — pull per their compose files
|
||||
```
|
||||
|
||||
- LiteLLM and SearXNG are the two explicitly required pulls; "any other running containers" means every stack with a compose file on .116 and .7.
|
||||
- Record pulled tag + digest per service in `image-baseline`.
|
||||
|
||||
### Phase 3 — Container restarts with health verification
|
||||
|
||||
Restart one stack at a time, verify health before moving to the next. Do not restart everything at once — a failure mid-wave must leave the rest running.
|
||||
|
||||
```bash
|
||||
# CT 116
|
||||
cd /opt/inference-harness && docker compose up -d
|
||||
# VM 109
|
||||
cd /opt/search-stack/searxng && docker compose up -d
|
||||
cd /opt/search-stack/firecrawl-source && docker compose up -d
|
||||
```
|
||||
|
||||
After each stack comes up, wait for health (max 120s):
|
||||
- `docker ps` shows the container `Up` (and `healthy` if a healthcheck is defined)
|
||||
- Service-specific probe passes (see Phase 4 probes)
|
||||
|
||||
If a stack fails to come up within 120s, **stop the wave** and go to rollback for that stack only; do not proceed to the next.
|
||||
|
||||
### Phase 4 — Post-update service verification
|
||||
|
||||
After ALL updates (apt + images + restarts), verify every critical service is back up and matches the pre-update baseline. This is the regression gate.
|
||||
|
||||
| Service | Probe | Expect |
|
||||
|---------|-------|--------|
|
||||
| LiteLLM proxy | `curl -sf http://192.168.68.116/litellm/v1/models` | 200 OK, models returned |
|
||||
| LiteLLM MCP gateway | `curl -sf http://192.168.68.116:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` | 90 tools (23 RA-H OS + 67 GitHub) |
|
||||
| SearXNG | `curl -sf http://192.168.68.7:8888` | 200 OK |
|
||||
| Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
|
||||
| Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
|
||||
| PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
|
||||
| Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
|
||||
|
||||
Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
|
||||
|
||||
## Rollback Protocol
|
||||
|
||||
If ANY service in Phase 4 fails to come back up (or regresses vs baseline):
|
||||
|
||||
1. **Image rollback** — for the failing stack, restore the previous image:
|
||||
```bash
|
||||
# Restore from recorded image-baseline digest
|
||||
docker compose down
|
||||
docker compose pull <previous_digest> # or set image: <digest> in compose and up -d
|
||||
docker compose up -d
|
||||
```
|
||||
2. **APT rollback** — restore the primary host from the Proxmox snapshot taken/confirmed in Phase 0. If no snapshot, `apt install <pkg>=<old_version>` per package using apt history (`/var/log/apt/history.log`).
|
||||
3. **Config rollback** — restore configs from `/tmp/infra-maintenance-backup-<date>/`.
|
||||
4. **Re-verify** — re-run the Phase 4 probes on the rolled-back service. If still failing, escalate (do not loop — circuit breaker below).
|
||||
5. **Escalate** — send a Zulip DM to abiba + mumuni with: failing service, phase, baseline vs current, rollback actions taken, backup path.
|
||||
|
||||
## Circuit Breaker
|
||||
|
||||
- `max_retries: 2` per failing phase — after 2 rollback attempts on the same service, stop and escalate.
|
||||
- `window: 7200` seconds — no more than 2 retries within a 2-hour window.
|
||||
- `trip_action: escalate_to_fatal` — when tripped, escalate to fatal (abiba + mumuni + kwame) and pause; a human must clear before the next scheduled run.
|
||||
|
||||
## Report
|
||||
|
||||
After completion (or on abort), emit a receipt (JSON) to `~/.hermes/runs/infrastructure-maintenance/` and send a Zulip DM summary:
|
||||
|
||||
```
|
||||
🛠 Infrastructure Maintenance — YYYY-MM-DD
|
||||
|
||||
Phase: apt | images | restarts | verify | rollback
|
||||
Primary host: <host>
|
||||
APT: <N> packages upgraded, <M> held/broken, kernel_reboot_required=<bool>
|
||||
Images pulled: LiteLLM <tag>, SearXNG <tag>, <others>
|
||||
Services: all GREEN | <service> FAILED (rolled back)
|
||||
Baseline regression: none | <details>
|
||||
Backup: /tmp/infra-maintenance-backup-YYYYMMDD/
|
||||
Escalation: none | warning | critical | fatal
|
||||
```
|
||||
|
||||
## Verification Postconditions
|
||||
|
||||
- All critical services running after update (Phase 4 all GREEN)
|
||||
- No regressions from pre-update health baseline (Phase 0 baseline)
|
||||
- Docker containers on latest stable tags (`image-baseline.pulled_tag` recorded)
|
||||
- APT packages up to date with no held broken packages (`apt-state.held_broken == 0`)
|
||||
|
||||
## Related Contracts
|
||||
|
||||
- `infrastructure-update.prose.md` — cluster-wide apt waves across the PVE cluster + CTs/VMs (the build-phase update); this contract handles the host-level maintenance loop and Docker image lifecycle.
|
||||
- `infrastructure-monitoring.prose.md` — provides the pre-update health baseline (depends_on).
|
||||
- `infrastructure-control.prose.md` — topology source of truth (CT IDs, IPs, hostnames).
|
||||
- `litellm-health.prose.md` — LiteLLM probe details.
|
||||
- `proxmox-monitor.prose.md` — Docker stats + monitoring stack health.
|
||||
Reference in New Issue
Block a user