prose-contracts/infrastructure-maintenance.prose.md

kind, name, description, agent, triggers, version

kind	name	description	agent	triggers	version
responsibility	infrastructure-maintenance	Weekly system-level maintenance for the Syslog inference fleet: OS package updates on the primary host, Docker image pulls for LiteLLM/SearXNG and other running containers, container restarts with health verification, post-update verification of every critical service (LiteLLM proxy, SearXNG, Zulip, Gitea, PM2 processes, Hermes gateways), and rollback on failure. Consolidates the raw shell scripts that previously did this piecemeal. This contract owns the HOST-LEVEL weekly maintenance loop on the primary host plus Docker image pulls ONLY for .116 and .7, while infrastructure-update owns the FULL-FLEET cluster-wide wave (apt across the full PVE cluster + CTs/VMs AND its Docker image Wave 3 across all stacks). Runs Sunday 2am ET. Owner: ops (firstmate secondmate). Blast radius: an unverified image pull can break LiteLLM (all agents lose inference) or SearXNG (search-stack down); a bad apt upgrade can leave the host in a half-upgraded state. Pre-update backup check and rollback are mandatory for this reason.	ops	weekly (Sunday 02:00 ET) via cron on demand when ops/abiba triggers "infra maintenance"	1.0.0

Maintains

• maintenance-status: { phase: idle|preflight|apt|images|restarts|verify|rollback|done|failed, host, step, result, timestamp }
• image-baseline: { service, current_tag, pulled_tag, digest, updated_at } — last known-good image per container
• apt-state: { upgradable_before, upgradable_after, held_broken, kernel_reboot_required }
• health-baseline: snapshot of critical-service health captured pre-update (used for regression check post-update)
• rollback-snapshot: { backup_path, configs, image_digests, timestamp } — restore point created in preflight
• maintenance-history: array of past runs with phase results and any escalations

Scope

Primary host is the maintenance host where apt updates apply. Docker image pulls span the two Docker ecosystems that run critical services. infrastructure-update runs the full-fleet cluster-wide wave (including its Wave 3 Docker pulls across all stacks/hosts); this contract runs a narrower host-level weekly pull limited to .116 and .7. Topology, CT IDs, and IPs are live-state fields — verify against infrastructure-control.prose.md (the source of truth) and the live system before mutating.

Host	IP	Role	Trust
CT 116 (syslog-api)	192.168.68.116	LiteLLM proxy + Grafana + Prometheus (inference harness)	⚠️ VERIFY-BEFORE-USE
VM 109 (docker-vm)	192.168.68.7	SearXNG + Firecrawl + home stack (Docker host)	⚠️ VERIFY-BEFORE-USE
CT 117 (zulip)	192.168.68.19	Zulip (storepve bridge IP .19)	⚠️ VERIFY-BEFORE-USE
Gitea	https://git.sysloggh.net	Prose-contracts + agent configs source control	⚠️ VERIFY-BEFORE-USE
CT 100 (abiba/pi)	192.168.68.24	PM2 processes (pi agent harness)	⚠️ VERIFY-BEFORE-USE

"Primary host" for the apt phase is the host the ops agent runs maintenance from. Confirm which host that is against infrastructure-control before running; do not assume. If the ops agent is containerized/CT-based, apt runs inside that CT.

Requires

• SSH/exec access to CT 116 (.116) and VM 109 (.7) for Docker operations
• apt, docker, docker compose available on target hosts
• LiteLLM master key available (Infisical vault, LITELLM_API_KEY) for health verification
• infrastructure-monitoring run completed within the last 30 minutes — provides the pre-update health baseline used by the regression check
• Writable backup directory /tmp/infra-maintenance-backup-<date>/ on each mutated host
• Proxmox snapshot of the primary host available (or confirmed not required) before apt phase

Continuity

• Self-driven: weekly cron 0 2 * * 0 (Sunday 02:00 ET)
• Also wakes on: explicit "infra maintenance" trigger from ops/abiba
• Depends on infrastructure-monitoring for the pre-update health baseline — do not run if the last monitoring run is stale (>30 min) or RED; abort and escalate instead

Execution

Phase 0 — Preflight (snapshot/backup check + health baseline)

1. Capture health baseline — run the infrastructure-monitoring postcondition checks (LiteLLM, Zulip, Gitea, SearXNG, Proxmox API) and record results as health-baseline. If any critical service is already down, abort: maintenance must not run on a degraded fleet.
2. Backup check — confirm a Proxmox snapshot of the primary host exists OR /tmp/infra-maintenance-backup-<date>/ was created this run. Snapshot critical config files into the backup dir:
   • /opt/inference-harness/docker-compose.yml, /opt/inference-harness/litellm_config.yaml (CT 116)
   • /opt/search-stack/searxng/docker-compose.yml, /opt/search-stack/firecrawl-source/docker-compose.yaml (VM 109)
3. Record image baseline — docker inspect --format '{{.Image}} {{.Config.Image}}' <container> for every running container on .116 and .7; store digests in image-baseline so rollback can restore them.
4. Disk check — df -h on each mutated host; abort if free space <20% (apt upgrade + image pulls need headroom).

Phase 1 — OS package updates (primary host)

# On the primary host only (VERIFY host against infrastructure-control first)
apt update
apt upgrade -y

• Capture apt list --upgradable before and after → store in apt-state.
• If apt reports held/broken packages (apt-get -s upgrade | grep -i broken, or non-zero exit), stop — do not force. Record held_broken and go to rollback/escalate.
• If /var/run/reboot-required exists after upgrade, flag kernel_reboot_required: true in apt-state but do not reboot automatically — that's a separate coordinated action (see infra-update Wave 4). Note it in the report.

Phase 2 — Docker image pulls

Pull latest stable tags for every running container. Do NOT pin to :main/:nightly — use stable tags where the compose file specifies them; otherwise latest.

# CT 116 (.116) — inference harness
cd /opt/inference-harness && docker compose pull

# VM 109 (.7) — search + home stacks
cd /opt/search-stack/searxng && docker compose pull
cd /opt/search-stack/firecrawl-source && docker compose pull
# any other running stacks on .7 (home stack, audiobookshelf) — pull per their compose files

• LiteLLM and SearXNG are the two explicitly required pulls; "any other running containers" means every stack with a compose file on .116 and .7.
• Record pulled tag + digest per service in image-baseline.

Phase 3 — Container restarts with health verification

Restart one stack at a time, verify health before moving to the next. Do not restart everything at once — a failure mid-wave must leave the rest running.

# CT 116
cd /opt/inference-harness && docker compose up -d
# VM 109
cd /opt/search-stack/searxng && docker compose up -d
cd /opt/search-stack/firecrawl-source && docker compose up -d

After each stack comes up, wait for health (max 120s):

• docker ps shows the container Up (and healthy if a healthcheck is defined)
• Service-specific probe passes (see Phase 4 probes)

If a stack fails to come up within 120s, stop the wave and go to rollback for that stack only; do not proceed to the next.

Phase 4 — Post-update service verification

After ALL updates (apt + images + restarts), verify every critical service is back up and matches the pre-update baseline. This is the regression gate.

Service	Probe	Expect
LiteLLM proxy	curl -sf http://192.168.68.116/litellm/v1/models	200 OK, models returned
LiteLLM MCP gateway	curl -sf http://192.168.68.116:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"	90 tools (23 RA-H OS + 67 GitHub)
SearXNG	curl -sf http://192.168.68.7:8888	200 OK
Zulip	curl -sf https://chat.sysloggh.net/api/v1/server_settings	200 OK
Gitea	curl -sf https://git.sysloggh.net/api/v1/version	200 OK
PM2 processes	pm2 jlist (CT 100)	all pi-agent processes online
Hermes gateways	SSH to Mumuni CT 114, Tanko CT 112; systemctl is-active hermes-gateway	active for each

Regression check: every service that was GREEN in health-baseline must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.

Rollback Protocol

If ANY service in Phase 4 fails to come back up (or regresses vs baseline):

1. Image rollback — for the failing stack, restore the previous image:

   # Restore from recorded image-baseline digest
   docker compose down
   # Pin the service image to the recorded digest in compose, then recreate
   #   image: <name>@sha256:<previous_digest>
   docker compose pull && docker compose up -d
   

2. APT rollback — restore the primary host from the Proxmox snapshot taken/confirmed in Phase 0. If no snapshot, apt install <pkg>=<old_version> per package using apt history (/var/log/apt/history.log).
3. Config rollback — restore configs from /tmp/infra-maintenance-backup-<date>/.
4. Re-verify — re-run the Phase 4 probes on the rolled-back service. If still failing, escalate (do not loop — circuit breaker below).
5. Escalate — send a Zulip DM to abiba + mumuni with: failing service, phase, baseline vs current, rollback actions taken, backup path.

Circuit Breaker

• max_retries: 2 per failing phase — after 2 rollback attempts on the same service, stop and escalate.
• window: 7200 seconds — no more than 2 retries within a 2-hour window.
• trip_action: escalate_to_fatal — when tripped, escalate to fatal (abiba + mumuni + kwame) and pause; a human must clear before the next scheduled run.

Report

After completion (or on abort), emit a receipt (JSON) to ~/.hermes/runs/infrastructure-maintenance/ and send a Zulip DM summary:

🛠 Infrastructure Maintenance — YYYY-MM-DD

Phase: apt | images | restarts | verify | rollback
Primary host: <host>
APT: <N> packages upgraded, <M> held/broken, kernel_reboot_required=<bool>
Images pulled: LiteLLM <tag>, SearXNG <tag>, <others>
Services: all GREEN | <service> FAILED (rolled back)
Baseline regression: none | <details>
Backup: /tmp/infra-maintenance-backup-YYYYMMDD/
Escalation: none | warning | critical | fatal

Verification Postconditions

• All critical services running after update (Phase 4 all GREEN)
• No regressions from pre-update health baseline (Phase 0 baseline)
• Docker containers on latest stable tags (image-baseline.pulled_tag recorded)
• APT packages up to date with no held broken packages (apt-state.held_broken == 0)

Related Contracts

• infrastructure-update.prose.md — owns the full-fleet cluster-wide wave INCLUDING its Wave 3 Docker image updates across all stacks (SearXNG, Firecrawl, Inference Harness on .116, home stack, audiobookshelf); infrastructure-maintenance is a deliberately narrower host-level weekly pull scoped to .116 and .7.
• infrastructure-monitoring.prose.md — provides the pre-update health baseline (depends_on).
• infrastructure-control.prose.md — topology source of truth (CT IDs, IPs, hostnames).
• litellm-health.prose.md — LiteLLM probe details.
• proxmox-monitor.prose.md — Docker stats + monitoring stack health.