fix: Firecrawl health endpoint is / not /health

infrastructure-update.prose.md

@@ -3,7 +3,7 @@ kind: responsibility
 name: infrastructure-update
 description: >
   Autonomous system-wide update contract covering all 5 Proxmox nodes,
-  15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
+  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
   Docker images, and container stacks in safe waves with health checks
   and automatic rollback on failure.
 agent: abiba
@@ -11,7 +11,7 @@ triggers:
   - on "infra update" command
   - weekly (Sunday 03:00 EDT) via cron
   - on security advisory relay from Mumuni
-version: 1.0.0
+version: 1.1.0
 ---
 
 ## Maintains
@@ -24,11 +24,11 @@ version: 1.0.0
 
 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
+2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
-3. ✅ Docker healthy on .7, .116
+3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
 4. ✅ LiteLLM health check passing
 5. ✅ Zulip server reachable
-6. ✅ GPU fleet healthy (all 3 GPUs)
+6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
 7. ✅ Disk >20% free on all nodes
 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
 
@@ -38,7 +38,7 @@ Before ANY update wave:
 |--------|------|---------|---------|
 | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
-| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
+| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
 | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
 | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
 
@@ -55,16 +55,16 @@ Before ANY update wave:
 | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
-| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
 | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
 | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
-| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
+| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
-| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |
+| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
 
 **Verify after Wave 2:**
-- All CTs running: check via Proxmox API
+- All VMs/CTs running: check via Proxmox API
-- LiteLLM healthy: `curl :4001/health/liveliness`
+- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
-- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router
 - Zulip agents connected: check Mumuni/Tanko gateway state
 - Abiba PM2 processes online: `pm2 status`
 
@@ -72,20 +72,20 @@ Before ANY update wave:
 
 | Host | Stack | Command |
 |------|-------|---------|
-| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
-| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
-| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
-| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
-| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
 | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
 
 **Verify after Wave 3:**
 - All containers healthy: `docker ps` on each host
-- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
+- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
 - Zulip test: send test message to #agent-hub
-- Dashboard loading: `curl :3001/`
+- Dashboard loading: `curl localhost:3001/` (via CT 116)
-- Firecrawl test: `curl :3002/health`
+- Firecrawl test: `curl :3002/`
-- SearXNG test: `curl storepve:8888`
+- SearXNG test: `curl :8888`
 
 ## Wave 4: Proxmox Kernel Reboot (if needed)
 
@@ -111,16 +111,18 @@ If ANY verification fails:
 
 Before Wave 1, snapshot these files:
 ```
-/opt/inference-harness/litellm_config.yaml        (.116)
+/opt/inference-harness/docker-compose.yml               (CT 116 .116)
-/opt/inference-harness/docker-compose.yml          (.116)
+/opt/inference-harness/litellm_config.yaml              (CT 116 .116)
-/etc/nginx/nginx.conf                              (.116 container)
+/opt/monitoring/prometheus.yml                          (CT 116 .116)
-/opt/search-stack/*/docker-compose.yaml            (.7)
+/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
-/root/.pi/agent/extensions/config.yaml             (.24)
+/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
-/etc/systemd/system/ornith-server.service          (.15)
+/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
-/etc/systemd/system/llama-server.service           (.8, .110)
+/opt/home_stack/docker-compose.yml                      (VM 109 .7)
+/opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
+/root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
 ```
 
-Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
 
 ## Security-Specific Updates
 
@@ -134,8 +136,8 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
 ## Success Criteria
 
 - [ ] All 5 PVE nodes updated, no reboot-loop
-- [ ] All CTs running post-update
+- [ ] All VMs/CTs running post-update
-- [ ] All 22 Docker containers healthy
+- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
 - [ ] LiteLLM inference passing (syslog-auto test)
 - [ ] Zulip server + all 3 agents connected
 - [ ] GPU fleet at full capacity (3/3)
@@ -148,7 +150,7 @@ After completion, send Zulip DM:
 ```
 📋 Infrastructure Update — YYYY-MM-DD
 
-Updated: 5 PVE nodes, 12 CTs, 22 containers
+Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
 Security fixes: N CVEs patched
 Downtime: <service> <duration>
 Failures: none / <details>