From d829f595b411d9265733bfccd7e8d35c1c730b28 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 9 Jul 2026 06:23:44 +0000 Subject: [PATCH] fix: IP drift and container count from live testing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Testing the consolidated litellm-self-heal contract against live infra found: - IP drift: pi host is .24, not .65 (confused with ra-h-os bridge) - Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations) - Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables - Container count: 10 running on CT 116, not 8 as contract stated - Added harness-docker-stats + harness-pve-exporter to litellm-self-heal - Updated container health check list Verified: GPU monitor restarted on .24:9100, ornith model inference working, all 10 containers healthy. --- gpu-fleet.prose.md | 10 +++++----- gpu-monitor.prose.md | 8 ++++---- litellm-self-heal.prose.md | 9 ++++++--- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 9e14cfd..307d084 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -142,7 +142,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act |-------|-----|-----|-----|--------| | Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome | | Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root | -| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) | +| Abiba | 100 | .24 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) | | Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM | | Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH | | Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH | @@ -159,10 +159,10 @@ If no SSH access, send Zulip DM via abiba-bot. | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) | | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health | | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) | -| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) | -| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) | -| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server | -| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard | +| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) | +| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) | +| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server | +| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard | | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) | | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. | diff --git a/gpu-monitor.prose.md b/gpu-monitor.prose.md index 11cb793..a368c32 100644 --- a/gpu-monitor.prose.md +++ b/gpu-monitor.prose.md @@ -19,7 +19,7 @@ agent: abiba ``` ┌─────────────────────────────────────────────────────────┐ -│ GPU Monitor Server (:9100) — 192.168.68.65 │ +│ GPU Monitor Server (:9100) — 192.168.68.24 │ │ Polls every 15s, serves dashboard + JSON API │ └──┬──────────┬──────────┬────────────────────────────────┘ │ │ │ @@ -35,7 +35,7 @@ agent: abiba Note: JSON sidecar exporters at :8090 were never deployed on any GPU host. Router falls back to GPU /health direct probe. Monitor should use router /health/unified as source of truth for GPU status. -Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot +Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot poll .15:8080 directly; must go through router on .116. ``` @@ -118,8 +118,8 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly). | File | Host | Purpose | |------|------|---------| -| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 | -| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard | +| `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 | +| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard | | `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router | ## Execution diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index 26a5309..53a8860 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -81,6 +81,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) | harness-dashboard | inference-harness-dashboard | :3000 | /health | | harness-grafana | grafana/grafana | :3000→:3001 | /api/health | | harness-prometheus | prom/prometheus | :9090 | /-/healthy | +| harness-docker-stats | python:3.12-alpine | — | container stats exporter | +| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus | ## Maintains @@ -111,9 +113,10 @@ Run this first on every cycle. Results feed into remediation rules below. - GET http://{{backend_host}}/litellm/health/liveliness → expect 200 ### 3. Check backend container health -- SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy -- Critical: harness-litellm, harness-router, harness-nginx, harness-postgres -- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus +- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy +- Critical: harness-litellm, harness-nginx, harness-postgres +- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter +- Deprecated but running: harness-router (not in path, reference only) ### 4. Check GPU fleet health (via fleet dashboard) - GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON