fix: IP drift and container count from live testing
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 13m1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 13m1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
Testing the consolidated litellm-self-heal contract against live infra found: - IP drift: pi host is .24, not .65 (confused with ra-h-os bridge) - Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations) - Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables - Container count: 10 running on CT 116, not 8 as contract stated - Added harness-docker-stats + harness-pve-exporter to litellm-self-heal - Updated container health check list Verified: GPU monitor restarted on .24:9100, ornith model inference working, all 10 containers healthy.
This commit is contained in:
@@ -81,6 +81,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
|
||||
| harness-dashboard | inference-harness-dashboard | :3000 | /health |
|
||||
| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
|
||||
| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
|
||||
| harness-docker-stats | python:3.12-alpine | — | container stats exporter |
|
||||
| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
|
||||
|
||||
## Maintains
|
||||
|
||||
@@ -111,9 +113,10 @@ Run this first on every cycle. Results feed into remediation rules below.
|
||||
- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
|
||||
|
||||
### 3. Check backend container health
|
||||
- SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
|
||||
- Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
|
||||
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
|
||||
- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
|
||||
- Critical: harness-litellm, harness-nginx, harness-postgres
|
||||
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
|
||||
- Deprecated but running: harness-router (not in path, reference only)
|
||||
|
||||
### 4. Check GPU fleet health (via fleet dashboard)
|
||||
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
|
||||
|
||||
Reference in New Issue
Block a user