fix: align contract name fields to filenames (litellm-health, infrastructure-monitoring)

- litellm-health: 'check-litellm-health' -> 'litellm-health' - infrastructure-monitoring: 'deploy-monitoring-stack' -> 'infrastructure-monitoring' Fixes 'prose run <filename>' mismatch. No external references broken.
prose: add Strix Halo thermal safeguard (-n 8192 cap) to gpu-fleet
2026-07-02 20:35:44 +00:00 · 2026-07-02 03:51:36 +00:00 · 2026-07-02 02:56:27 +00:00 · 2026-07-02 02:50:59 +00:00 · 2026-07-02 00:29:10 +00:00 · 2026-07-02 00:01:56 +00:00
5 changed files with 387 additions and 69 deletions
@@ -77,7 +77,7 @@ triggers:
 |-------|-----|------|------|--------|
 | qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
 | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
-| ornith-1.0-35b | Strix Halo CPU | .15 (amdpve) | 28.2/64GB | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |

 ## Operations

@@ -130,7 +130,7 @@ Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, k
 | Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
 | Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
 | Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
-| Koby | `sk-wb86PvcZXWkjShAIBxrpDg` | ? | Zulip DM |
+| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
 | Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
 | Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |

@@ -150,7 +150,8 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
 | `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
 | `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
-| `/etc/systemd/system/llama-server.service` | .8, .110, .15 | llama-server daemons |
+| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
+| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |

 ## Prometheus & Grafana

@@ -170,7 +171,11 @@ If no SSH access, send Zulip DM via abiba-bot.
 - **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
 - **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
 - **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
- **Strix Halo GPU**: ROCm 6.3.1 installed, HIP compiler works via clang++ + device libs. **Blocked**: HSA runtime driver needs `rocm-llvm` → `libstdc++-11-dev` not on Debian 13. Fixes: Docker with ROCm, or reinstall to Ubuntu 24.04.
+- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
+- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
+- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
+- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
+- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
 - **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
 - **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
 - **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.
@@ -181,8 +186,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 |-----|-------|-----------|--------------|----------|---------|
 | RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
 | RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
-| Strix Halo (.15) | ornith-1.0-35b | 70 | 209 | 70 | 6 |
+| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |

 Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
 Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
 History stored at `/root/data/toks-history.json` with 7-day rolling window.
+
+**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
@@ -21,28 +21,33 @@ agent: abiba
 ┌─────────────────────────────────────────────────────────┐
 │  GPU Monitor Server (:9100) — 192.168.68.65              │
 │  Polls every 15s, serves dashboard + JSON API            │
-└──┬──────────┬──────────┬──────────┬─────────────────────┘
-   │          │          │          │
-   ▼          ▼          ▼          ▼
-┌──────┐ ┌──────┐ ┌────────┐ ┌──────────┐
-│.8:8090│ │.110  │ │.116:80 │ │.15 (SSH) │
-│RTX3090│ │:8090 │ │nginx   │ │Strix Halo│
-│gemma  │ │RTX5070│ │router  │ │ornith35B │
-└──────┘ │qwen27B│ │LiteLLM │ └──────────┘
+└──┬──────────┬──────────┬────────────────────────────────┘
+   │          │          │
+   ▼          ▼          ▼
+┌──────┐ ┌──────┐ ┌────────┐
+│.8:8080│ │.110  │ │.116:80 │
+│RTX3090│ │:8080 │ │nginx   │
+│gemma  │ │RTX5070│ │router  │
+└──────┘ │qwen27B│ │LiteLLM │
         └──────┘ │dashboard│
                  └────────┘
+
+Note: JSON sidecar exporters at :8090 were never deployed on any
+GPU host. Router falls back to GPU /health direct probe. Monitor
+should use router /health/unified as source of truth for GPU status.
+Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
+poll .15:8080 directly; must go through router on .116.
 ```

 ### Subsystems Polled

 | Subsystem | Endpoint | Frequency | Metrics |
 |-----------|----------|-----------|---------|
-| GPU Sidecar (RTX 3090) | `http://192.168.68.8:8090/health` | 15s | temp, VRAM, util, power, fan |
-| GPU Sidecar (RTX 5070) | `http://192.168.68.110:8090/health` | 15s | temp, VRAM, util, power, fan |
+| GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) |
 | Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
 | Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
 | LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
-| Strix Halo | SSH `192.168.68.15` + `:8080/health` | 15s | process status, CPU load, llama health |
+| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | ornith status via router — cannot poll .15:8080 directly (firewalled to .116 only) |
 | Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |

 ### Alert Delivery
@@ -61,7 +66,7 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and
 | GPU Temp | >80°C | >90°C |
 | VRAM Usage | >90% | >95% |
 | GPU Util | >95% | >98% |
-| Sidecar Unreachable | — | critical (connectivity) |
+| Sidecar Unreachable | — | info (sidecars not deployed — use router /health/unified) |
 | Model Down | — | critical (circuit breaker open) |

 ### JSON API Response Schema (/gpu-data)
@@ -119,10 +124,10 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).

 ## Execution

-1. **Poll sidecars** (every 15s): GET .8:8090/health, .110:8090/health
-2. **Poll router** (every 15s): GET .116/health/unified + /health via nginx:80
+1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback)
+2. **Poll router** (every 15s): GET .116/health via nginx:80
 3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
-4. **Poll Strix** (every 15s): SSH .15 for process check + GET .15:8080/health
+4. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only)
 5. **Poll dashboard** (every 15s): GET .116/dashboard/
 6. **Check alerts**: Compare metrics against thresholds
 7. **Compute summary**: Fleet-wide health aggregation
@@ -0,0 +1,150 @@
+---
+kind: function
+name: infrastructure-monitoring
+description: >
+  Deploys Prometheus + GPU exporters + Grafana to monitor the entire
+  inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
+  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
+  via existing /metrics Prometheus endpoint. Replaces the deprecated
+  harness-dashboard with a production-grade monitoring stack.
+version: 1.0.0
+---
+
+## Architecture
+
+```
+GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
+  nvidia-exporter       nvidia-exporter       amdgpu-exporter
+  :9400                 :9400                 :9400
+       │                     │                     │
+       └─────────────────────┼─────────────────────┘
+                             ▼
+              ┌──────────────────────────┐
+              │       Prometheus         │
+              │       CT 116 :9090       │
+              │                          │
+              │  Scrape targets:         │
+              │  • 192.168.68.8:9400     │
+              │  • 192.168.68.110:9400   │
+              │  • 192.168.68.15:9400    │
+              │  • litellm:4000/metrics  │
+              └──────────┬───────────────┘
+                         │
+              ┌──────────▼───────────────┐
+              │        Grafana           │
+              │       CT 116 :3001       │
+              │                          │
+              │  Preloaded dashboards:   │
+              │  • GPU Fleet Overview    │
+              │  • LiteLLM Proxy Stats   │
+              └──────────────────────────┘
+```
+
+## Components
+
+### 1. NVIDIA GPU Exporter (hosts: .8, .110)
+- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
+- Listens on `:9400`, exposes `/metrics` in Prometheus format
+- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed
+
+### 2. AMD GPU Exporter (host: .15)
+- Custom exporter: Python script wrapping `amdgpu_top --json`
+- Listens on `:9400`, exposes `/metrics` in Prometheus format
+- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
+- Runs as systemd service for persistence
+
+### 3. Prometheus (CT 116)
+- Container: `prom/prometheus:latest`
+- Port: `9090` (internal Docker network)
+- Scrape interval: 15s
+- Config: `/opt/monitoring/prometheus.yml`
+- Storage: Docker volume `prometheus-data`
+
+### 4. Grafana (CT 116)
+- Container: `grafana/grafana:latest`
+- Port: `3001` (mapped to host)
+- Data source: Prometheus at `http://prometheus:9090`
+- Provisioned dashboards for GPU fleet + LiteLLM
+- Accessible at `http://192.168.68.116:3001`
+
+## Parameters
+
+- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
+- gpu_amd_hosts: ["192.168.68.15"]
+- monitoring_host: "192.168.68.116"
+- prometheus_port: 9090
+- grafana_port: 3001
+- gpu_exporter_port: 9400
+
+## Requires
+
+- SSH access to all GPU hosts for exporter deployment
+- Docker on CT 116 for Prometheus + Grafana containers
+- Python 3 on AMD host for custom exporter
+- nvidia-smi on NVIDIA hosts
+
+## Maintains
+
+- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
+- Prometheus scrapes all targets every 15s
+- Grafana dashboards show real-time GPU utilization, temp, VRAM, power
+- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
+- Stack persists across reboots (systemd for exporters, Docker restart policy)
+
+## Execution
+
+### Phase 1: GPU Exporters
+
+**NVIDIA (.8 and .110)**:
+1. Download `nvidia_gpu_exporter` binary
+2. Create systemd service `nvidia-gpu-exporter.service`
+3. Start and enable
+
+**AMD (.15)**:
+1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
+2. Parses `amdgpu_top --json -d 1000` output
+3. Exposes key metrics at `:9400/metrics` via Python http.server
+4. Create systemd service
+5. Start and enable
+
+### Phase 2: Prometheus
+
+1. Create `/opt/monitoring/` directory on CT 116
+2. Write `prometheus.yml` with scrape configs for all targets
+3. Add to docker-compose (or separate compose file)
+4. Start container
+
+### Phase 3: Grafana
+
+1. Create `/opt/monitoring/grafana/` directories
+2. Provision Prometheus datasource
+3. Provision GPU fleet dashboard JSON
+4. Provision LiteLLM dashboard JSON
+5. Add to docker-compose
+6. Start container
+
+### Phase 4: Verification
+
+1. Verify all 3 GPU exporters return 200 at :9400/metrics
+2. Verify Prometheus targets all UP at :9090/targets
+3. Verify Grafana accessible at :3001 with dashboards
+4. Verify LiteLLM metrics flowing to Prometheus
+5. Update nginx to proxy `/monitoring/` → Grafana (optional)
+
+## Verification Commands
+
+```bash
+# GPU exporters
+curl -s http://192.168.68.8:9400/metrics | grep nvidia
+curl -s http://192.168.68.110:9400/metrics | grep nvidia
+curl -s http://192.168.68.15:9400/metrics | grep amdgpu
+
+# Prometheus
+curl -s http://192.168.68.116:9090/api/v1/targets
+
+# Grafana
+curl -s http://192.168.68.116:3001/api/health
+
+# LiteLLM metrics (already live)
+curl -s http://192.168.68.116:4001/metrics | head -20
+```
@@ -1,18 +1,44 @@
 ---
 kind: function
-name: check-litellm-health
+name: litellm-health
 description: >
-  Verifies LiteLLM deployment is healthy by checking admin UI, API docs,
-  OIDC auth endpoint, container status, and aggregate health on the
-  backend host. Designed as a reusable contract for any Syslog agent.
+  Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
+  nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
+  Router runs as internal-only backend behind LiteLLM. GPU monitoring via
+  Prometheus/Grafana and fleet dashboard.
+  Designed as a reusable contract for any Syslog agent.
 ---

+## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
+
+```
+Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
+                        │                    │
+                  Key validation      Model routing
+                  Fallback chains     Slot booking
+                  Complexity router   Circuit breakers
+                  Budget tracking     Redis-backed
+                        │                    │
+                  Prometheus ← metrics  ←───┘
+                        │
+                    Grafana :3001
+```
+
+**What changed (v3.1.0 → v3.2.0)**:
+- Router is BACK — runs internally on :9000 behind LiteLLM
+- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
+- Router :9000 is 127.0.0.1-only, not publicly accessible
+- GPU fleet dashboard: http://192.168.68.24:9100
+- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
+
 ## Parameters

 - public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
- backend_host: string — Internal CT host to check containers (default: "192.168.68.116")
+- backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
 - auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
- gpu_hosts: array — GPU inference hosts to check (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
+- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
+- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
+- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")

 ## Returns

@@ -24,60 +50,75 @@ description: >
 ## Requires

 - SSH key access to backend_host for container checks
- Network access to public_url and auth_host
- curl and openssl available on the execution host
-
-## Ensures
-
- Each check returns a clear pass/fail status with detail message
- If any endpoint returns non-200, overall_status is "degraded"
- If backend host unreachable or >2 containers down, overall_status is "down"
- If /health/unified reports any non-healthy components, status reflects it
- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy
+- Network access to public_url, auth_host, and gpu_dashboard_url
+- LiteLLM master key for key management endpoints

 ## GPU Fleet Topology

 | Host | IP | Hardware | Models Served | Engine |
 |------|-----|----------|---------------|--------|
-| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | gemma-4-12b (Dense) | llama-server Docker |
-| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | qwen3.6-27B-code (MoE), all Light tier | llama-server Docker |
-| amdpve | 192.168.68.15 | AMD Strix Halo (CPU-only, -ngl 0) | ornith-1.0-35b (35B) | llama-server bare-metal |
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
+| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |

-## Model Routing (LiteLLM → Router → GPU)
+## Model Fallback Chains (LiteLLM)

-| Model Request | Router Routes To |
-|--------------|-----------------|
-| `gemma-4-12b` | `GPU_DENSE_URL` → 192.168.68.8:8080 |
-| `qwen3.6-27B-code` | `GPU_MOE_URL` → 192.168.68.110:8080 |
-| `ornith-1.0-35b` | Router routes to 192.168.68.15:8080 |
-| `syslog-auto` | Auto-routed by router tier logic |
+| Primary | Timeout | Fallback | Timeout |
+|---------|---------|----------|---------|
+| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
+| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
+| ornith-1.0-35b | 120s | qwen → gemma | — |
+| syslog-auto (balanced) | — | qwen → gemma | — |
+
+## Containers on CT 116
+
+| Container | Image | Port | Health Check |
+|-----------|-------|------|-------------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
+| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
+| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
+| harness-redis | redis:7-alpine | :6379 | PING |
+| harness-dashboard | inference-harness-dashboard | :3000 | /health |
+| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
+| harness-prometheus | prom/prometheus | :9090 | /-/healthy |

 ## Execution

 1. **Read parameters** — Use provided values or defaults
+
 2. **Check public endpoints**:
   - GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
   - GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
-   - GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths)
-   - GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc")
-3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly):
-   - GET http://{{backend_host}}/litellm/ui/ → expect 200
-   - GET http://{{backend_host}}/litellm/docs → expect 200
-4. **Check aggregate health endpoint**:
-   - GET {{backend_host}}:9000/health/unified → expect 200, check all components
-5. **Check backend container health**:
-   - SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy
-   - Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard
-6. **Check GPU fleet health**:
-   - For each {{gpu_hosts}}:
-     - SSH or nvidia-smi check → verify GPU is reachable
-     - Check VRAM usage (warn if >90%)
-     - Check temperature (warn if >80°C)
-     - Verify model responds to a small test prompt
-7. **Check model inference** — Send test prompt to each model via LiteLLM API:
-   - POST /v1/chat/completions with model name → expect 200 + valid response
-   - Record token counts for baseline performance
-8. **Check OIDC auth endpoint**:
-   - Verify auth.sysloggh.net resolves to {{auth_host}}
-   - GET https://auth.sysloggh.net/ → expect login page
-9. **Compile and report** — Determine overall_status from individual check results
+
+3. **Check LiteLLM health (no-auth)**:
+   - GET http://{{backend_host}}/litellm/health/liveliness → expect 200
+
+4. **Check backend container health**:
+   - SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
+   - Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
+   - Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
+
+5. **Check router roster loaded**:
+   - GET http://{{backend_host}}:9000/health → expect 200
+   - GET http://{{backend_host}}:9000/health/unified → expect 3 models
+   - If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload
+
+6. **Check GPU fleet health** (via fleet dashboard):
+   - GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
+   - Verify GPUs reporting status "healthy"
+   - Check alerts array for active warnings/critical
+
+7. **Check model inference via LiteLLM** — Test each model:
+   - POST /v1/chat/completions model=gemma-4-12b → expect 200
+   - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
+   - POST /v1/chat/completions model=ornith-1.0-35b → expect 200
+   - Use master key for auth
+
+8. **Check agent keys**:
+   - GET /key/list with master key → verify all 6 agents have keys
+
+9. **Check Grafana**:
+   - GET {{grafana_url}}/api/health → expect 200
+
+10. **Compile and report** — Determine overall_status from individual check results
@@ -0,0 +1,115 @@
+---
+kind: responsibility
+name: proxmox-monitor
+description: >
+  Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
+  on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
+  exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
+  instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
+  (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
+  the layerdb). Dashboards exposed at http://192.168.68.116/grafana/.
+agent: abiba
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  CT 116 (syslog-api) — monitoring compose /opt/monitoring/   │
+│  Prometheus :9090  →  Grafana :3001 (direct LAN, 0.0.0.0)    │
+└───────┬──────────────┬──────────────┬───────────────────────┘
+        │              │              │
+        ▼              ▼              ▼
+   pve-exporter    docker-stats    (scrapes 5x node_exporter)
+   :9221           :9324
+        │              │
+        ▼              ▼
+   PVE API         Docker API
+   (amdpve .15,    (unix socket,
+    cluster-wide)  10 containers)
+```
+
+## Exporters
+
+| Exporter | Host:Port | Scope | Notes |
+|----------|-----------|-------|-------|
+| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
+| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
+| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
+
+## PVE API Token
+
+- User: `monitoring@pve` (cluster-replicated)
+- Role: `PVEAuditor` on `/` (read-only, whole cluster)
+- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c`
+- `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
+
+## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
+
+| UID | Title | Panels | Source |
+|-----|-------|--------|--------|
+| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
+| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
+| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
+| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
+
+- Dashboards built by `/opt/monitoring/grafana/dashboards/build-dashboards.py` → JSON in `.../dashboards/json/`
+- Provider config: `/opt/monitoring/grafana/dashboards/dashboards.yml`
+- Datasource: Prometheus uid `afqpgfay4g9hce` (provisioned, `/opt/monitoring/grafana/datasources/prometheus.yml`)
+- Edit dashboards in build-dashboards.py + re-run; `allowUiUpdates: true` for ad-hoc UI tweaks
+
+## Access
+
+- **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
+- **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
+- **Credentials**: admin / syslog-grafana-2026
+- Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
+- grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
+| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) |
+| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
+| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
+| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
+| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
+| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
+| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
+
+## Cluster "Tabiri" — 5 Nodes
+
+| Node | IP | Role |
+|------|----|----|
+| ocupve | 192.168.68.5 | PVE |
+| storepve | 192.168.68.6 | PVE |
+| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
+| minipve | 192.168.68.12 | PVE |
+| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
+
+## Operations
+
+### view-dashboards
+Open `http://192.168.68.116/grafana/` → "Syslog Fleet" folder
+
+### add-dashboard
+Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
+
+### check-targets
+`curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'`
+
+### restart-exporter
+`cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
+
+### rotate-pve-token
+`pveum user token add monitoring@pve prometheus` on any PVE node → update `/opt/monitoring/pve.yml` → `docker compose restart pve-exporter`
+
+## Known Issues & Notes
+
+- **cAdvisor abandoned**: v0.51 can't resolve Docker 29 containerd image-store layerdb (`/var/lib/docker/image/` only has `identity-cache.db`). Returns 0 named containers. Replaced by custom docker-stats-exporter.
+- **Docker native metrics** (`/etc/docker/daemon.json` `metrics-addr: 127.0.0.1:9323`, experimental:true) enabled but only gives `engine_daemon_*` (daemon-level), not per-container. Kept for daemon health.
+- **PVE exporter metric schema**: NOT name-prefixed. `pve_cpu_usage_ratio`, `pve_memory_usage_bytes`, `pve_disk_usage_bytes`, `pve_uptime_seconds` are GUEST-level only (24 series, `id=lxc/100` etc). Node-level host metrics come from node_exporter. Storage pool usage: `pve_storage_info` (info only, no usage bytes — use node_filesystem_* for actual disk usage).
+- **grafana piechart plugin removed** from `GF_INSTALL_PLUGINS` (Angular, unsupported in Grafana 13).
+- **Single pve-exporter points at amdpve .15** — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.
Author	SHA1	Message	Date
Abiba	bb20637b32	fix: align contract name fields to filenames (litellm-health, infrastructure-monitoring) - litellm-health: 'check-litellm-health' -> 'litellm-health' - infrastructure-monitoring: 'deploy-monitoring-stack' -> 'infrastructure-monitoring' Fixes 'prose run <filename>' mismatch. No external references broken.	2026-07-02 20:35:44 +00:00
root	b15771bfd9	prose: add Strix Halo thermal safeguard (-n 8192 cap) to gpu-fleet Runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (9°C past critical) and throttling 70->29 t/s. Server-side -n 8192 hard cap now bounds generation. Documented connection source (all from .116/LiteLLM) and the thermal constraint (fanless APU).	2026-07-02 03:51:36 +00:00
root	3700b3bf46	prose: fix proxmox-monitor access to direct :3001 (revert sub-path) Revert the /grafana/ nginx sub-path I wrongly added — it broke the existing http://192.168.68.116:3001/d/gpu-fleet URL. Restored direct 0.0.0.0:3001 LAN access. Contract is now the source of truth: no GF_SERVER_SERVE_FROM_SUB_PATH, no nginx /grafana/ route. gpu-fleet + 3 new dashboards all at :3001.	2026-07-02 02:56:27 +00:00
root	4df2fa8329	prose: add proxmox-monitor contract (Grafana Pulse replacement) - New proxmox-monitor.prose.md: pve-exporter + node_exporter(5) + docker-stats-exporter - Replaces Pulse with file-provisioned Grafana dashboards (cluster/node/docker) - Documents cAdvisor/Docker-29 incompatibility + custom exporter workaround - PVE API token, access URL, config file map, operations	2026-07-02 02:50:59 +00:00
root	a07ff2d96a	prose: align all contracts with Vulkan rebuild + Koby key regen - gpu-fleet: Koby key updated (sk-lDTsWy7H3T...), config table (ornith service + masked services), benchmarks (prompt 209→532 tok/s), ROCm→Vulkan status - gpu-monitor: topology (sidecars :8090 removed, .15 via router), subsystem polling (router /health/unified as source of truth), execution steps, alert threshold (sidecar-unreachable downgraded from critical)	2026-07-02 00:29:10 +00:00
root	49e8d968f5	gpu-fleet: Strix Halo Vulkan rebuild + LiteLLM alignment (2026-07-01) - ornith now on Vulkan build 4fc4ec5, 256K ctx, ~70 tok/s - Fixed GPU_MOE_URL bug in docker-compose (.110→.15) - Added router sidecar fallback for health-checks - Updated roster context 4096→262144 + correct model_path - Documented port 8080 firewall (restricted to .116)	2026-07-02 00:01:56 +00:00