docs: lessons learned from 2026-07-12 session
- gpu-fleet: Corrected architecture (direct GPU, no router in path). Updated context values (RTX 3090=256K, not 128K). Added api-key standardization requirement. - gpu-self-heal: Added Lessons Learned section with 5 critical findings: L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed) L2: Fallback chain cascading failure loop detection L3: Verify running state, not documentation L4: Infisical fallback requirement (.env must have uncommented key) L5: Zulip event queue can silently die after ~40 reconnects - litellm-self-heal: Updated status manual-only→deployed, cron schedule - litellm-api-keys: Added Infisical token expiry warning + .env fallback - hermes-config-template: Rule 3 updated with .env fallback requirement
This commit is contained in:
@@ -41,7 +41,7 @@ agent: abiba
|
||||
|
||||
- User: `monitoring@pve` (cluster-replicated)
|
||||
- Role: `PVEAuditor` on `/` (read-only, whole cluster)
|
||||
- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c`
|
||||
- Token: `monitoring@pve!prometheus` — stored in Infisical vault (`PROXMOX_MONITOR_TOKEN`)
|
||||
- `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
|
||||
|
||||
## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
|
||||
@@ -62,7 +62,7 @@ agent: abiba
|
||||
|
||||
- **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
|
||||
- **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
|
||||
- **Credentials**: admin / syslog-grafana-2026
|
||||
- **Credentials**: admin / password stored in Infisical vault (`GRAFANA_ADMIN_PASSWORD`)
|
||||
- Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
|
||||
- grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user