docs: lessons learned from 2026-07-12 session

- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement

infrastructure-control.prose.md

@@ -62,19 +62,19 @@ description: >
 
 | Resource | Auth Method | Credential Source | Status |
 |----------|------------|-------------------|--------|
-| Proxmox Cluster | PVE API Token | `monitoring@pve!mumuni=...` | ✅ |
-| Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ |
+| Proxmox Cluster | PVE API Token | Infisical vault (`PROXMOX_API_TOKEN`) | ✅ |
+| Proxmox Root | Password via API ticket | Infisical vault (`PROXMOX_ROOT_PASSWORD`) | ✅ |
 | docker-vm (.7) | SSH root | SSH key | ✅ |
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
 | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
 | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
 | Baggy CT (113) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
-| Gitea | API token | abiba-bot token | ✅ |
-| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
+| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
+| Zulip | Bot API key | Infisical vault (`ZULIP_BOT_KEY`) | ✅ |
 | RA-H OS | MCP bridge | port 3100 | ✅ |
-| LiteLLM Admin | master key | `sk-litellm-7f96...` | ✅ VERIFY-BEFORE-USE |
-| Grafana | admin password | `syslog-grafana-2026` | ✅ VERIFY-BEFORE-USE |
+| LiteLLM Admin | master key | Infisical vault (`LITELLM_MASTER_KEY`) | ✅ VERIFY-BEFORE-USE |
+| Grafana | admin password | Infisical vault (`GRAFANA_ADMIN_PASSWORD`) | ✅ VERIFY-BEFORE-USE |
 
 > **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this
 > contract are live-state fields. Test them against the live system before
@@ -550,13 +550,13 @@ curl -s http://192.168.68.116/health/unified | jq .status
 curl -s http://192.168.68.24:9100/gpu-data | jq .summary
 
 # Grafana status
-curl -s http://admin:syslog-grafana-2026@192.168.68.116:3001/api/health
+curl -s http://admin:$(infisical secrets get GRAFANA_ADMIN_PASSWORD --project=infrastructure --env=production --plain)@192.168.68.116:3001/api/health
 
 # Prometheus targets
 curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
 
 # LiteLLM key check
-curl -s -H "Authorization: Bearer sk-litellm-7f96080dd99b15c36bd4b333b58a6796" \
+curl -s -H "Authorization: Bearer $(infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production --plain)" \
   http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'
 
 # Storage check
@@ -586,7 +586,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 108 | media | storepve | — | Media | ❌ |
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
 | 110 | gitea | minipve | — | Git | ❌ |
-| 111 | tdunna | amdpve | — | ? | ❌ |
+| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
 | 113 | baggy | amdpve | ? | Hermes agent | ✅ |
 | 114 | mumuni | minipve | .123 | Hermes agent | ✅ |