From 0db95a4d1cb8f44aed60a3e259036b6ccd529eba Mon Sep 17 00:00:00 2001 From: root Date: Sun, 5 Jul 2026 19:58:14 +0000 Subject: [PATCH] fix: correct IPs, CT IDs, and key status across all contracts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hermes-key-enforcement.prose.md: - Add IP column to verified agents table - Fix Tanko status: was using master key via systemd drop-in (not compliant) - Add detection query step 2: check systemd drop-ins for master key leaks - Add step 3: verify running process env against dedicated key - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only) gpu-fleet.prose.md: - Add CT ID column to agent table for cross-reference - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host) - Update Abiba LiteLLM key to match actual models.json key - Set Koonimo IP to unknown (was .114 — incorrect per user) - Fix pi-specific paths .24 → .65 hermes-config-template.prose.md: - Set Koonimo IP to unknown infrastructure-control.prose.md: - Set Koonimo IP to unknown - Fix Koonimo CT reference --- gpu-fleet.prose.md | 24 ++++++++++++------------ hermes-config-template.prose.md | 2 +- hermes-key-enforcement.prose.md | 31 ++++++++++++++++++++----------- infrastructure-control.prose.md | 4 ++-- 4 files changed, 35 insertions(+), 26 deletions(-) diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 0e78a5f..735280f 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -124,14 +124,14 @@ Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, k ## Agent Keys (LiteLLM DB — Current 2026-06-30) -| Agent | Key | Host | Access | -|-------|-----|------|--------| -| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome | -| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root | -| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local | -| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) | -| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH | -| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH | +| Agent | CT | IP | Key | Access | +|-------|-----|-----|-----|--------| +| Tanko | 112 | .122 | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | SSH jerome | +| Mumuni | 114 | .123 | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | SSH root | +| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) | +| Koby | 111 | ? | `sk-lDTsWy7H3T19UwUFEN6JSg` | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) | +| Koonimo | 113 | ? | `sk-Prx_vRXYb2VEzDmhPCbNeg` | no SSH | +| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH | **Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes. If no SSH access, send Zulip DM via abiba-bot. @@ -145,10 +145,10 @@ If no SSH access, send Zulip DM via abiba-bot. | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) | | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health | | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) | -| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) | -| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) | -| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server | -| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard | +| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) | +| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) | +| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server | +| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard | | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) | | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. | diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 543f14e..89e9918 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -30,7 +30,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed | Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ | | Abiba | `abiba-*` | 192.168.68.24 | local | — | | Koby | `koby-*` | ? | Zulip | — | -| Koonimo | `koonimo-*` | 192.168.68.114 | Zulip | — | +| Koonimo | `koonimo-*` | ? | Zulip | — | | Kagenz0 | `kagenz0-*` | ? | Zulip | — | ✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research, diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index 67bc906..3b2b91b 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -74,12 +74,21 @@ model: Run on any Hermes host to detect violations: ```bash -grep -rn 'api_key: sk-' /root/.hermes/ \ +# 1. Check config.yaml for hardcoded harness keys +grep -rn 'api_key: sk-' /home/jerome/.hermes/ \ --include='config.yaml' \ | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK' + +# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this) +grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null +grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null + +# 3. Verify running process env matches dedicated key +cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \ + | tr '\0' '\n' | grep LITELLM_API_KEY ``` -If any output — violation. Fix immediately. +If any output from step 2 — **critical violation** (master key leaked). Fix immediately. ## Rotation Procedure @@ -117,16 +126,16 @@ litellm_settings: purpose: "agent-inference" ``` -## Verified Agents (2026-07-04 final) +## Verified Agents (2026-07-05 update) -| Agent | CT | Status | LiteLLM Alias | Key Type | Systemd Source | Last Verified | -|-------|-----|--------|---------------|----------|----------------|---------------| -| Tanko | 112 | ✅ Compliant | `tanko` | Dedicated | `/etc/environment` only | 23:00 EDT | -| Mumuni | 114 | ✅ Compliant | `mumuni` | Dedicated | `/etc/environment` only | 23:00 EDT | -| Koby | 111 | ✅ Compliant | `koby` | Dedicated | User service + drop-in | 23:00 EDT | -| Koonimo | 113 | ✅ Compliant | `koonimo` | Dedicated | System service + drop-in | 23:00 EDT | -| Abiba | 100 | ✅ N/A (pi native) | — | — | — | 23:00 EDT | -| Kagenz0 | 105 | ❌ DOWN | — | — | — | 19:14 EDT | +| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified | +|-------|-----|-----|---------------|-----|--------|----------------|---------------| +| Tanko | 112 | .122 | `tanko` | `sk-620a05e95a-e93d...` | ✅ Fixed | User drop-in `env.conf` | 19:44 UTC Jul 5 | +| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 | +| Koby | 111 | ? | `koby` | `sk-lDTsWy7H3T19UwUFEN6JSg` | ⚠️ Unverified | User service + drop-in | 23:00 EDT Jul 4 | +| Koonimo | 113 | ? | `koonimo` | `sk-Prx_vRXYb2VEzDmhPCbNeg` | ⚠️ Unverified | System service + drop-in | 23:00 EDT Jul 4 | +| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 | +| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 | ### Systemd Service Pattern (2026-07-04 fix) diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index ab935fd..0aa8d09 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -68,7 +68,7 @@ description: > | CT 116 (syslog-api) | SSH root | SSH key | ✅ | | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ | | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ | -| Koonimo CT (.114) | SSH jerome | ❌ no key access | +| Koonimo CT (113) | SSH jerome | ❌ no key access | | Netbird (.17) | SSH root | SSH key | ✅ | | Gitea | API token | abiba-bot token | ✅ | | Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ | @@ -585,7 +585,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server" | 110 | gitea | minipve | — | Git | ❌ | | 111 | tdunna | amdpve | — | ? | ❌ | | 112 | tanko | amdpve | .122 | Hermes agent | ✅ | -| 113 | baggy/koonimo | amdpve | .114 | Hermes agent | ✅ | +| 113 | baggy/koonimo | amdpve | ? | Hermes agent | ✅ | | 114 | mumuni | minipve | .123 | Hermes agent | ✅ | | 115 | scottdenya | amdpve | — | ? | ❌ | | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |