fix: correct IPs, CT IDs, and key status across all contracts
hermes-key-enforcement.prose.md: - Add IP column to verified agents table - Fix Tanko status: was using master key via systemd drop-in (not compliant) - Add detection query step 2: check systemd drop-ins for master key leaks - Add step 3: verify running process env against dedicated key - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only) gpu-fleet.prose.md: - Add CT ID column to agent table for cross-reference - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host) - Update Abiba LiteLLM key to match actual models.json key - Set Koonimo IP to unknown (was .114 — incorrect per user) - Fix pi-specific paths .24 → .65 hermes-config-template.prose.md: - Set Koonimo IP to unknown infrastructure-control.prose.md: - Set Koonimo IP to unknown - Fix Koonimo CT reference
This commit is contained in:
+12
-12
@@ -124,14 +124,14 @@ Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, k
|
|||||||
|
|
||||||
## Agent Keys (LiteLLM DB — Current 2026-06-30)
|
## Agent Keys (LiteLLM DB — Current 2026-06-30)
|
||||||
|
|
||||||
| Agent | Key | Host | Access |
|
| Agent | CT | IP | Key | Access |
|
||||||
|-------|-----|------|--------|
|
|-------|-----|-----|-----|--------|
|
||||||
| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
|
| Tanko | 112 | .122 | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | SSH jerome |
|
||||||
| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
|
| Mumuni | 114 | .123 | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | SSH root |
|
||||||
| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
|
| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
|
||||||
| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
|
| Koby | 111 | ? | `sk-lDTsWy7H3T19UwUFEN6JSg` | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
|
||||||
| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
|
| Koonimo | 113 | ? | `sk-Prx_vRXYb2VEzDmhPCbNeg` | no SSH |
|
||||||
| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |
|
| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
|
||||||
|
|
||||||
**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
|
**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
|
||||||
If no SSH access, send Zulip DM via abiba-bot.
|
If no SSH access, send Zulip DM via abiba-bot.
|
||||||
@@ -145,10 +145,10 @@ If no SSH access, send Zulip DM via abiba-bot.
|
|||||||
| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
|
| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
|
||||||
| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
|
| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
|
||||||
| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
|
| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
|
||||||
| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
|
| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
|
||||||
| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
|
| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
|
||||||
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
|
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
|
||||||
| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
|
| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
|
||||||
| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
|
| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
|
||||||
| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
|
| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
|
||||||
|
|
||||||
|
|||||||
@@ -30,7 +30,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
|
|||||||
| Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
|
| Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
|
||||||
| Abiba | `abiba-*` | 192.168.68.24 | local | — |
|
| Abiba | `abiba-*` | 192.168.68.24 | local | — |
|
||||||
| Koby | `koby-*` | ? | Zulip | — |
|
| Koby | `koby-*` | ? | Zulip | — |
|
||||||
| Koonimo | `koonimo-*` | 192.168.68.114 | Zulip | — |
|
| Koonimo | `koonimo-*` | ? | Zulip | — |
|
||||||
| Kagenz0 | `kagenz0-*` | ? | Zulip | — |
|
| Kagenz0 | `kagenz0-*` | ? | Zulip | — |
|
||||||
|
|
||||||
✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
|
✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
|
||||||
|
|||||||
@@ -74,12 +74,21 @@ model:
|
|||||||
Run on any Hermes host to detect violations:
|
Run on any Hermes host to detect violations:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
grep -rn 'api_key: sk-' /root/.hermes/ \
|
# 1. Check config.yaml for hardcoded harness keys
|
||||||
|
grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
|
||||||
--include='config.yaml' \
|
--include='config.yaml' \
|
||||||
| grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
|
| grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
|
||||||
|
|
||||||
|
# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
|
||||||
|
grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
|
||||||
|
grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
|
||||||
|
|
||||||
|
# 3. Verify running process env matches dedicated key
|
||||||
|
cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
|
||||||
|
| tr '\0' '\n' | grep LITELLM_API_KEY
|
||||||
```
|
```
|
||||||
|
|
||||||
If any output — violation. Fix immediately.
|
If any output from step 2 — **critical violation** (master key leaked). Fix immediately.
|
||||||
|
|
||||||
## Rotation Procedure
|
## Rotation Procedure
|
||||||
|
|
||||||
@@ -117,16 +126,16 @@ litellm_settings:
|
|||||||
purpose: "agent-inference"
|
purpose: "agent-inference"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Verified Agents (2026-07-04 final)
|
## Verified Agents (2026-07-05 update)
|
||||||
|
|
||||||
| Agent | CT | Status | LiteLLM Alias | Key Type | Systemd Source | Last Verified |
|
| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
|
||||||
|-------|-----|--------|---------------|----------|----------------|---------------|
|
|-------|-----|-----|---------------|-----|--------|----------------|---------------|
|
||||||
| Tanko | 112 | ✅ Compliant | `tanko` | Dedicated | `/etc/environment` only | 23:00 EDT |
|
| Tanko | 112 | .122 | `tanko` | `sk-620a05e95a-e93d...` | ✅ Fixed | User drop-in `env.conf` | 19:44 UTC Jul 5 |
|
||||||
| Mumuni | 114 | ✅ Compliant | `mumuni` | Dedicated | `/etc/environment` only | 23:00 EDT |
|
| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
|
||||||
| Koby | 111 | ✅ Compliant | `koby` | Dedicated | User service + drop-in | 23:00 EDT |
|
| Koby | 111 | ? | `koby` | `sk-lDTsWy7H3T19UwUFEN6JSg` | ⚠️ Unverified | User service + drop-in | 23:00 EDT Jul 4 |
|
||||||
| Koonimo | 113 | ✅ Compliant | `koonimo` | Dedicated | System service + drop-in | 23:00 EDT |
|
| Koonimo | 113 | ? | `koonimo` | `sk-Prx_vRXYb2VEzDmhPCbNeg` | ⚠️ Unverified | System service + drop-in | 23:00 EDT Jul 4 |
|
||||||
| Abiba | 100 | ✅ N/A (pi native) | — | — | — | 23:00 EDT |
|
| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
|
||||||
| Kagenz0 | 105 | ❌ DOWN | — | — | — | 19:14 EDT |
|
| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
|
||||||
|
|
||||||
### Systemd Service Pattern (2026-07-04 fix)
|
### Systemd Service Pattern (2026-07-04 fix)
|
||||||
|
|
||||||
|
|||||||
@@ -68,7 +68,7 @@ description: >
|
|||||||
| CT 116 (syslog-api) | SSH root | SSH key | ✅ |
|
| CT 116 (syslog-api) | SSH root | SSH key | ✅ |
|
||||||
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
|
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
|
||||||
| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
|
| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
|
||||||
| Koonimo CT (.114) | SSH jerome | ❌ no key access |
|
| Koonimo CT (113) | SSH jerome | ❌ no key access |
|
||||||
| Netbird (.17) | SSH root | SSH key | ✅ |
|
| Netbird (.17) | SSH root | SSH key | ✅ |
|
||||||
| Gitea | API token | abiba-bot token | ✅ |
|
| Gitea | API token | abiba-bot token | ✅ |
|
||||||
| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
|
| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
|
||||||
@@ -585,7 +585,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
|
|||||||
| 110 | gitea | minipve | — | Git | ❌ |
|
| 110 | gitea | minipve | — | Git | ❌ |
|
||||||
| 111 | tdunna | amdpve | — | ? | ❌ |
|
| 111 | tdunna | amdpve | — | ? | ❌ |
|
||||||
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
|
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
|
||||||
| 113 | baggy/koonimo | amdpve | .114 | Hermes agent | ✅ |
|
| 113 | baggy/koonimo | amdpve | ? | Hermes agent | ✅ |
|
||||||
| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
|
| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
|
||||||
| 115 | scottdenya | amdpve | — | ? | ❌ |
|
| 115 | scottdenya | amdpve | — | ? | ❌ |
|
||||||
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
||||||
|
|||||||
Reference in New Issue
Block a user