fix: correct IPs, CT IDs, and key status across all contracts
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped

hermes-key-enforcement.prose.md:
  - Add IP column to verified agents table
  - Fix Tanko status: was using master key via systemd drop-in (not compliant)
  - Add detection query step 2: check systemd drop-ins for master key leaks
  - Add step 3: verify running process env against dedicated key
  - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)

gpu-fleet.prose.md:
  - Add CT ID column to agent table for cross-reference
  - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
  - Update Abiba LiteLLM key to match actual models.json key
  - Set Koonimo IP to unknown (was .114 — incorrect per user)
  - Fix pi-specific paths .24 → .65

hermes-config-template.prose.md:
  - Set Koonimo IP to unknown

infrastructure-control.prose.md:
  - Set Koonimo IP to unknown
  - Fix Koonimo CT reference
This commit is contained in:
root
2026-07-05 19:58:14 +00:00
parent 2d2259035a
commit 0db95a4d1c
4 changed files with 35 additions and 26 deletions
+12 -12
View File
@@ -124,14 +124,14 @@ Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, k
## Agent Keys (LiteLLM DB — Current 2026-06-30)
| Agent | Key | Host | Access |
|-------|-----|------|--------|
| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |
| Agent | CT | IP | Key | Access |
|-------|-----|-----|-----|--------|
| Tanko | 112 | .122 | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | SSH jerome |
| Mumuni | 114 | .123 | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | SSH root |
| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
| Koby | 111 | ? | `sk-lDTsWy7H3T19UwUFEN6JSg` | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
| Koonimo | 113 | ? | `sk-Prx_vRXYb2VEzDmhPCbNeg` | no SSH |
| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
**Key update procedure**: Update `/etc/environment``LITELLM_API_KEY=sk-...` → restart Hermes.
If no SSH access, send Zulip DM via abiba-bot.
@@ -145,10 +145,10 @@ If no SSH access, send Zulip DM via abiba-bot.
| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
+1 -1
View File
@@ -30,7 +30,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
| Abiba | `abiba-*` | 192.168.68.24 | local | — |
| Koby | `koby-*` | ? | Zulip | — |
| Koonimo | `koonimo-*` | 192.168.68.114 | Zulip | — |
| Koonimo | `koonimo-*` | ? | Zulip | — |
| Kagenz0 | `kagenz0-*` | ? | Zulip | — |
✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
+20 -11
View File
@@ -74,12 +74,21 @@ model:
Run on any Hermes host to detect violations:
```bash
grep -rn 'api_key: sk-' /root/.hermes/ \
# 1. Check config.yaml for hardcoded harness keys
grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
--include='config.yaml' \
| grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
# 3. Verify running process env matches dedicated key
cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
| tr '\0' '\n' | grep LITELLM_API_KEY
```
If any output — violation. Fix immediately.
If any output from step 2 — **critical violation** (master key leaked). Fix immediately.
## Rotation Procedure
@@ -117,16 +126,16 @@ litellm_settings:
purpose: "agent-inference"
```
## Verified Agents (2026-07-04 final)
## Verified Agents (2026-07-05 update)
| Agent | CT | Status | LiteLLM Alias | Key Type | Systemd Source | Last Verified |
|-------|-----|--------|---------------|----------|----------------|---------------|
| Tanko | 112 | ✅ Compliant | `tanko` | Dedicated | `/etc/environment` only | 23:00 EDT |
| Mumuni | 114 | ✅ Compliant | `mumuni` | Dedicated | `/etc/environment` only | 23:00 EDT |
| Koby | 111 | ✅ Compliant | `koby` | Dedicated | User service + drop-in | 23:00 EDT |
| Koonimo | 113 | ✅ Compliant | `koonimo` | Dedicated | System service + drop-in | 23:00 EDT |
| Abiba | 100 | ✅ N/A (pi native) | — | — | — | 23:00 EDT |
| Kagenz0 | 105 | ❌ DOWN | — | — | — | 19:14 EDT |
| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
|-------|-----|-----|---------------|-----|--------|----------------|---------------|
| Tanko | 112 | .122 | `tanko` | `sk-620a05e95a-e93d...` | ✅ Fixed | User drop-in `env.conf` | 19:44 UTC Jul 5 |
| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
| Koby | 111 | ? | `koby` | `sk-lDTsWy7H3T19UwUFEN6JSg` | ⚠️ Unverified | User service + drop-in | 23:00 EDT Jul 4 |
| Koonimo | 113 | ? | `koonimo` | `sk-Prx_vRXYb2VEzDmhPCbNeg` | ⚠️ Unverified | System service + drop-in | 23:00 EDT Jul 4 |
| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
### Systemd Service Pattern (2026-07-04 fix)
+2 -2
View File
@@ -68,7 +68,7 @@ description: >
| CT 116 (syslog-api) | SSH root | SSH key | ✅ |
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
| Koonimo CT (.114) | SSH jerome | ❌ no key access |
| Koonimo CT (113) | SSH jerome | ❌ no key access |
| Netbird (.17) | SSH root | SSH key | ✅ |
| Gitea | API token | abiba-bot token | ✅ |
| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
@@ -585,7 +585,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| 110 | gitea | minipve | — | Git | ❌ |
| 111 | tdunna | amdpve | — | ? | ❌ |
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
| 113 | baggy/koonimo | amdpve | .114 | Hermes agent | ✅ |
| 113 | baggy/koonimo | amdpve | ? | Hermes agent | ✅ |
| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
| 115 | scottdenya | amdpve | — | ? | ❌ |
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |