diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 5d91c55..a76a35e 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -5,7 +5,13 @@ description: > Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. - Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online. + Updated 2026-07-03 (19:14 EDT): Full sweep — Koby (11 fixes), Mumuni (1 fix), + Koonimo (3 fixes). All agents now CLEAN: zero model:auto, zero deprecated + sk-syslog-* keys. Tanko has hardcoded tanko-v2 key (correct, but should use + api_key_env for rotation safety). Kagenz0 is DOWN (no config, no key). + Previous: full key audit — mumuni-v3 + context7-mcp registered, + /etc/environment aligned, double-B8nw incident documented, all 17 LiteLLM + keys mapped with active/inactive status. --- ## Maintains @@ -16,15 +22,90 @@ description: > - agent_keys: managed in LiteLLM DB — current aliases in Key Management section below - infra_endpoints_verified: array +### Agent topology + +| Agent | CT | IP | Key Alias | Startup | Zulip Owner | +|-------|-----|-----|-----------|---------|-------------| +| Tanko | 112 | 192.168.68.122 | `tanko-v3` | systemd user + `.env` | Jerome only ✅ | +| Mumuni | 114 | 192.168.68.123 | `mumuni-v3` | systemd user + `.env` | Jerome only ✅ | +| Koonimo | 113 | 192.168.68.114 | `koonimo-v2` | systemd (Discord/Telegram) | N/A | +| Koby | 111 | 192.168.68.129 | `koby-v3` | systemd user (Discord/Telegram) | N/A | +| Abiba | — | 192.168.68.24 | `abiba` | `pi` harness | N/A | +| Kagenz0 | ? | ? | `kagenz0` | unknown | not verified | + +### Zulip ownership rules (ALL agents) + +**Every Zulip-enabled agent MUST only respond to its owner.** This is enforced +via two env vars in the agent's `.env` file: + +```bash +ZULIP_ALLOW_ALL_USERS=false +ZULIP_ALLOWED_USERS=jerome@sysloggh.com +``` + +- `ZULIP_ALLOW_ALL_USERS=false` — disables the default open-to-everyone mode +- `ZULIP_ALLOWED_USERS=jerome@sysloggh.com` — only Jerome can trigger responses + +This applies regardless of whether the agent uses `.env` directly (Mumuni) +or a systemd drop-in (Tanko). Both mechanisms work — the key is that the +env vars reach the Hermes process. + ## Key Management **Source of truth:** LiteLLM PostgreSQL DB. Plaintext keys live ONLY in `/etc/environment` on each agent host, never hardcoded in contracts or config.yaml. This contract defines the procedure; it does NOT store plaintext keys. -### Current key aliases (2026-07-02 rotation) -- `tanko-v2`, `mumuni-v2`, `abiba-v2`, `koby-v2`, `koonimo-v2`, `kagenz0-v2` -- `synthetic-monitor` (daily infra report) +### Current key aliases (2026-07-03 audit) + +**Active keys (by agent):** + +| Agent | Alias | Key (last 8) | Source | Status | +|-------|-------|-------------|--------|--------| +| Tanko | `tanko-v3` | ...JN2OsJw | Process env | Active (needs restart → v2) | +| Tanko | `tanko-v2` | ...1ka8JA | `/etc/environment` | Registered, idle | +| Mumuni | `mumuni-v3` | ...OiWU_B8nw | Process + /etc/env | ✅ Aligned | +| Koonimo | `koonimo-v3` | ...A_wIOA | Config (hardcoded) | Active | +| Koonimo | `koonimo-v2` | ...R7FAobA | `/etc/environment` | Registered, idle | +| Koby | `koby-v3` | ...P4DrWw | `/etc/environment` | ✅ Registered | +| Koby | `koby-hermes` | ...syslog-koby | Config (deprecated) | Active legacy | +| Abiba | `abiba` | ...aLgMw | pi config | ✅ Active | +| Kagenz0 | `kagenz0` | ...Uggg | unknown | Last active Jul 1 | +| Context7 | `context7-mcp` | ...ZPh6VA | Process env | ✅ Registered | + +**Unused registrations:** `abiba-v2`, `kagenz0-v2`, `koonimo-v2`, `synthetic-monitor`, `UI Dashboard`, unnamed key + +> **2026-07-03 alignment**: Tanko/Koonimo configs updated to use `api_key_env: +> LITELLM_API_KEY` (was hardcoded `sk-syslog-*`). Base URLs corrected to +> `/v1` (was `/litellm/v1`). All active keys now registered in LiteLLM. +> Tanko and Koonimo need Hermes gateway restart for config changes to take +> effect. + +### Key lineage + +| Agent | Alias | Key (last 8) | Status | Date | +|-------|-------|-------------|--------|------| +| Tanko | `tanko-v3` | ...JN2OsJw | **active** (process) | 2026-07-03 | +| Tanko | `tanko-v2` | ...1ka8JA | idle (in /etc/env) | 2026-07-02 | +| Mumuni | `mumuni-v3` | ...OiWU_B8nw | **active** | 2026-07-03 | +| Mumuni | `mumuni-v2` | ...HcnuOPtQ | superseded | 2026-07-02 | +| Mumuni | `mumuni` | ...62da807e | superseded | 2026-06-30 | +| Koonimo | `koonimo-v3` | ...A_wIOA | **active** (hardcoded) | 2026-07-03 | +| Koonimo | `koonimo-v2` | ...R7FAobA | idle (in /etc/env) | 2026-07-02 | +| Koby | `koby-v3` | ...P4DrWw | **active** (/etc/env) | 2026-07-03 | +| Koby | `koby-hermes` | ...syslog-koby | legacy (in config) | 2026-06-30 | + +> **2026-07-03 incidents**: +> - Mumuni: double-B8nw — process had `...POiWU_B8nw`, /etc/environment +> had `...VDtHWd-XiWU_B8nw`. Wrong one registered first. +> - Context7 MCP: `sk-XY2aUfvy2BIs6kp1ZPh6VA` was never registered — the +> original error Kwame reported. +> - Tanko: process key `sk-3ZsdWJbbNSo9zSnJN2OsJw` unregistered. Config +> used deprecated `sk-syslog-tanko`. Both fixed. +> - Koonimo: hardcoded `sk-KFnE3Td6aKEBZxwVA_wIOA` unregistered. Base URL +> used old `/litellm/v1` path. Both fixed. +> - Koby: `/etc/environment` key `sk-x5BnZUUKvz3r8EdLP4DrWw` unregistered. +> Config used deprecated `sk-syslog-koby`. Fixed. ### Verify: all keys working ```bash @@ -39,7 +120,7 @@ done ### Rotate: generate new key, deploy, verify ```bash -AGENT=mumuni && ALIAS="${AGENT}-v2" +AGENT=mumuni && ALIAS="${AGENT}-v3" # 1. Generate new key in LiteLLM (master key required) KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \ @@ -60,11 +141,72 @@ ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f ### Key deployment rules 1. **Plaintext keys live ONLY in `/etc/environment`** — never in a contract or config.yaml 2. **All configs use `api_key_env: LITELLM_API_KEY`** — indirection prevents staleness -3. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access -4. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly -5. **After rotating any key:** restart the corresponding agent gateway -6. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login -7. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid) +3. **`custom_providers[].api_key` MUST use `api_key_env`** — hardcoded keys shadow env vars and bypass rotation +4. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access +5. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly +6. **After rotating any key:** restart the corresponding agent gateway +7. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login +8. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid) + +### Model name enforcement (2026-07-03) + +**CRITICAL**: Hermes `model: auto` in any config section gets sent to LiteLLM +as the literal model name "auto", which LiteLLM rejects with 403 because +the registered model is `syslog-auto`, not `auto`. + +**Must-fix pattern** — any auxiliary section with `model: auto`: +```yaml +# ❌ BROKEN — sends model "auto" to LiteLLM (403 Forbidden) +auxiliary: + approval: + model: auto + provider: auto + skills_hub: + model: auto + provider: auto + +# ✅ FIXED — registered model name +auxiliary: + approval: + model: syslog-auto + provider: harness + api_key_env: LITELLM_API_KEY + base_url: http://192.168.68.116/v1 +``` + +**Detection**: `grep -rn "model: auto" /root/.hermes/config.yaml` on any +Hermes host — if any results found in auxiliary sections, fix immediately. + +### Key-to-agent mapping (2026-07-03 final sweep, 19:14 EDT) + +| Agent | CT | Key Alias | Config Status | Key Verified | +|-------|-----|-----------|---------------|-------------| +| Tanko | 112 | tanko-v2 | ⚠️ Hardcoded (correct key, no api_key_env) | ✅ 200 | +| Mumuni | 114 | mumuni-v3 | ✅ Clean (api_key_env) | ✅ 200 | +| Koby | 111 | koby-v3 | ✅ Clean (11 fixes applied) | ✅ 200 | +| Koonimo | 113 | koonimo-v2 | ✅ Clean (3 fixes applied) | ✅ 200 | +| Abiba | 100 | abiba | ✅ pi native | ✅ 200 | +| Kagenz0 | 105 | kagenz0-v2 | ❌ DOWN (no hermes, no config) | — | + +### Fix summary (2026-07-03 second pass) + +**Koby (11 fixes):** +- 7 auxiliary sections: model:auto → model:syslog-auto (compression, flush_memories, mcp, session_search, skills_hub, vision, web_extract) +- custom_providers.harness: deprecated sk-syslog-koby → api_key_env, model:auto → syslog-auto +- delegation: model:auto → syslog-auto, deprecated sk-syslog-koby → api_key_env + +**Mumuni (1 fix):** +- delegation: model:auto → syslog-auto + provider:harness + api_key_env + +**Koonimo (3 fixes, first pass):** +- auxiliary.approval: model:auto → syslog-auto, provider:auto → harness +- auxiliary.skills_hub: model:auto → syslog-auto, provider:auto → harness +- custom_providers.harness: hardcoded koonimo-v3 → api_key_env + +**Tanko (flagged, not urgent):** +- Config has hardcoded tanko-v2 key (correct) instead of api_key_env +- DeepSeek fallback key (sk-b7d9...) is intentional — external service +- Will need fix before next key rotation ### Mumuni sub-agent profiles (6) Sub-agents in `/root/.hermes/profiles//config.yaml`: diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md new file mode 100644 index 0000000..62cb69f --- /dev/null +++ b/hermes-key-enforcement.prose.md @@ -0,0 +1,123 @@ +--- +kind: enforcement +name: hermes-key-enforcement +version: 1.0.0 +description: > + Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM + providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI, + Anthropic) may use hardcoded keys. Single source of truth: /etc/environment on each + agent host. Designed to make key rotation a one-step operation. +author: Abiba (pi agent) +--- + +# Hermes Key Enforcement Contract + +## Rule (One Sentence) + +**Any `api_key` pointing to a Syslog-hosted LiteLLM/harness provider MUST be replaced with `api_key_env: LITELLM_API_KEY` — hardcoded harness keys are forbidden.** + +## Scope + +Applies to all Hermes agent configs across all hosts. Covers these config sections: +- `model.api_key` +- `custom_providers[].api_key` (when `name` contains `harness` or `litellm`) +- `auxiliary.*.api_key` (when `provider` is `harness` or contains `litellm`) +- `delegation.api_key` (when `provider` is `harness` or contains `litellm`) +- `compression.api_key` (when `provider` is `harness` or contains `litellm`) +- `fallback_providers[].api_key` (when provider is harness) + +## Exemptions + +External providers are **explicitly exempt** and may use hardcoded keys: +- DeepSeek (`api.deepseek.com`) +- OpenAI (`api.openai.com`) +- Anthropic (`api.anthropic.com`) +- OpenRouter +- Any provider whose base_url does NOT match `192.168.68.116` or `litellm.sysloggh.net` + +## Standard Pattern + +```yaml +# ✅ CORRECT — all harness/litellm providers +model: + provider: harness # or custom:litellm.sysloggh.net + base_url: http://192.168.68.116/v1 + api_key_env: LITELLM_API_KEY # ← indirection + +custom_providers: + - name: harness + base_url: http://192.168.68.116/v1 + api_key_env: LITELLM_API_KEY # ← indirection + +auxiliary: + compression: + provider: harness + api_key_env: LITELLM_API_KEY # ← indirection + +# ✅ ALSO CORRECT — external providers +fallback_providers: + - provider: deepseek + base_url: https://api.deepseek.com + api_key: sk-b7d9... # ← hardcoded OK (external) + api_key_env: DEEPSEEK_API_KEY # ← also OK if set in /etc/environment +``` + +```yaml +# ❌ FORBIDDEN — hardcoded harness/litellm key +model: + provider: harness + api_key: sk-Flc62smlegyMEaSo1ka8JA # ← RULE VIOLATION +``` + +## Detection Query + +Run on any Hermes host to detect violations: + +```bash +grep -n 'api_key: sk-' /root/.hermes/config.yaml \ + | grep -v 'deepseek\|b7d9\|openai\|anthropic\|DEEPSEEK' +``` + +If any output — violation. Fix immediately. + +## Rotation Procedure + +With this standard enforced, key rotation is one step: + +```bash +# 1. Generate new key in LiteLLM +# 2. Update /etc/environment on agent host +ssh root@ "sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW_KEY/' /etc/environment" +# 3. Restart agent gateway +ssh root@ "pkill -f 'hermes_cli.main gateway run'; sleep 2; nohup ... &" +# 4. Verify +curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models +``` + +**Done.** No config file changes needed. The agent picks up the new key on restart. + +## Verified Agents (2026-07-03 final) + +| Agent | CT | Status | api_key_env entries | Hardcoded harness | Last Verified | +|-------|-----|--------|---------------------|-------------------|---------------| +| Tanko | 112 | ✅ Compliant | 4 | 0 | 19:14 EDT | +| Mumuni | 114 | ✅ Compliant | 8 | 0 | 19:14 EDT | +| Koby | 111 | ✅ Compliant | 14 | 0 | 19:14 EDT | +| Koonimo | 113 | ✅ Compliant | 7 | 0 | 19:14 EDT | +| Abiba | 100 | ✅ N/A (pi native) | — | — | 19:14 EDT | +| Kagenz0 | 105 | ❌ DOWN | — | — | 19:14 EDT | + +## Violation Response + +1. **Detect** — run detection query above +2. **Fix** — replace `api_key: sk-...` with `api_key_env: LITELLM_API_KEY` in all harness/litellm sections +3. **Verify** — `grep -c "api_key_env" config.yaml` should increase, hardcoded harness keys should be 0 +4. **Restart** — gateway must restart to pick up env var +5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200 +6. **Update** — bump the verified table above + +## Related Contracts + +- `hermes-config-template.prose.md` — full configuration template +- `litellm-health.prose.md` — LiteLLM stack health verification +- `zulip-platform-verification.prose.md` — cross-platform agent verification diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index e802c81..1344403 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -137,9 +137,25 @@ description: > |-------|------|-----------| | **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis | | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey | -| **Home stack** | `/opt/home_stack/` | jdownloader, bentopdf, pulse | +| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse | | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf | +### Home Stack Services + +**Stirling-PDF** (replaced Bento-PDF 2026-07-03): +- URL: `http://192.168.68.7:8989` +- Swagger: `http://192.168.68.7:8989/swagger-ui.html` +- Admin credentials: `admin` / `kakashi20stirling` +- API key: `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88` +- Compose: `/opt/home_stack/docker-compose.yml` +- Control script: `/opt/home_stack/infra-control.sh` + +**JDownloader**: +- URL: `http://192.168.68.7:5800` (web UI via VNC) + +**Pulse** (Uptime Kuma): +- URL: `http://192.168.68.7:3001` + ### Ecosystem B: CT 116 syslog-api (192.168.68.116) 8 containers in inference-harness stack: @@ -171,27 +187,69 @@ description: > - 192.168.68.24:9401 (Router metrics exporter) - harness-litellm:4000 (LiteLLM health) -### Ecosystem C: Netbird (72.61.0.17) +### Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud) + +5 containers, single compose stack at `/root/docker-compose.yml`: | Container | Image | Role | |-----------|-------|------| -| netbird-server | netbird | VPN controller | -| netbird-dashboard | netbird UI | Web management | -| netbird-proxy | nginx | TLS termination | -| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF | -| netbird-traefik | traefik | Reverse proxy | +| netbird-server | netbirdio/netbird-server:latest | VPN controller, management API | +| netbird-dashboard | netbirdio/dashboard:latest | Web management UI | +| netbird-proxy | netbirdio/reverse-proxy:latest | SNI router + TLS passthrough for *.sysloggh.net | +| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF / threat intelligence | +| netbird-traefik | traefik:v3.6 | Edge reverse proxy, Let's Encrypt TLS | + +**Architecture:** Internet → Traefik (TLS + HTTP→HTTPS redirect) → Netbird proxy +(TLS passthrough via HostSNI, PROXY protocol) → Netbird mesh → LAN backends. +Traefik handles the Netbird dashboard directly; all other `*.sysloggh.net` +domains are TLS-passthrough to the proxy on port 8443. + +**Proxy config:** `/root/proxy.env` — token, ACME certs in `/certs/`, geolocation +DB auto-downloaded. Certs are per-domain Let's Encrypt via TLS-ALPN-01 challenge. > **Dependency note:** NetBird is an **access layer only** — it exists so a > human on a PC or away from the LAN can reach services by URL. It is **not** > a dependency of any service backend. Per Section 7, every config and agent > must reach backends by LAN IP; the `*.sysloggh.net` URLs are reserved for -> human browsers. The 2026-07-02 outage (NetBird VPS hung) took out every -> CNAME'd URL while every LAN-IP backend stayed up — proof the doctrine is -> necessary. -> -> **Recovery:** a hung NetBird VPS (TCP accepts, SSH banner exchange times -> out, TLS hangs) cannot be self-remediated over the network. It requires a -> hard reboot from the hosting provider's console. See Section 5.3. +> human browsers. + +#### Known Issues + +**A. Access log bloat (CRITICAL)** — 2026-07-03 discovery. The management server +writes an `access_log_entries` row for every proxied request (~45K/day). SQLite +has no built-in TTL. By 2026-07-03 the table reached 10.7M rows / 5.4GB, causing: +- DB writes slow → gRPC deadlines exceeded → proxy loses management connection +- On proxy restart, management server too slow to push route mappings +- Proxy starts HTTPS listener before routes arrive → all domains "unknown" +- **Fix:** batch-delete old entries, VACUUM. Prevention: weekly systemd timer (`netbird-cleanup.timer`) deletes entries >7 days old when DB exceeds 500MB. +- **Recurrence signal:** proxy logs show `unknown domain "git.sysloggh.net"` or `match: false` for known domains → management route sync failed. + +**B. Stale management state after proxy restart.** Management server sometimes +stops pushing route mappings to a reconnecting proxy. Symptom: proxy starts, +mapping stream established, but no routes arrive → all domains `unknown`. +**Fix:** restart management server FIRST, THEN proxy. Order matters. + +**C. Peer `sha-OWLpE8nS4Rp4IWzep+y2ayFeZ6C3lJ4eNRB29L5yQTE=` flapping.** +One peer (connecting through Traefik relay at 172.30.0.10) disconnects every +~2-3 minutes. This may be a symptom of DB bloat (healthcheck timeout due to +slow DB writes) rather than a network issue. Monitor after bloat fix. + +#### Recovery Procedures + +1. **Proxy losing all routes ("unknown domain" for all services):** + ```bash + ssh root@72.61.0.17 + cd /root + docker compose restart netbird-server + sleep 12 + docker compose restart proxy + ``` +2. **Access log bloat detected (DB > 500MB):** + ```bash + ssh root@72.61.0.17 + /root/cleanup-access-logs.sh + ``` +3. **VPS hung (TCP accepts, SSH banner timeout):** hard reboot from Hostinger console. ### Checks (every 60s) @@ -209,6 +267,13 @@ For each Docker host: - docker inspect --format "{{.RestartCount}}" → < 3/hour - df -h / | awk '{print $5}' → usage < 80% +For Netbird VPS specifically: + - netbird-store-db-size: du -m /var/lib/docker/volumes/root_netbird_data/_data/store.db → < 500MB + - netbird-access-log-count: sqlite3 store.db 'SELECT COUNT(*) FROM access_log_entries' → < 500K + - If DB > 500MB: alert + run /root/cleanup-access-logs.sh + - proxy-route-check: curl https://git.sysloggh.net/ → 2xx (proves routes loaded) + - proxy-sync-check: docker logs netbird-proxy --since 60s | grep 'Initial mapping sync complete' → present after restart + For docker-vm specifically: - mountpoint -q /media/storage → NFS mounted - mountpoint -q /media/mediastore → NFS mounted