SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

prose: review, fix, and consolidate infrastructure contracts

Browse Source 
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.
This commit is contained in:
root
2026-07-04 21:47:33 +00:00
parent 3077fda0b7
commit 253e19f680
14 changed files with 628 additions and 213 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hermes-config-template.prose.md
+18 -201
View File
@@ -1,17 +1,11 @@
---
kind: pattern
kind: template
name: hermes-config-template
description: >
Standard Hermes configuration template for Syslog Solution LLC agents.
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
RA-H OS MCP) while keeping agent-specific API keys and model choices.
Updated 2026-07-03 (19:14 EDT): Full sweep — Koby (11 fixes), Mumuni (1 fix),
Koonimo (3 fixes). All agents now CLEAN: zero model:auto, zero deprecated
sk-syslog-* keys. Tanko has hardcoded tanko-v2 key (correct, but should use
api_key_env for rotation safety). Kagenz0 is DOWN (no config, no key).
Previous: full key audit — mumuni-v3 + context7-mcp registered,
/etc/environment aligned, double-B8nw incident documented, all 17 LiteLLM
keys mapped with active/inactive status.
Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
---
## Maintains
@@ -19,210 +13,33 @@ description: >
- template_version: "2.0.0"
- last_applied: timestamp
- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
- agent_keys: managed in LiteLLM DB — current aliases in Key Management section below
- agent_keys: map (see Agent Keys section)
- infra_endpoints_verified: array
### Agent topology
## Agent Keys (LiteLLM — Current 2026-06-30)
| Agent | CT | IP | Key Alias | Startup | Zulip Owner |
|-------|-----|-----|-----------|---------|-------------|
| Tanko | 112 | 192.168.68.122 | `tanko-v3` | systemd user + `.env` | Jerome only ✅ |
| Mumuni | 114 | 192.168.68.123 | `mumuni-v3` | systemd user + `.env` | Jerome only ✅ |
| Koonimo | 113 | 192.168.68.114 | `koonimo-v2` | systemd (Discord/Telegram) | N/A |
| Koby | 111 | 192.168.68.129 | `koby-v3` | systemd user (Discord/Telegram) | N/A |
| Abiba | — | 192.168.68.24 | `abiba` | `pi` harness | N/A |
| Kagenz0 | ? | ? | `kagenz0` | unknown | not verified |
Each agent has a unique LiteLLM API key. Keys are stored in LiteLLM's PostgreSQL DB.
The env var `LITELLM_API_KEY` is set in `/etc/environment` on each agent host.
Sub-agent profiles inherit auth from the main config — no separate keys needed.
### Zulip ownership rules (ALL agents)
| Agent | Key | Host | SSH | Sub-Agents |
|-------|-----|------|-----|-----------|
| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | 192.168.68.122 | jerome@.122 | — |
| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | 192.168.68.24 | local | — |
| Koby | `sk-wb86PvcZXWkjShAIBxrpDg` | ? | Zulip | — |
| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | 192.168.68.114 | Zulip | — |
| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | Zulip | — |
**Every Zulip-enabled agent MUST only respond to its owner.** This is enforced
via two env vars in the agent's `.env` file:
```bash
ZULIP_ALLOW_ALL_USERS=false
ZULIP_ALLOWED_USERS=jerome@sysloggh.com
```
- `ZULIP_ALLOW_ALL_USERS=false` — disables the default open-to-everyone mode
- `ZULIP_ALLOWED_USERS=jerome@sysloggh.com` — only Jerome can trigger responses
This applies regardless of whether the agent uses `.env` directly (Mumuni)
or a systemd drop-in (Tanko). Both mechanisms work — the key is that the
env vars reach the Hermes process.
## Key Management
**Source of truth:** LiteLLM PostgreSQL DB. Plaintext keys live ONLY in
`/etc/environment` on each agent host, never hardcoded in contracts or config.yaml.
This contract defines the procedure; it does NOT store plaintext keys.
### Current key aliases (2026-07-03 audit)
**Active keys (by agent):**
| Agent | Alias | Key (last 8) | Source | Status |
|-------|-------|-------------|--------|--------|
| Tanko | `tanko-v3` | ...JN2OsJw | Process env | Active (needs restart → v2) |
| Tanko | `tanko-v2` | ...1ka8JA | `/etc/environment` | Registered, idle |
| Mumuni | `mumuni-v3` | ...OiWU_B8nw | Process + /etc/env | ✅ Aligned |
| Koonimo | `koonimo-v3` | ...A_wIOA | Config (hardcoded) | Active |
| Koonimo | `koonimo-v2` | ...R7FAobA | `/etc/environment` | Registered, idle |
| Koby | `koby-v3` | ...P4DrWw | `/etc/environment` | ✅ Registered |
| Koby | `koby-hermes` | ...syslog-koby | Config (deprecated) | Active legacy |
| Abiba | `abiba` | ...aLgMw | pi config | ✅ Active |
| Kagenz0 | `kagenz0` | ...Uggg | unknown | Last active Jul 1 |
| Context7 | `context7-mcp` | ...ZPh6VA | Process env | ✅ Registered |
**Unused registrations:** `abiba-v2`, `kagenz0-v2`, `koonimo-v2`, `synthetic-monitor`, `UI Dashboard`, unnamed key
> **2026-07-03 alignment**: Tanko/Koonimo configs updated to use `api_key_env:
> LITELLM_API_KEY` (was hardcoded `sk-syslog-*`). Base URLs corrected to
> `/v1` (was `/litellm/v1`). All active keys now registered in LiteLLM.
> Tanko and Koonimo need Hermes gateway restart for config changes to take
> effect.
### Key lineage
| Agent | Alias | Key (last 8) | Status | Date |
|-------|-------|-------------|--------|------|
| Tanko | `tanko-v3` | ...JN2OsJw | **active** (process) | 2026-07-03 |
| Tanko | `tanko-v2` | ...1ka8JA | idle (in /etc/env) | 2026-07-02 |
| Mumuni | `mumuni-v3` | ...OiWU_B8nw | **active** | 2026-07-03 |
| Mumuni | `mumuni-v2` | ...HcnuOPtQ | superseded | 2026-07-02 |
| Mumuni | `mumuni` | ...62da807e | superseded | 2026-06-30 |
| Koonimo | `koonimo-v3` | ...A_wIOA | **active** (hardcoded) | 2026-07-03 |
| Koonimo | `koonimo-v2` | ...R7FAobA | idle (in /etc/env) | 2026-07-02 |
| Koby | `koby-v3` | ...P4DrWw | **active** (/etc/env) | 2026-07-03 |
| Koby | `koby-hermes` | ...syslog-koby | legacy (in config) | 2026-06-30 |
> **2026-07-03 incidents**:
> - Mumuni: double-B8nw — process had `...POiWU_B8nw`, /etc/environment
> had `...VDtHWd-XiWU_B8nw`. Wrong one registered first.
> - Context7 MCP: `sk-XY2aUfvy2BIs6kp1ZPh6VA` was never registered — the
> original error Kwame reported.
> - Tanko: process key `sk-3ZsdWJbbNSo9zSnJN2OsJw` unregistered. Config
> used deprecated `sk-syslog-tanko`. Both fixed.
> - Koonimo: hardcoded `sk-KFnE3Td6aKEBZxwVA_wIOA` unregistered. Base URL
> used old `/litellm/v1` path. Both fixed.
> - Koby: `/etc/environment` key `sk-x5BnZUUKvz3r8EdLP4DrWw` unregistered.
> Config used deprecated `sk-syslog-koby`. Fixed.
### Verify: all keys working
```bash
# Run from Abiba against each agent host
for agent in tanko mumuni; do
host=192.168.68.$([ "$agent" = tanko ] && echo 122 || echo 123)
key=$(ssh root@$host "grep LITELLM_API_KEY /etc/environment | cut -d= -f2")
code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $key" http://192.168.68.116/v1/models)
printf " %-10s -> %s\n" "$agent" "$code"
done
```
### Rotate: generate new key, deploy, verify
```bash
AGENT=mumuni && ALIAS="${AGENT}-v3"
# 1. Generate new key in LiteLLM (master key required)
KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \
-H 'Authorization: Bearer sk-litellm-7f960...' \
-H 'Content-Type: application/json' \
-d "{\"key_alias\":\"$ALIAS\",\"models\":[\"gemma-4-12b\",\"qwen3.6-27B-code\",\"ornith-1.0-35b\",\"syslog-auto\"],\"max_budget\":1000}" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("key"))')
# 2. Verify new key
curl -s -H "Authorization: Bearer $KEY" http://192.168.68.116/v1/models
# 3. Deploy: update /etc/environment on agent host
ssh root@192.168.68.123 "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=$KEY/' /etc/environment"
# 4. Restart agent (systemd or kill + relaunch)
ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f hermes_cli.main; cd /root/.hermes && nohup python -m hermes_cli.main gateway run > /dev/null 2>&1 &)"
```
### Key deployment rules
1. **Plaintext keys live ONLY in `/etc/environment`** — never in a contract or config.yaml
2. **All configs use `api_key_env: LITELLM_API_KEY`** — indirection prevents staleness
3. **`custom_providers[].api_key` MUST use `api_key_env`** — hardcoded keys shadow env vars and bypass rotation
4. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access
5. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly
6. **After rotating any key:** restart the corresponding agent gateway
7. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login
8. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid)
### Model name enforcement (2026-07-03)
**CRITICAL**: Hermes `model: auto` in any config section gets sent to LiteLLM
as the literal model name "auto", which LiteLLM rejects with 403 because
the registered model is `syslog-auto`, not `auto`.
**Must-fix pattern** — any auxiliary section with `model: auto`:
```yaml
# ❌ BROKEN — sends model "auto" to LiteLLM (403 Forbidden)
auxiliary:
approval:
model: auto
provider: auto
skills_hub:
model: auto
provider: auto
# ✅ FIXED — registered model name
auxiliary:
approval:
model: syslog-auto
provider: harness
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1
```
**Detection**: `grep -rn "model: auto" /root/.hermes/config.yaml` on any
Hermes host — if any results found in auxiliary sections, fix immediately.
### Key-to-agent mapping (2026-07-03 final sweep, 19:14 EDT)
| Agent | CT | Key Alias | Config Status | Key Verified |
|-------|-----|-----------|---------------|-------------|
| Tanko | 112 | tanko-v2 | ⚠️ Hardcoded (correct key, no api_key_env) | ✅ 200 |
| Mumuni | 114 | mumuni-v3 | ✅ Clean (api_key_env) | ✅ 200 |
| Koby | 111 | koby-v3 | ✅ Clean (11 fixes applied) | ✅ 200 |
| Koonimo | 113 | koonimo-v2 | ✅ Clean (3 fixes applied) | ✅ 200 |
| Abiba | 100 | abiba | ✅ pi native | ✅ 200 |
| Kagenz0 | 105 | kagenz0-v2 | ❌ DOWN (no hermes, no config) | — |
### Fix summary (2026-07-03 second pass)
**Koby (11 fixes):**
- 7 auxiliary sections: model:auto → model:syslog-auto (compression, flush_memories, mcp, session_search, skills_hub, vision, web_extract)
- custom_providers.harness: deprecated sk-syslog-koby → api_key_env, model:auto → syslog-auto
- delegation: model:auto → syslog-auto, deprecated sk-syslog-koby → api_key_env
**Mumuni (1 fix):**
- delegation: model:auto → syslog-auto + provider:harness + api_key_env
**Koonimo (3 fixes, first pass):**
- auxiliary.approval: model:auto → syslog-auto, provider:auto → harness
- auxiliary.skills_hub: model:auto → syslog-auto, provider:auto → harness
- custom_providers.harness: hardcoded koonimo-v3 → api_key_env
**Tanko (flagged, not urgent):**
- Config has hardcoded tanko-v2 key (correct) instead of api_key_env
- DeepSeek fallback key (sk-b7d9...) is intentional — external service
- Will need fix before next key rotation
### Mumuni sub-agent profiles (6)
Sub-agents in `/root/.hermes/profiles/<name>/config.yaml`:
`syslog-code`, `syslog-devops`, `syslog-email`, `syslog-research`,
`syslog-review`, `syslog-writer`.
All sub-agent profiles inherit auth: `api_key: ''`, `base_url: ''`.
When the main key is rotated, all 7 configs (main + 6 subs) stay valid —
only `/etc/environment` needs updating.
✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
syslog-review, syslog-writer — all at `/root/.hermes/profiles/<name>/config.yaml`
## Infrastructure Stack
| Component | Endpoint | Purpose |
|---|---|---|
| Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
| SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search |
| SearXNG | `http://storepve:8888` | Privacy-respecting web search |
| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
| RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |