SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: lessons learned from 2026-07-12 session

Browse Source 
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
This commit is contained in:
root
2026-07-12 22:49:39 +00:00
parent 19b6db9891
commit 79d4a73895
14 changed files with 372 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gpu-fleet.prose.md
+28 -18
View File
@@ -5,9 +5,13 @@ description: >
Manages the GPU inference fleet across all hosts. Handles model deployment, Manages the GPU inference fleet across all hosts. Handles model deployment,
registration, health checks, LiteLLM sync, agent key management, GPU registration, health checks, LiteLLM sync, agent key management, GPU
saturation watchdog, Prometheus/Grafana monitoring, and self-healing. saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
Current as of 2026-07-08: context reduced to 128K on NVIDIA GPUs, parallel 2 UPDATED 2026-07-12: Architecture is DIRECT GPU — LiteLLM routes directly
on all GPUs, LiteLLM timeouts tuned (gemma 25→120s, qwen 40→90s), router fully to llama-server on each GPU host (no router in inference path). Router
deprecated — nginx routes /v1 → LiteLLM directly. (port 9000) is running but NOT in request path. All GPUs standardized on
api-key 'not-needed'. RTX 5070 had api-key mismatch (sk-loc...5678) that
caused cascading 401→timeout→401 fallback loops — fixed.
Context: RTX 3090 verified at 256K (was documented as 128K — WRONG).
Workload: Compression moved to Strix Halo, RTX 5070 → vision/web only.
agent: abiba agent: abiba
triggers: triggers:
- on model add/remove - on model add/remove
@@ -114,10 +118,10 @@ triggers:
### sync-keys ### sync-keys
1. List all agent keys in LiteLLM DB via `GET /key/list` 1. List all agent keys in LiteLLM DB via `GET /key/list`
2. Compare against expected agent list: [tanko, mumuni, abiba, tdunna, baggy, kagenz0] 2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
3. Generate missing keys via `POST /key/generate` with unlimited budget 3. Generate missing keys via `POST /key/generate` with unlimited budget
4. Update agent configs — `/etc/environment` LITELLM_API_KEY 4. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=<key> --project=agents --env=production`
5. Send Zulip DM to agents that can't be reached via SSH 5. Send Zulip DM to agents that can't be reached via SSH (provide vault login instructions)
6. Verify each key with test request through full chain 6. Verify each key with test request through full chain
7. Document keys in knowledge graph 7. Document keys in knowledge graph
@@ -136,19 +140,25 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
7. Check port conflicts: verify only one llama-server on :8080 per host 7. Check port conflicts: verify only one llama-server on :8080 per host
8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`) 8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)
## Agent Keys (LiteLLM DB — Current 2026-06-30) ## Agent Keys (LiteLLM DB — Current 2026-07-11)
| Agent | CT | IP | Key | Access | Keys stored in Infisical vault (project=agents, env=production, secret=LITELLM_API_KEY).
|-------|-----|-----|-----|--------| Agent gateways inject keys at runtime via `infisical run --` wrapper.
| Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome | Plaintext keys removed from this contract post-vault-migration.
| Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
| Abiba | 100 | .24 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
| Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
| Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes. | Agent | CT | IP | LiteLLM Alias | Key Source | Access |
If no SSH access, send Zulip DM via abiba-bot. |-------|-----|-----|---------------|------------|--------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
| Kagenz0 | 105 | ? | `kagenz0` | Infisical vault | no SSH |
> **Note**: CT hostnames differ from agent identities. CT111=tdunna runs koby; CT113=baggy runs koonimo.
**Key update procedure**: Update Infisical vault → `infisical secrets set LITELLM_API_KEY=sk-... --project=agents --env=production` → restart agent gateway. Agent picks up new key via `infisical run --` wrapper at startup.
If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
## Configuration Files ## Configuration Files
@@ -170,7 +180,7 @@ If no SSH access, send Zulip DM via abiba-bot.
| Component | URL | Details | | Component | URL | Details |
|-----------|-----|---------| |-----------|-----|---------|
| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 | | Grafana | `http://192.168.68.116:3001/` | admin / vault (`GRAFANA_ADMIN_PASSWORD`) |
| GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series | | GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
| Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets | | Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
| GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics | | GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
gpu-self-heal.prose.md
+29
View File
@@ -256,3 +256,32 @@ Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Original baseline kept in Grafana. 6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Original baseline kept in Grafana.
7. **Predictive alerts**: Two-tier — warn at >70°C+rising (>2°C/min), critical at >80°C+rising. 7. **Predictive alerts**: Two-tier — warn at >70°C+rising (>2°C/min), critical at >80°C+rising.
8. **Prometheus**: Primary source. Fall back to nvidia-smi/rocm-smi direct probes if exporter down. 8. **Prometheus**: Primary source. Fall back to nvidia-smi/rocm-smi direct probes if exporter down.
## Lessons Learned (2026-07-12)
### L1: API Key Standardization Is Critical
- All GPU llama-servers MUST use the same api-key as the LiteLLM config.
- RTX 5070 had `--api-key sk-loc...5678` while LiteLLM sent `not-needed`.
This caused cascading 401 → fallback → timeout → 401 loops, burning all retries.
- **Rule**: Any new GPU or model restart MUST verify api-key matches LiteLLM config.
### L2: Fallback Chain Cascading Failures
- When one model returns 401 (auth) and another is slow (timeout), the fallback
chain creates an infinite loop: gemma 401 → qwen timeout → gemma 401 → ...
- **Rule**: If a model returns 401 (auth error), do NOT fall back to it again.
Mark it as permanently failed for this request.
### L3: Verify Running State, Not Docs
- RTX 3090 was documented at 128K context. Actually running at 256K.
- Parallel count wrong (docs said 2, actual is 1 on RTX 3090).
- **Rule**: Before making decisions, check `/proc/PID/cmdline` on GPU hosts.
### L4: Infisical Is Not Always Available
- Tanko's Infisical service token was 404 — gateway ran without API key for hours.
- **Rule**: Always keep a local `.env` fallback for `LITELLM_API_KEY`.
- Contract hermes-config-template Rule 3 updated.
### L5: Zulip Event Queue Can Silently Die
- Mumuni's queue accumulated 41 errors/reconnects then stopped polling.
Gateway was running but ignoring all messages.
- **Rule**: litellm-health-check now monitors gateway responsiveness via Zulip API.
hermes-agent-baseline.prose.md
+45 -24
View File
@@ -22,30 +22,38 @@ done
## Agent Map ## Agent Map
| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | Platform | | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|-------------|---------------|----------| |-------|-----|------|-----|---------------|------------|----------|
| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | Hermes | | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | Hermes | | Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
| Tdunna | 111 | amdpve | srv1079750 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | `tdunna` | **pi** | | Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | Hermes | | Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes |
> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH. Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
Keys are stored in Infisical vault (project=agents, env=production) and injected at
runtime via `infisical run --` wrapper. Plaintext keys removed from this baseline.
## Key Architecture ## Key Architecture
``` ```
Infisical vault → infisical run -- hermes gateway → LITELLM_API_KEY (runtime)
↓
Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server) Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
└── Key DB (Postgres) └── Key DB (Postgres)
``` ```
- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs - **Master key**: stored in Infisical vault (project=infrastructure, secret=LITELLM_MASTER_KEY) — ADMIN ONLY
- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name - **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this) - **Key injection**: `infisical run --project=agents --env=production -- hermes gateway run` injects `LITELLM_API_KEY` at runtime
- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match) - **Key source**: Infisical vault → runtime env var. /etc/environment is CLEAN (stripped, tagged `# [INFISICAL]`)
- **Legacy override** (pre-migration): `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` — should be REMOVED
## Config Pattern — Mandatory Fields ## Config Pattern — Mandatory Fields
### For Hermes Agents (Tanko, Mumuni, Baggy) ### For Hermes Agents (Tanko, Mumuni, Koonimo)
Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have: Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
@@ -76,7 +84,7 @@ auxiliary:
model: gemma-4-12b # or syslog-auto model: gemma-4-12b # or syslog-auto
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
api_key: <ACTUAL_KEY_FROM_/etc/environment> # ← MANDATORY workaround api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround
timeout: 60 timeout: 60
download_timeout: 30 download_timeout: 30
``` ```
@@ -91,7 +99,7 @@ auxiliary:
model: syslog-auto # or gemma-4-12b model: syslog-auto # or gemma-4-12b
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
api_key: <ACTUAL_KEY_FROM_/etc/environment> # ← MANDATORY workaround api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround
timeout: 120 timeout: 120
``` ```
@@ -110,8 +118,7 @@ LiteLLM/harness will fail with:
401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-' 401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
``` ```
**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside **Workaround**: Set `api_key` directly (copy the value from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=agents --env=production`) alongside `api_key_env` in every auxiliary task config that uses the harness provider.
`api_key_env` in every auxiliary task config that uses the harness provider.
**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when **Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
`api_key` is empty: `api_key` is empty:
@@ -129,7 +136,11 @@ if not cfg_api_key:
```bash ```bash
for ct in 112 114 111 113; do for ct in 112 114 111 113; do
echo "=== CT $ct ===" echo "=== CT $ct ==="
pct-run $ct grep LITELLM_API_KEY /etc/environment # Verify /etc/environment is CLEAN (no LITELLM_API_KEY)
pct-run $ct "grep -c LITELLM_API_KEY /etc/environment 2>/dev/null || echo '0 (clean)'"
# Verify gateway uses infisical run wrapper
pct-run $ct "ps aux | grep 'infisical run' | grep -v grep"
# Check for hardcoded harness keys
pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
echo "" echo ""
done done
@@ -137,15 +148,17 @@ done
### Master Key Leak Check ### Master Key Leak Check
```bash ```bash
# On every agent: # On every agent — must return empty:
pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null pct-run <CT> grep -rl "sk-litellm" /root/ /etc/ 2>/dev/null
# Must return empty # Vault is the only place the master key should exist
``` ```
### Verify Key Works ### Verify Key Works
```bash ```bash
# Retrieve key from vault and test:
KEY=$(infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
curl -s http://192.168.68.116:80/v1/models \ curl -s http://192.168.68.116:80/v1/models \
-H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto -H "Authorization: Bearer $KEY" | grep syslog-auto
# Must return model list # Must return model list
``` ```
@@ -156,19 +169,27 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
``` ```
### For pi Agents (Tdunna) ### For Koby (CT 111 / tdunna)
Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture). Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
### For pi Agents (Abiba)
Abiba (CT100) runs pi via PM2 with the Zulip extension.
Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`. Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
**models.json** — Must only list models authorized for the agent's LiteLLM key: **models.json** — Must only list models authorized for the agent's LiteLLM key.
Key is injected via `infisical run --` wrapper at PM2 startup:
```json ```json
{ {
"providers": { "providers": {
"syslog-harness": { "syslog-harness": {
"baseUrl": "http://192.168.68.116/v1", "baseUrl": "http://192.168.68.116/v1",
"api": "openai-completions", "api": "openai-completions",
"apiKey": "sk-...", "apiKey": "${LITELLM_API_KEY}",
"models": [ "models": [
{ "id": "syslog-auto" }, { "id": "syslog-auto" },
{ "id": "ornith-1.0-35b" }, { "id": "ornith-1.0-35b" },
@@ -257,6 +278,6 @@ If they differ → ghost detected → kill ghost → start fresh.
| Date | Change | | Date | Change |
|------|--------| |------|--------|
| 2026-07-08 | Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons. | | 2026-07-08 | Koby: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added config section. Key rotated and stored in vault. Added Failure Mode #11 to zulip-adapter-lessons. |
| 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. | | 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied | | 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |
hermes-config-template.prose.md
+7
View File
@@ -203,6 +203,13 @@ The following MUST be identical across ALL profiles:
- Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys - Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys
- Hardcoded keys in config.yaml become stale after key rotation - Hardcoded keys in config.yaml become stale after key rotation
- Infisical vault secrets persist across config updates / reinstalls - Infisical vault secrets persist across config updates / reinstalls
- **NEW (July 2026): Always keep a local `.env` fallback.** Infisical service tokens
can expire/404 (tanko incident: token not found, gateway ran without key for hours).
The `.env` file should have the key uncommented as a fallback:
```
LITELLM_API_KEY=sk-...
# [INFISICAL] Also sourced from vault.sysloggh.net
```
- Restart Hermes after env var updates - Restart Hermes after env var updates
### Rule 4: Sub-Agent Profiles Inherit Auth ### Rule 4: Sub-Agent Profiles Inherit Auth
hermes-key-enforcement.prose.md
+119 -48
View File
@@ -5,8 +5,10 @@ version: 1.0.0
description: > description: >
Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM
providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI, providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI,
Anthropic) may use hardcoded keys. Single source of truth: /etc/environment on each Anthropic) may use hardcoded keys. Single source of truth: Infisical vault
agent host. Designed to make key rotation a one-step operation. (project=agents, env=production) — injected at runtime via `infisical run --` wrapper.
/etc/environment is DEPRECATED for agent keys post-migration. Designed to make key
rotation a one-step vault operation.
author: Abiba (pi agent) author: Abiba (pi agent)
--- ---
@@ -14,7 +16,7 @@ author: Abiba (pi agent)
## Rule (One Sentence) ## Rule (One Sentence)
**Any `api_key` pointing to a Syslog-hosted LiteLLM/harness provider MUST be replaced with `api_key_env: LITELLM_API_KEY` — hardcoded harness keys are forbidden.** **All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with authenticated path `http://192.168.68.116/litellm/v1/responses` — hardcoded keys AND unauthenticated `/v1` direct access are both forbidden.**
## Scope ## Scope
@@ -26,6 +28,39 @@ Applies to all Hermes agent configs across all hosts. Covers these config sectio
- `compression.api_key` (when `provider` is `harness` or contains `litellm`) - `compression.api_key` (when `provider` is `harness` or contains `litellm`)
- `fallback_providers[].api_key` (when provider is harness) - `fallback_providers[].api_key` (when provider is harness)
## Architecture (2026-07-10)
Syslog is migrating away from **unauthenticated direct access** to the shared inference harness.
| Path | Auth | Status |
|------|------|--------|
| `http://192.168.68.116/v1` | None (direct) | ❌ **DEPRECATED** — being phased out |
| `http://192.168.68.116/litellm/v1/responses` | Bearer `sk-*` key | ✅ **CURRENT** — authenticated LiteLLM proxy |
All harness/litellm providers MUST use the authenticated `/litellm/v1/responses` path.
Any `base_url` pointing to bare `/v1` on 192.168.68.116 is a **migration violation**.
### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
When `api_mode: responses` is set, Hermes **appends `/v1/responses`** to `base_url`.
If `base_url` already includes `/litellm/v1/responses`, the result is:
```
http://192.168.68.116/litellm/v1/responses/v1/responses → 404
```
**The `base_url` must end at `/v1` — never include `/responses`:**
```yaml
# ✅ CORRECT — Hermes appends /v1/responses for api_mode: responses
base_url: http://192.168.68.116/litellm/v1
# ❌ WRONG — produces double path
base_url: http://192.168.68.116/litellm/v1/responses
```
This applies to ALL sections using the harness provider: `custom_providers`, `delegation`, `auxiliary.*`.
## Exemptions ## Exemptions
External providers are **explicitly exempt** and may use hardcoded keys: External providers are **explicitly exempt** and may use hardcoded keys:
@@ -38,35 +73,42 @@ External providers are **explicitly exempt** and may use hardcoded keys:
## Standard Pattern ## Standard Pattern
```yaml ```yaml
# ✅ CORRECT — all harness/litellm providers # ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
model: model:
provider: harness # or custom:litellm.sysloggh.net provider: harness
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/litellm/v1 # ← Hermes appends /v1/responses
api_key_env: LITELLM_API_KEY # ← indirection api_key_env: LITELLM_API_KEY
custom_providers: custom_providers:
- name: harness - name: harness
base_url: http://192.168.68.116/v1 api_mode: responses
api_key_env: LITELLM_API_KEY # ← indirection base_url: http://192.168.68.116/litellm/v1 # ← NO /responses suffix!
api_key_env: LITELLM_API_KEY
auxiliary: auxiliary:
compression: compression:
provider: harness provider: harness
api_key_env: LITELLM_API_KEY # ← indirection base_url: http://192.168.68.116/litellm/v1 # ← NO /responses suffix!
api_key_env: LITELLM_API_KEY
# ✅ ALSO CORRECT — external providers # ✅ ALSO CORRECT — external providers
fallback_providers: fallback_providers:
- provider: deepseek - provider: deepseek
base_url: https://api.deepseek.com base_url: https://api.deepseek.com
api_key: sk-b7d9... # ← hardcoded OK (external) api_key: sk-b7d9... # ← hardcoded OK (external)
api_key_env: DEEPSEEK_API_KEY # ← also OK if set in /etc/environment api_key_env: DEEPSEEK_API_KEY # ← also OK if set in environment (vault or /etc/environment)
``` ```
```yaml ```yaml
# ❌ FORBIDDEN — hardcoded harness/litellm key # ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
model: model:
provider: harness provider: harness
api_key: sk-Flc62smlegyMEaSo1ka8JA # ← RULE VIOLATION api_key: sk-Flc62smlegyMEaSo1ka8JA # ← RULE VIOLATION: hardcoded key
model:
provider: harness
base_url: http://192.168.68.116/v1 # ← RULE VIOLATION: unauthenticated path
api_key_env: LITELLM_API_KEY
``` ```
## Detection Query ## Detection Query
@@ -75,13 +117,18 @@ Run on any Hermes host to detect violations:
```bash ```bash
# 1. Check config.yaml for hardcoded harness keys # 1. Check config.yaml for hardcoded harness keys
grep -rn 'api_key: sk-' /home/jerome/.hermes/ \ grep -rn 'api_key: sk-' /root/.hermes/ \
--include='config.yaml' \ --include='config.yaml' \
| grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK' | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
# 1b. Check for double-path bug: base_url ending with /responses
# (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1)
grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
# ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this) # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /root/.config/systemd/ 2>/dev/null
# 3. Verify running process env matches dedicated key # 3. Verify running process env matches dedicated key
cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \ cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
@@ -92,26 +139,32 @@ If any output from step 2 — **critical violation** (master key leaked). Fix im
## Rotation Procedure ## Rotation Procedure
With this standard enforced, key rotation is one step: With this standard enforced, key rotation is one vault update:
```bash ```bash
# 1. Generate new key in LiteLLM # 1. Generate new key in LiteLLM: POST /key/generate with agent alias
# 2. Update /etc/environment on agent host # 2. Update Infisical vault secret
ssh root@<host> "sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW_KEY/' /etc/environment" infisical secrets set LITELLM_API_KEY=sk-NEW_KEY \
# 3. Restart agent gateway --project=agents --env=production
ssh root@<host> "pkill -f 'hermes_cli.main gateway run'; sleep 2; nohup ... &" # 3. Restart agent gateway (key auto-injected via infisical run -- wrapper)
ssh root@<host> "systemctl restart hermes-gateway"
# 4. Verify # 4. Verify
curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/litellm/v1/models
``` ```
**Done.** No config file changes needed. The agent picks up the new key on restart. **Done.** No config file changes needed. No /etc/environment edits needed.
The agent picks up the new key via `infisical run --` at gateway startup.
> **Post-migration note**: /etc/environment is NO LONGER the key source.
> Strip all `LITELLM_API_KEY` lines from /etc/environment (comment out with `# [INFISICAL]`)
> and let the `infisical run --` wrapper inject the key at runtime.
## Key Longevity Policy (2026-07-04) ## Key Longevity Policy (2026-07-04)
**Keys are permanent and use bare agent name aliases.** **Keys are permanent and use bare agent name aliases.**
- **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`. - **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `tdunna`, `baggy`). No dates, no versions. The alias IS the identity. - **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. - **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
- **Max budget**: $100 per key (config default). - **Max budget**: $100 per key (config default).
@@ -128,36 +181,54 @@ litellm_settings:
## Verified Agents (2026-07-05 update) ## Verified Agents (2026-07-05 update)
| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified | | Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
|-------|-----|-----|---------------|-----|--------|----------------|---------------| |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
| Tanko | 112 | .122 | `tanko` | `sk-CggiHWlamQyShxWC3Hx6uw` | ✅ Fixed | User drop-in `env.conf` | 20:17 UTC Jul 5 | | Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 | | Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
| Tdunna | 111 | ? | `tdunna` | `sk-6sbCNjz2T6lTVDBdlNHXsA` | ✅ Fixed | `/etc/environment` + drop-in | 23:30 UTC Jul 5 | | Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
| Baggy | 113 | ? | `baggy` | `sk-krnw_zGBwvvL5b7l2t-s-A` | ✅ Fixed | `/etc/environment` | 23:30 UTC Jul 5 | | Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 | | Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 | | Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
### Systemd Service Pattern (2026-07-04 fix) > **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
> LiteLLM key aliases use agent identity, not CT hostname.
All Hermes agents use systemd to manage their gateway. Two issues were fixed: ### Migration Status: Authenticated Path
1. **Drop-in override** — `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` (or user equivalent) had hardcoded `LITELLM_API_KEY` that bypassed `/etc/environment`. | Agent | `/litellm/v1/responses` | Deprecated `/v1` | Status |
2. **Missing EnvironmentFile** — Services did not source `/etc/environment`. |-------|--------------------------|--------------------|--------|
| Mumuni | ✅ 5 sections | 0 | ✅ Authenticated |
| Tanko | ⚠️ No SSH access | — | Needs check |
| Koby | ⚠️ No route to host | — | Needs check |
| Koonimo | ⚠️ Connection timed out | — | Needs check |
**Correct pattern:** ### Systemd Service Pattern (2026-07-11 — vault migration)
All Hermes agents use systemd to manage their gateway. The gateway service is wrapped
with `infisical run --` to inject secrets at runtime.
**Correct pattern (post-migration):**
```ini ```ini
# In service file: # Service file wraps gateway with Infisical:
EnvironmentFile=/etc/environment [Service]
ExecStart=/usr/bin/infisical run --project=agents --env=production -- \
/usr/bin/hermes gateway run
# Drop-in only for overrides, NOT primary key storage. # /etc/environment is CLEAN — no LITELLM_API_KEY present
# If a drop-in exists, it must match /etc/environment. # (strip it and tag with # [INFISICAL] if present)
``` ```
**Rotation procedure** (one step with this standard): **Legacy pattern (deprecated — pre-migration only):**
```ini
# DO NOT USE post-migration:
EnvironmentFile=/etc/environment
# This pattern was replaced by infisical run -- wrapper
```
**Rotation procedure** (one vault operation with this standard):
1. Generate new key in LiteLLM: `curl /key/generate` with agent alias 1. Generate new key in LiteLLM: `curl /key/generate` with agent alias
2. Update `/etc/environment`: `sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW/' /etc/environment` 2. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=sk-NEW --project=agents --env=production`
3. Update drop-in (if exists): same sed on `litellm-key.conf` 3. Restart: `systemctl restart hermes-gateway` (key auto-injected via wrapper)
4. Restart: `systemctl [--user] restart hermes-gateway`
## Violation Response ## Violation Response
@@ -167,7 +238,7 @@ EnvironmentFile=/etc/environment
4. **Restart** — gateway must restart to pick up env var 4. **Restart** — gateway must restart to pick up env var
5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200 5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200
6. **Update** — bump the verified table above 6. **Update** — bump the verified table above
7. **Use safe-mutate** — if the fix requires changing `/etc/environment` or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating. 7. **Use safe-mutate** — if the fix requires updating vault secrets or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
## Related Contracts ## Related Contracts
@@ -206,13 +277,13 @@ task config:
```yaml ```yaml
auxiliary: auxiliary:
vision: vision:
api_key: sk-CggiHWlamQyShxWC3Hx6uw # ← workaround api_key: sk-<agent-key-from-vault> # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
model: gemma-4-12b model: gemma-4-12b
provider: harness provider: harness
compression: compression:
api_key: sk-CggiHWlamQyShxWC3Hx6uw # ← workaround api_key: sk-<agent-key-from-vault> # ← workaround (same as above)
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
model: gemma-4-12b model: gemma-4-12b
hermes-zulip-plugin.prose.md
+2 -1
View File
@@ -28,7 +28,7 @@ connectivity recovery including end-to-end DM validation.
| Param | Type | Required | Default | Description | | Param | Type | Required | Default | Description |
|-------|------|----------|---------|-------------| |-------|------|----------|---------|-------------|
| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` | | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
| `branch` | string | no | `master` | Git branch to pull (overridable for pinning) | | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
## Maintains ## Maintains
@@ -58,6 +58,7 @@ connectivity recovery including end-to-end DM validation.
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root | | Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
| Field | Value | Trust | | Field | Value | Trust |
|-------|-------|-------| |-------|-------|-------|
hermes-zulip-restore.prose.md
+3 -2
View File
@@ -3,7 +3,7 @@ kind: function
name: hermes-zulip-restore name: hermes-zulip-restore
description: > description: >
Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112, Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
Koby CT111). Deploys the zulip-platform adapter to the correct bundled plugin Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
path, verifies env credentials, restarts the gateway, and confirms Zulip path, verifies env credentials, restarts the gateway, and confirms Zulip
connects. Run this whenever a Hermes agent stops responding on Zulip or after connects. Run this whenever a Hermes agent stops responding on Zulip or after
a fresh agent deployment. a fresh agent deployment.
@@ -23,7 +23,7 @@ gateway restart, and connection validation.
| Param | Type | Required | Default | Description | | Param | Type | Required | Default | Description |
|-------|------|----------|---------|-------------| |-------|------|----------|---------|-------------|
| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` | | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
## Maintains ## Maintains
@@ -54,6 +54,7 @@ gateway restart, and connection validation.
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root | | Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
| Field | Value | Trust | | Field | Value | Trust |
|-------|-------|-------| |-------|-------|-------|
infrastructure-control.prose.md
+9 -9
View File
@@ -62,19 +62,19 @@ description: >
| Resource | Auth Method | Credential Source | Status | | Resource | Auth Method | Credential Source | Status |
|----------|------------|-------------------|--------| |----------|------------|-------------------|--------|
| Proxmox Cluster | PVE API Token | `monitoring@pve!mumuni=...` | ✅ | | Proxmox Cluster | PVE API Token | Infisical vault (`PROXMOX_API_TOKEN`) | ✅ |
| Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ | | Proxmox Root | Password via API ticket | Infisical vault (`PROXMOX_ROOT_PASSWORD`) | ✅ |
| docker-vm (.7) | SSH root | SSH key | ✅ | | docker-vm (.7) | SSH root | SSH key | ✅ |
| CT 116 (syslog-api) | SSH root | SSH key | ✅ | | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ | | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ | | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
| Baggy CT (113) | SSH jerome | ❌ no key access | | Baggy CT (113) | SSH jerome | ❌ no key access |
| Netbird (.17) | SSH root | SSH key | ✅ | | Netbird (.17) | SSH root | SSH key | ✅ |
| Gitea | API token | abiba-bot token | ✅ | | Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ | | Zulip | Bot API key | Infisical vault (`ZULIP_BOT_KEY`) | ✅ |
| RA-H OS | MCP bridge | port 3100 | ✅ | | RA-H OS | MCP bridge | port 3100 | ✅ |
| LiteLLM Admin | master key | `sk-litellm-7f96...` | ✅ VERIFY-BEFORE-USE | | LiteLLM Admin | master key | Infisical vault (`LITELLM_MASTER_KEY`) | ✅ VERIFY-BEFORE-USE |
| Grafana | admin password | `syslog-grafana-2026` | ✅ VERIFY-BEFORE-USE | | Grafana | admin password | Infisical vault (`GRAFANA_ADMIN_PASSWORD`) | ✅ VERIFY-BEFORE-USE |
> **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this > **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this
> contract are live-state fields. Test them against the live system before > contract are live-state fields. Test them against the live system before
@@ -550,13 +550,13 @@ curl -s http://192.168.68.116/health/unified | jq .status
curl -s http://192.168.68.24:9100/gpu-data | jq .summary curl -s http://192.168.68.24:9100/gpu-data | jq .summary
# Grafana status # Grafana status
curl -s http://admin:syslog-grafana-2026@192.168.68.116:3001/api/health curl -s http://admin:$(infisical secrets get GRAFANA_ADMIN_PASSWORD --project=infrastructure --env=production --plain)@192.168.68.116:3001/api/health
# Prometheus targets # Prometheus targets
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}' curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
# LiteLLM key check # LiteLLM key check
curl -s -H "Authorization: Bearer sk-litellm-7f96080dd99b15c36bd4b333b58a6796" \ curl -s -H "Authorization: Bearer $(infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production --plain)" \
http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}' http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'
# Storage check # Storage check
@@ -586,7 +586,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| 108 | media | storepve | — | Media | ❌ | | 108 | media | storepve | — | Media | ❌ |
| 109 | docker-vm | storepve | .7 | Docker host | ❌ | | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
| 110 | gitea | minipve | — | Git | ❌ | | 110 | gitea | minipve | — | Git | ❌ |
| 111 | tdunna | amdpve | — | ? | ❌ | | 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ | | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
| 113 | baggy | amdpve | ? | Hermes agent | ✅ | | 113 | baggy | amdpve | ? | Hermes agent | ✅ |
| 114 | mumuni | minipve | .123 | Hermes agent | ✅ | | 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
infrastructure-update.prose.md
+55 -9
View File
@@ -11,7 +11,7 @@ triggers:
- on "infra update" command - on "infra update" command
- weekly (Sunday 03:00 EDT) via cron - weekly (Sunday 03:00 EDT) via cron
- on security advisory relay from Mumuni - on security advisory relay from Mumuni
version: 1.1.0 version: 1.2.0
--- ---
## Maintains ## Maintains
@@ -27,11 +27,12 @@ Before ANY update wave:
2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os) 2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo) 3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116) 4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
5. ✅ LiteLLM health check passing 5. ✅ LiteLLM health check passing (port 4000, /mcp-rest/tools/list with master key)
6. ✅ Zulip server reachable 6. ✅ LiteLLM MCP gateway serving RA-H OS tools (90 tools)
6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600) 7. ✅ Zulip server reachable
7. ✅ Disk >20% free on all nodes 8. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files) 9. ✅ Disk >20% free on all nodes
10. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
## Wave 1: Storage & Infra Nodes (lowest impact) ## Wave 1: Storage & Infra Nodes (lowest impact)
@@ -65,6 +66,7 @@ Before ANY update wave:
**Verify after Wave 2:** **Verify after Wave 2:**
- All VMs/CTs running: check via Proxmox API - All VMs/CTs running: check via Proxmox API
- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116) - LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
- LiteLLM MCP tools: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools
- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only) - GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
- Zulip agents connected: check Mumuni/Tanko gateway state - Zulip agents connected: check Mumuni/Tanko gateway state
- Abiba PM2 processes online: `pm2 status` - Abiba PM2 processes online: `pm2 status`
@@ -83,6 +85,7 @@ Before ANY update wave:
**Verify after Wave 3:** **Verify after Wave 3:**
- All containers healthy: `docker ps` on each host - All containers healthy: `docker ps` on each host
- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto - End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
- MCP integration test: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools (23 RA-H OS + 67 GitHub)
- Zulip test: send test message to #agent-hub - Zulip test: send test message to #agent-hub
- Dashboard loading: `curl localhost:3001/` (via CT 116) - Dashboard loading: `curl localhost:3001/` (via CT 116)
- Firecrawl test: `curl :3002/` - Firecrawl test: `curl :3002/`
@@ -112,8 +115,8 @@ If ANY verification fails:
Before Wave 1, snapshot these files: Before Wave 1, snapshot these files:
``` ```
/opt/inference-harness/docker-compose.yml (CT 116 .116) /opt/inference-harness/docker-compose.yml (CT 116 .116) ⚡ contains MCP_SERVER env vars
/opt/inference-harness/litellm_config.yaml (CT 116 .116) /opt/inference-harness/litellm_config.yaml (CT 116 .116) ⚡ contains mcp_servers.ra_h_os
/opt/monitoring/prometheus.yml (CT 116 .116) /opt/monitoring/prometheus.yml (CT 116 .116)
/etc/nginx/nginx.conf (harness-nginx on CT 116) /etc/nginx/nginx.conf (harness-nginx on CT 116)
/opt/search-stack/firecrawl-source/docker-compose.yaml (VM 109 .7) /opt/search-stack/firecrawl-source/docker-compose.yaml (VM 109 .7)
@@ -123,9 +126,51 @@ Before Wave 1, snapshot these files:
/root/.pi/agent/extensions/config.yaml (CT 100 .24) /root/.pi/agent/extensions/config.yaml (CT 100 .24)
/etc/systemd/system/ornith-server.service (amdpve .15) /etc/systemd/system/ornith-server.service (amdpve .15)
/etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110) /etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110)
# Hermes agent configs (key enforcement — 2026-07-10)
/root/.hermes/config.yaml (Mumuni CT 114, Tanko CT 112, etc.)
/root/.config/systemd/user/hermes-gateway.service (Mumuni CT 114 — EnvironmentFile fixed)
/etc/environment (Mumuni CT 114 — LITELLM_API_KEY)
``` ```
Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...` ## MCP Gateway (2026-07-10)
LiteLLM CT 116 now serves as an authenticated MCP gateway for RA-H OS tools.
### Configuration
**litellm_config.yaml** (`/opt/inference-harness/litellm_config.yaml`):
```yaml
mcp_servers:
ra_h_os:
url: "http://192.168.68.65:3100/mcp"
transport: "http"
auth_type: "none"
```
**docker-compose.yml** env vars:
```yaml
- MCP_SERVER_RAHOS_URL=http://192.168.68.65:3100/mcp
- MCP_SERVER_RAHOS_TRANSPORT=http
```
### Access
| Key | MCP Access |
|-----|-----------|
| Master key | ✅ Full — 90 tools (vault-injected) |
| Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.90.0-rc.1 |
### Known Limitations
- Per-key MCP server grants not functional — only master key has access
- Responses API (`/v1/responses`) with MCP tools broken on llama.cpp backends
- HTTP 307 redirect on `/mcp` → use `/mcp/` (trailing slash) or `/mcp-rest/` endpoints
- `api_mode: responses` in Hermes appends `/v1/responses` to base_url → **base_url must end at `/v1`, never `/responses`** (double-path bug)
### Migration Path
When LiteLLM is upgraded to a version supporting per-key MCP grants:
1. Grant agent keys `mcp_servers: ["ra_h_os"]`
2. Update Hermes `mcp_servers.ra-h-os.url` from `http://192.168.68.65:3100/mcp` → `http://192.168.68.116:4000/mcp/`
3. Add `headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"}` to MCP config
## Security-Specific Updates ## Security-Specific Updates
@@ -144,6 +189,7 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
- [ ] LiteLLM inference passing (syslog-auto test) - [ ] LiteLLM inference passing (syslog-auto test)
- [ ] Zulip server + all 3 agents connected - [ ] Zulip server + all 3 agents connected
- [ ] GPU fleet at full capacity (3/3) - [ ] GPU fleet at full capacity (3/3)
- [ ] LiteLLM MCP gateway healthy (90 tools via master key)
- [ ] Zero security CVEs remaining - [ ] Zero security CVEs remaining
- [ ] <10 min total downtime per service - [ ] <10 min total downtime per service
litellm-api-keys.prose.md
+18 -6
View File
@@ -9,6 +9,11 @@ description: >
not calendar-driven — rotate only on compromise, personnel change, or not calendar-driven — rotate only on compromise, personnel change, or
periodic security hygiene (quarterly/annually). periodic security hygiene (quarterly/annually).
UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
expire/404. The .env fallback prevents agents from running without keys.
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
on CT 116. Last verified: 2026-07-09. on CT 116. Last verified: 2026-07-09.
@@ -19,7 +24,10 @@ description: >
- agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni") - agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
- action: "create" | "rotate" | "verify" | "list" — What to do (default: "create") - action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
- litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000") - litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
- master_key: string — LiteLLM master key (default from environment) - master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
- vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
- vault_project: string — Infisical project slug (default: "infrastructure")
- vault_env: string — Infisical environment (default: "production")
- agent_host: string — Agent's IP for SSH (default: resolved from infra) - agent_host: string — Agent's IP for SSH (default: resolved from infra)
- agent_user: string — SSH user (default: "jerome") - agent_user: string — SSH user (default: "jerome")
@@ -30,12 +38,13 @@ description: >
- key_prefix: string — First 10 chars of the new key (for identification) - key_prefix: string — First 10 chars of the new key (for identification)
- previous_key_alias: string | null — Previous key alias if rotating - previous_key_alias: string | null — Previous key alias if rotating
- litellm_response: object — Raw response from LiteLLM /key/generate - litellm_response: object — Raw response from LiteLLM /key/generate
- agent_config_updated: boolean — Whether /etc/environment was updated - vault_updated: boolean — Whether Infisical vault secret was updated
- agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
- verification: { status: string, detail: string } — Final health check - verification: { status: string, detail: string } — Final health check
## Execution ## Execution
1. **Authenticate** — Verify master_key works against LiteLLM /key/list 1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project=<vault_project> --env=<vault_env>`, verify against LiteLLM /key/list
2. **Check existing keys** — List all keys, find any with agent_name alias 2. **Check existing keys** — List all keys, find any with agent_name alias
3. **If action == "list"**: Return all keys with their aliases and spend 3. **If action == "list"**: Return all keys with their aliases and spend
4. **If action == "create"**: 4. **If action == "create"**:
@@ -47,11 +56,14 @@ description: >
- Return the new key - Return the new key
5. **If action == "rotate"**: 5. **If action == "rotate"**:
- Generate new key with same alias (LiteLLM replaces the old key) - Generate new key with same alias (LiteLLM replaces the old key)
- SSH to agent_host, update /etc/environment LITELLM_API_KEY - Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>`
- Restart agent gateway (hermes gateway restart for Hermes agents) - Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process)
The gateway automatically picks up the new key via `infisical run --` wrapper
- Verify: curl test against /v1/models with new key - Verify: curl test against /v1/models with new key
- Rotation policy: on-demand only (compromise, departure, quarterly hygiene) - Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
- Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
6. **If action == "verify"**: 6. **If action == "verify"**:
- SSH to agent, read /etc/environment - Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>`
- Test the key against LiteLLM /v1/models - Test the key against LiteLLM /v1/models
- Confirm key alias matches agent_name in LiteLLM key list - Confirm key alias matches agent_name in LiteLLM key list
- Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
litellm-self-heal.prose.md
+6 -4
View File
@@ -1,11 +1,13 @@
--- ---
kind: responsibility kind: responsibility
name: litellm-self-heal name: litellm-self-heal
status: manual-only status: deployed
note: > note: >
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04). DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * *
This contract is now manual-only — triggers require explicit user request. Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
Consider reimplementing as a standalone cron job or prose contract. now reimplemented as `litellm-health-check.sh` on CT 116.
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
GPU monitoring integrated from gpu-monitor on .24:9100.
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
duplication of architecture diagrams, GPU topology, timeout tables, and container duplication of architecture diagrams, GPU topology, timeout tables, and container
memory-audit-maintenance.prose.md
+4 -4
View File
@@ -1,7 +1,7 @@
--- ---
name: memory-audit-maintenance name: memory-audit-maintenance
kind: responsibility kind: responsibility
description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens. description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
id: 067NC4KG01RG50R40M30E20918 id: 067NC4KG01RG50R40M30E20918
--- ---
@@ -15,13 +15,13 @@ Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md,
### Scope ### Scope
This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own memory files only no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data. This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
**Agent Roster:** **Agent Roster:**
- Mumuni - Mumuni
- Tanko - Tanko
- Tdunna - Koby (CT 111 / tdunna)
- Baggy - Koonimo (CT 113 / baggy)
**Isolation Principle:** Each agent has its own: **Isolation Principle:** Each agent has its own:
- `MEMORY.md` and `USER.md` - `MEMORY.md` and `USER.md`
proxmox-monitor.prose.md
+2 -2
View File
@@ -41,7 +41,7 @@ agent: abiba
- User: `monitoring@pve` (cluster-replicated) - User: `monitoring@pve` (cluster-replicated)
- Role: `PVEAuditor` on `/` (read-only, whole cluster) - Role: `PVEAuditor` on `/` (read-only, whole cluster)
- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c` - Token: `monitoring@pve!prometheus` — stored in Infisical vault (`PROXMOX_MONITOR_TOKEN`)
- `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`) - `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
## Grafana Dashboards (file-provisioned, folder "Syslog Fleet") ## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
@@ -62,7 +62,7 @@ agent: abiba
- **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`) - **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
- **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers` - **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
- **Credentials**: admin / syslog-grafana-2026 - **Credentials**: admin / password stored in Infisical vault (`GRAFANA_ADMIN_PASSWORD`)
- Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route. - Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
- grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1) - grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)
scripts/agent-health-check.py
+44 -4
View File
@@ -19,13 +19,53 @@ from datetime import datetime
LITELLM = "http://192.168.68.116:80" LITELLM = "http://192.168.68.116:80"
def _get_agent_key(agent_name):
"""Retrieve agent key from Infisical vault."""
try:
result = subprocess.run(
["infisical", "secrets", "get", "LITELLM_API_KEY",
"--project=agents", "--env=production", "--plain"],
capture_output=True, text=True, timeout=10
)
if result.returncode == 0:
return result.stdout.strip()
except Exception:
pass
# Fallback: try exporting all secrets
try:
result = subprocess.run(
["infisical", "export", "--project=agents", "--env=production",
"--format=dotenv"],
capture_output=True, text=True, timeout=10
)
if result.returncode == 0:
for line in result.stdout.splitlines():
if line.startswith(f"LITELLM_API_KEY_{agent_name.upper()}") or \
(line.startswith("LITELLM_API_KEY=") and agent_name == os.uname().nodename):
return line.split("=", 1)[1].strip().strip('"').strip("'")
except Exception:
pass
return None
# Agent keys are pulled from Infisical vault at runtime.
# The 'key' field is populated dynamically below.
AGENTS = { AGENTS = {
"tanko": {"ct": 112, "host": "192.168.68.122", "key": "sk-CggiHWlamQyShxWC3Hx6uw", "user": "jerome"}, "tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome"},
"mumuni": {"ct": 114, "host": "192.168.68.123", "key": "sk-VrqCNlwUgzoNGOpikJ7nwQ", "user": "root"}, "mumuni": {"ct": 114, "host": "192.168.68.123", "user": "root"},
"tdunna": {"ct": 111, "host": None, "key": "sk-6sbCNjz2T6lTVDBdlNHXsA", "user": None}, "koby": {"ct": 111, "host": None, "user": None},
"baggy": {"ct": 113, "host": None, "key": "sk-krnw_zGBwvvL5b7l2t-s-A", "user": None}, "koonimo": {"ct": 113, "host": None, "user": None},
} }
# Inject keys from vault
for agent_name in AGENTS:
key = _get_agent_key(agent_name)
if key:
AGENTS[agent_name]["key"] = key
else:
AGENTS[agent_name]["key"] = None
GPU_HOSTS = { GPU_HOSTS = {
"gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-server"}, "gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-server"},
"gpu-rtx5070 (.110)": {"host": "192.168.68.110", "port": 8080, "service": "llama-server"}, "gpu-rtx5070 (.110)": {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},