hermes-key-enforcement.prose.md: - Add IP column to verified agents table - Fix Tanko status: was using master key via systemd drop-in (not compliant) - Add detection query step 2: check systemd drop-ins for master key leaks - Add step 3: verify running process env against dedicated key - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only) gpu-fleet.prose.md: - Add CT ID column to agent table for cross-reference - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host) - Update Abiba LiteLLM key to match actual models.json key - Set Koonimo IP to unknown (was .114 — incorrect per user) - Fix pi-specific paths .24 → .65 hermes-config-template.prose.md: - Set Koonimo IP to unknown infrastructure-control.prose.md: - Set Koonimo IP to unknown - Fix Koonimo CT reference
7.6 KiB
kind, name, version, description, author
| kind | name | version | description | author |
|---|---|---|---|---|
| enforcement | hermes-key-enforcement | 1.0.0 | Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI, Anthropic) may use hardcoded keys. Single source of truth: /etc/environment on each agent host. Designed to make key rotation a one-step operation. | Abiba (pi agent) |
Hermes Key Enforcement Contract
Rule (One Sentence)
Any api_key pointing to a Syslog-hosted LiteLLM/harness provider MUST be replaced with api_key_env: LITELLM_API_KEY — hardcoded harness keys are forbidden.
Scope
Applies to all Hermes agent configs across all hosts. Covers these config sections:
model.api_keycustom_providers[].api_key(whennamecontainsharnessorlitellm)auxiliary.*.api_key(whenproviderisharnessor containslitellm)delegation.api_key(whenproviderisharnessor containslitellm)compression.api_key(whenproviderisharnessor containslitellm)fallback_providers[].api_key(when provider is harness)
Exemptions
External providers are explicitly exempt and may use hardcoded keys:
- DeepSeek (
api.deepseek.com) - OpenAI (
api.openai.com) - Anthropic (
api.anthropic.com) - OpenRouter
- Any provider whose base_url does NOT match
192.168.68.116orlitellm.sysloggh.net
Standard Pattern
# ✅ CORRECT — all harness/litellm providers
model:
provider: harness # or custom:litellm.sysloggh.net
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY # ← indirection
custom_providers:
- name: harness
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY # ← indirection
auxiliary:
compression:
provider: harness
api_key_env: LITELLM_API_KEY # ← indirection
# ✅ ALSO CORRECT — external providers
fallback_providers:
- provider: deepseek
base_url: https://api.deepseek.com
api_key: sk-b7d9... # ← hardcoded OK (external)
api_key_env: DEEPSEEK_API_KEY # ← also OK if set in /etc/environment
# ❌ FORBIDDEN — hardcoded harness/litellm key
model:
provider: harness
api_key: sk-Flc62smlegyMEaSo1ka8JA # ← RULE VIOLATION
Detection Query
Run on any Hermes host to detect violations:
# 1. Check config.yaml for hardcoded harness keys
grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
--include='config.yaml' \
| grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
# 3. Verify running process env matches dedicated key
cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
| tr '\0' '\n' | grep LITELLM_API_KEY
If any output from step 2 — critical violation (master key leaked). Fix immediately.
Rotation Procedure
With this standard enforced, key rotation is one step:
# 1. Generate new key in LiteLLM
# 2. Update /etc/environment on agent host
ssh root@<host> "sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW_KEY/' /etc/environment"
# 3. Restart agent gateway
ssh root@<host> "pkill -f 'hermes_cli.main gateway run'; sleep 2; nohup ... &"
# 4. Verify
curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
Done. No config file changes needed. The agent picks up the new key on restart.
Key Longevity Policy (2026-07-04)
Keys are permanent and use bare agent name aliases.
- Duration:
null— keys never expire. This is enforced bydefault_key_generate_paramsinlitellm_config.yaml. - Alias convention: bare agent name only (e.g.,
tanko,mumuni,koby,koonimo). No dates, no versions. The alias IS the identity. - Rotation triggers: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
- Max budget: $100 per key (config default).
# In litellm_config.yaml — ensures all future keys inherit these defaults:
litellm_settings:
default_key_generate_params:
models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b"]
duration: null # ← permanent
max_budget: 100
metadata:
purpose: "agent-inference"
Verified Agents (2026-07-05 update)
| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
|---|---|---|---|---|---|---|---|
| Tanko | 112 | .122 | tanko |
sk-620a05e95a-e93d... |
✅ Fixed | User drop-in env.conf |
19:44 UTC Jul 5 |
| Mumuni | 114 | .123 | mumuni |
sk-XY2aUfvy2BIs6kp1ZPh6VA |
⚠️ Unverified | /etc/environment |
23:00 EDT Jul 4 |
| Koby | 111 | ? | koby |
sk-lDTsWy7H3T19UwUFEN6JSg |
⚠️ Unverified | User service + drop-in | 23:00 EDT Jul 4 |
| Koonimo | 113 | ? | koonimo |
sk-Prx_vRXYb2VEzDmhPCbNeg |
⚠️ Unverified | System service + drop-in | 23:00 EDT Jul 4 |
| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
Systemd Service Pattern (2026-07-04 fix)
All Hermes agents use systemd to manage their gateway. Two issues were fixed:
- Drop-in override —
/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf(or user equivalent) had hardcodedLITELLM_API_KEYthat bypassed/etc/environment. - Missing EnvironmentFile — Services did not source
/etc/environment.
Correct pattern:
# In service file:
EnvironmentFile=/etc/environment
# Drop-in only for overrides, NOT primary key storage.
# If a drop-in exists, it must match /etc/environment.
Rotation procedure (one step with this standard):
- Generate new key in LiteLLM:
curl /key/generatewith agent alias - Update
/etc/environment:sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW/' /etc/environment - Update drop-in (if exists): same sed on
litellm-key.conf - Restart:
systemctl [--user] restart hermes-gateway
Violation Response
- Detect — run detection query above
- Fix — replace
api_key: sk-...withapi_key_env: LITELLM_API_KEYin all harness/litellm sections - Verify —
grep -c "api_key_env" config.yamlshould increase, hardcoded harness keys should be 0 - Restart — gateway must restart to pick up env var
- Confirm — test key against LiteLLM:
curl -H "Authorization: Bearer $KEY" .../v1/models→ 200 - Update — bump the verified table above
- Use safe-mutate — if the fix requires changing
/etc/environmentor restarting the gateway on a remote host, usesafe-mutateto verify current state before mutating.
Related Contracts
hermes-config-template.prose.md— full configuration templatelitellm-health.prose.md— LiteLLM stack health verificationzulip-platform-verification.prose.md— cross-platform agent verificationlitellm-api-keys.prose.md— API key creation, rotation, and verification
CI Pipeline (2026-07-04)
All contract changes must pass the PR Pipeline before merge:
auth → validate → lint → ai-review → gate
- Trigger: push to master (abiba-bot only) or pull request
- Branch protection: Only
abiba-botcan push directly to master. All other users must use PRs. - Status check:
PR Pipeline — Authorize → Validate → Review → Mergerequired before merge - Runner:
runner-ct110(Gitea Actions v0.6.1) on CT 110 - Config:
.gitea/workflows/pr-pipeline.yaml