vault: Infisical cleanup + contract sync (WAL #1316)

- litellm-api-keys: vault audit, tanko/koby/koonimo migration, key rotation log
- gpu-fleet: ornith-1.0-35b→strix-moe (remaining refs)
- hermes-agent-baseline: 256K all GPUs, CT IPs updated, Shumba retired
- delegation-prose-contract: removed (renamed to mumuni-delegation)
- contract-registry.yaml + cron-prompts-review.md: from feat/contract-registry

cron-prompts-review.md

@@ -0,0 +1,612 @@
+# Cron Prompts Review — All 10 Scheduled Contracts
+
+Generated: 2026-07-13 20:59:18 ET
+
+---
+
+## hermes-key-enforcement
+
+**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 6 * * *
+
+```
+Contract Enforcement: hermes-key-enforcement
+
+Category: compliance
+Domain: hermes-agent
+Owner: abiba
+Schedule: Daily compliance scan at 6am ET
+
+This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: hermes-key-enforcement.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/hermes-key-enforcement/
+
+Postconditions to verify:
+[
+  {
+    "check": "no plaintext API keys in config",
+    "verify": "grep -rc 'api_key: sk-' /root/.hermes/config.yaml",
+    "expect": "0 matches"
+  },
+  {
+    "check": "api_key_env used for harness/litellm providers",
+    "verify": "grep -c 'api_key_env.*LITELLM_API_KEY' /root/.hermes/config.yaml",
+    "expect": "count > 0"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + pause
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/hermes-key-enforcement/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-key-enforcement, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## hermes-config-template
+
+**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 4 * * 1
+
+```
+Contract Enforcement: hermes-config-template
+
+Category: compliance
+Domain: hermes-agent
+Owner: abiba
+Schedule: Weekly config drift check Monday at 4am ET
+
+This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: hermes-config-template.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/hermes-config-template/
+
+Postconditions to verify:
+[
+  {
+    "check": "agent config template_version matches template file",
+    "verify": "grep -q 'template_version' /root/.hermes/config.yaml && diff <(grep 'template_version' /root/.hermes/config.yaml | cut -d: -f2 | xargs) <(grep 'template_version' /root/prose-contracts/hermes-config-template.prose.md | cut -d: -f2 | xargs) && echo match || echo mismatch",
+    "expect": "match"
+  },
+  {
+    "check": "config file is valid YAML",
+    "verify": "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
+    "expect": "valid"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/hermes-config-template/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-config-template, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## hermes-agent-baseline
+
+**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 5 * * 1
+
+```
+Contract Enforcement: hermes-agent-baseline
+
+Category: compliance
+Domain: hermes-agent
+Owner: abiba
+Schedule: Weekly baseline verification Monday at 5am ET
+
+This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: hermes-agent-baseline.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/hermes-agent-baseline/
+
+Postconditions to verify:
+[
+  {
+    "check": "Hermes agent process running",
+    "verify": "pgrep -f 'hermes' > /dev/null && echo running || echo stopped",
+    "expect": "running"
+  },
+  {
+    "check": "agent config file exists and valid YAML",
+    "verify": "test -f /root/.hermes/config.yaml && python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
+    "expect": "valid"
+  },
+  {
+    "check": "no uncommitted changes in hermes directory",
+    "verify": "cd /root/.hermes && git status --porcelain | wc -l",
+    "expect": "0"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/hermes-agent-baseline/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-agent-baseline, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## proxmox-monitor
+
+**Category:** monitoring | **Domain:** proxmox | **Owner:** abiba | **Schedule:** */15 * * * *
+
+```
+Contract Enforcement: proxmox-monitor
+
+Category: monitoring
+Domain: proxmox
+Owner: abiba
+Schedule: Every 15 minutes
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: proxmox-monitor.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/proxmox-monitor/
+
+Postconditions to verify:
+[
+  {
+    "check": "all Proxmox nodes reachable",
+    "verify": "curl -sf http://192.168.68.10:8006/api2/json/status | jq '.status'",
+    "expect": "healthy"
+  },
+  {
+    "check": "no VMs in crashed state",
+    "verify": "pvesh get /nodes -output-format=json | jq '.[] | select(.status==\"Crashed\")'",
+    "expect": "empty"
+  },
+  {
+    "check": "backups running on schedule",
+    "verify": "pbs-info --check",
+    "expect": "last_backup < 24h ago"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/proxmox-monitor/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=proxmox-monitor, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## gpu-monitor
+
+**Category:** monitoring | **Domain:** gpu | **Owner:** abiba | **Schedule:** */15 * * * *
+
+```
+Contract Enforcement: gpu-monitor
+
+Category: monitoring
+Domain: gpu
+Owner: abiba
+Schedule: Every 15 minutes — polls all GPU subsystems
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: gpu-monitor.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/gpu-monitor/
+
+Postconditions to verify:
+[
+  {
+    "check": "GPU metrics accessible",
+    "verify": "curl -sf http://localhost:9100/gpu-data",
+    "expect": "200 OK, populated data"
+  },
+  {
+    "check": "dashboard serving",
+    "verify": "curl -sf http://localhost:9100/gpu-fleet.html",
+    "expect": "200 OK, HTML returned"
+  },
+  {
+    "check": "health endpoint responsive",
+    "verify": "curl -sf http://localhost:9100/health",
+    "expect": "200 OK"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/gpu-monitor/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=gpu-monitor, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## infrastructure-monitoring
+
+**Category:** monitoring | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** */30 * * * *
+
+```
+Contract Enforcement: infrastructure-monitoring
+
+Category: monitoring
+Domain: infrastructure
+Owner: abiba
+Schedule: Every 30 minutes
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: infrastructure-monitoring.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/infrastructure-monitoring/
+
+Postconditions to verify:
+[
+  {
+    "check": "Proxmox API reachable",
+    "verify": "curl -sf http://192.168.68.10:8006/api2/json",
+    "expect": "200 OK"
+  },
+  {
+    "check": "Zulip API reachable",
+    "verify": "curl -sf https://chat.sysloggh.net/api/v1/me",
+    "expect": "200 OK"
+  },
+  {
+    "check": "LiteLLM proxy reachable",
+    "verify": "curl -sf http://192.168.68.116/litellm/v1/models",
+    "expect": "200 OK"
+  },
+  {
+    "check": "Gitea API reachable",
+    "verify": "curl -sf https://git.sysloggh.net/api/v1/version",
+    "expect": "200 OK"
+  },
+  {
+    "check": "SearXNG reachable",
+    "verify": "curl -sf http://192.168.68.17:8080",
+    "expect": "200 OK"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/infrastructure-monitoring/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-monitoring, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## zulip-health
+
+**Category:** monitoring | **Domain:** zulip | **Owner:** abiba | **Schedule:** */15 * * * *
+
+```
+Contract Enforcement: zulip-health
+
+Category: monitoring
+Domain: zulip
+Owner: abiba
+Schedule: Every 15 minutes — monitors all Zulip-connected agents
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: zulip-health.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/zulip-health/
+
+Postconditions to verify:
+[
+  {
+    "check": "bot registration active",
+    "verify": "curl -sf https://chat.sysloggh.net/api/v1/me | jq '.user_id'",
+    "expect": "bot_id present"
+  },
+  {
+    "check": "DM delivery working",
+    "verify": "curl -sf https://chat.sysloggh.net/api/v1/users/me/is-online",
+    "expect": "online: true"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/zulip-health/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=zulip-health, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## litellm-health
+
+**Category:** monitoring | **Domain:** litellm | **Owner:** abiba | **Schedule:** */10 * * * *
+
+```
+Contract Enforcement: litellm-health
+
+Category: monitoring
+Domain: litellm
+Owner: abiba
+Schedule: Every 10 minutes — LiteLLM proxy health
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: litellm-health.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/litellm-health/
+
+Postconditions to verify:
+[
+  {
+    "check": "LiteLLM proxy reachable",
+    "verify": "curl -sf http://192.168.68.116/litellm/v1/models",
+    "expect": "200 OK, models returned"
+  },
+  {
+    "check": "router deprecated, nginx routes work",
+    "verify": "curl -sf https://litellm.sysloggh.net/v1/models",
+    "expect": "200 OK (via nginx)"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/litellm-health/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=litellm-health, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## memory-audit-maintenance
+
+**Category:** maintenance | **Domain:** memory | **Owner:** mumuni | **Schedule:** 0 3 * * *
+
+```
+Contract Enforcement: memory-audit-maintenance
+
+Category: maintenance
+Domain: memory
+Owner: mumuni
+Schedule: Daily at 3am ET
+
+This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: memory-audit-maintenance.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/memory-audit-maintenance/
+
+Postconditions to verify:
+[
+  {
+    "check": "memory files below 80% capacity",
+    "verify": "wc -l ~/.hermes/memories/*.md",
+    "expect": "total lines < threshold"
+  },
+  {
+    "check": "no stale entries",
+    "verify": "grep -r 'STALE' ~/.hermes/memories/",
+    "expect": "0 matches"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify mumuni → action: relay_alert
+- CRITICAL: notify mumuni, abiba → action: relay_alert
+- FATAL: notify mumuni, abiba, kwame → action: relay_alert + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/memory-audit-maintenance/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=memory-audit-maintenance, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## infrastructure-update
+
+**Category:** maintenance | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** 0 2 * * 0
+
+```
+Contract Enforcement: infrastructure-update
+
+Category: maintenance
+Domain: infrastructure
+Owner: abiba
+Schedule: Weekly system updates Sunday at 2am ET
+
+This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: infrastructure-update.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/infrastructure-update/
+
+Postconditions to verify:
+[
+  {
+    "check": "all services running after update",
+    "verify": "systemctl list-units --state=running",
+    "expect": "all critical services"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
+
+Circuit Breaker:
+- Max retries: 1 per 120.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/infrastructure-update/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-update, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+
+---
+*End of review*

gpu-fleet.prose.md

@@ -5,13 +5,12 @@ description: >
   Manages the GPU inference fleet across all hosts. Handles model deployment,
   registration, health checks, LiteLLM sync, agent key management, GPU
   saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
-  UPDATED 2026-07-12: Architecture is DIRECT GPU — LiteLLM routes directly
+  UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe,
-  to llama-server on each GPU host (no router in inference path). Router
+  gpu-dense, gpu-light. These never change — only the underlying model does.
-  (port 9000) is running but NOT in request path. All GPUs standardized on
+  Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
-  api-key 'not-needed'. RTX 5070 had api-key mismatch (sk-loc...5678) that
+  RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
-  caused cascading 401→timeout→401 fallback loops — fixed.
+  RTX 5070 context: 131K → 256K. VRAM: 88% (10.8/12.2GB).
-  Context: RTX 3090 verified at 256K (was documented as 128K — WRONG).
+  Compression timeout: 300s (was 120s). Mumuni context: 128K (was 256K).
-  Workload: Compression moved to Strix Halo, RTX 5070 → vision/web only.
 agent: abiba
 triggers:
   - on model add/remove
@@ -68,48 +67,72 @@ triggers:
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
-│ 256K ctx │  │ 131K ctx    │  │ 256K ctx  │  │ Watchdog     │
+│ 256K ctx │  │ 256K ctx    │  │ 256K ctx  │  │ Watchdog     │
-│ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
+│ qwen3.6  │  │ gemma-4-12b │  │ qwen3.6   │  │ Prometheus   │
-│ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
+│ 27B-code │  │ :8080       │  │ -35B-udq4 │  │ exporter     │
 │ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
 │ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
 └──────────┘
 ```
 
-## Current Model Assignments (2026-07-12)
+## Stable Role-Based Aliases (Introduced 2026-07-15)
+
+Agent configs, cron jobs, and workflows MUST use these aliases, never model-specific names.
+When a model is swapped on a GPU, ONLY the infrastructure layer changes — agent configs are untouched.
+
+| Alias | GPU | Current Model | Will Route To |
+|-------|-----|---------------|---------------|
+| `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
+| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 |
+| `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
+
+**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
+but are deprecated for agent configs. Only the stable aliases survive model swaps.
+
+## Current Model Assignments (2026-07-15)
 
 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
-| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.7/24GB (84%) | **256K** | turbo4 | 1 | default | ✅ healthy |
+| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | 22.2/24.6GB (90%) | **256K** 🚀 | turbo4 | 2 | default | ✅ 63 tok/s |
-| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 256K | q4_0 | 2 | 2048/1024 | ✅ healthy |
-| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
+| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
 
 ## Routing Configuration (LiteLLM — July 2026)
 
-### syslog-auto Weighted Pool
+### syslog-auto Weighted Pool (Direct GPU — bypasses router)
 
-| Model | GPU | Weight | RPM Cap | Purpose |
+| Model | GPU | Weight | RPM Cap | Timeout |
 |-------|-----|--------|---------|---------|
-| qwen3.6-27B-code | RTX 3090 | 0.55 | 500 | Heavy reasoning, code, long context |
+| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
-| ornith-1.0-35b | Strix Halo | 0.30 | **60** | Agentic workflows, tool calling |
+| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
-| gemma-4-12b | RTX 5070 | 0.15 | 200 | Overflow + vision |
+| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
+
+Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path.
 
 ### Direct Model Endpoints
 
 | Model | RPM Cap | Notes |
 |-------|---------|-------|
-| ornith-1.0-35b | 40 | Tight cap — prevents Strix overload |
+| qwen3.6-35B-udq4 | 40 | Tight cap — prevents Strix overload |
 | qwen3.6-27B-code | 500 | High cap — primary workhorse |
-| gemma-4-12b | 500 | High cap — fast 12B |
+| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
+
+### Stable Aliases (for agent configs — never change)
+
+| Alias | RPM Cap | Routes To | Purpose |
+|-------|---------|-----------|---------|
+| `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) |
+| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
+| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
 
 ### Fallback Chains
 - gemma → qwen
 - qwen → gemma
-- ornith → qwen → gemma
+- qwen3.6-35B-udq4 → qwen → gemma
-- syslog-auto → qwen → gemma → ornith
+- syslog-auto → qwen → gemma → qwen3.6-35B-udq4
 
-### Why ornith RPM Is Capped
+### Why Strix Halo RPM Is Capped
-- Direct: 40 RPM (tight) — Strix Halo is shared with compression tasks
+- Direct (qwen3.6-35B-udq4): 40 RPM (tight) — Strix Halo is shared with compression tasks
 - Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
 - Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
 
@@ -163,7 +186,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
 5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
-   - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
+   - gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (strix-moe does NOT exist — legacy name, do not use)
    - global request_timeout: 300s, nginx proxy_read_timeout: 600s
 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
 7. Check port conflicts: verify only one llama-server on :8080 per host
@@ -203,7 +226,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 | `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
 | `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
-| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
+| `/etc/systemd/system/strix-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo) running unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
 
 ## Prometheus & Grafana
 
@@ -220,14 +243,14 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
   Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
-- **VRAM (2026-07-12)**: RTX 3090 at 20.7/24GB (84%), RTX 5070 at 10.0/12.2GB (82%). RTX 3090 context increased to 256K (was incorrectly documented as 128K — verified via /proc/PID/cmdline).
+- **VRAM (2026-07-15)**: RTX 3090 at 22.2/24.6GB (90%) with **256K context** (corrected from 131K). RTX 5070 at 10.8/12.2GB (88%) with 256K context + MTP. Strix Halo at ~9GB/64GB.
-- **RTX 3090 runs `--parallel 1`** (verified 2026-07-12). RTX 5070 and Strix at parallel 2.
+- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
-- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching`. Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context corrected to 256K (2026-07-15). VRAM: 90%. Service: `/home/llmuser/llama-wrapper.sh`.
-- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 1024 --parallel 2`. Api-key standardized to `not-needed` (was `sk-loc...5678` causing 401 loops). Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 256K context. Gen speed: 122 tok/s (was 70). VRAM: 10.8/12.2GB (88%). No draft model pre-upgrade due to VRAM constraints. Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 262144`.
-- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
+- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
-- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
+- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080 (was `ornith-server.service`), model changed to `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (UD-Q4_K_M), alias `qwen3.6-35B-udq4`, 256K context, flash-attn + q8 KV. MTP support enabled for 1.4-2.2x faster inference.
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
-- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
+- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
 - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
 - **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
 - **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
@@ -237,11 +260,14 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 
 ## GPU Inference Benchmarks (Current)
 
-| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
+| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 |
+| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **256K** |
-| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 |
+| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **256K** |
-| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |
+| Strix Halo (.15) | qwen3.6-35B-udq4 | **71** | — | — | **256K** |
+
+Benchmarks from 2026-07-15 verification run. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
+All 3 GPUs now at 256K context (2026-07-15).
 
 Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
 Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
@@ -249,12 +275,55 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window.
 
 **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
 
-## Agent Config Implications (2026-07-12)
+## Agent Config Implications (2026-07-15)
 
-With RTX 3090 at 256K context (verified July 2026):
+### Stable Aliases — CRITICAL
-- Agents using `syslog-auto` (55/30/15 qwen+ornith+gemma): `context_length: 262144` — ornith and qwen both support it
+
-- Agents using `qwen3.6-27B-code` directly: `context_length: 262144` (256K ctx verified)
+All agent configs MUST use stable role-based aliases, never model-specific names:
-- Agents using `gemma-4-12b` directly (auxiliary tasks): `context_length: 131072`
+- `compression.model: strix-moe` (NOT `qwen3.6-35B-udq4`)
-- Compression threshold at 0.65: fires at ~170K for 262K context window on Strix Halo
+- `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`)
-- All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
+- `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`)
-- Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)
+- `auxiliary.web_extract.model: gpu-light`
+
+When the underlying model is swapped, only the LiteLLM config changes — agent configs are untouched.
+
+### Context Windows
+- RTX 3090: **256K** (was 131K, bumped 2026-07-15) | RTX 5070: **256K** (up from 131K) | Strix Halo: **256K**
+- **Mumuni compression context**: 128K (down from 256K) — ensures compression model doesn't timeout
+- Compression threshold 0.65: fires at ~85K for 128K context window
+- Mumuni compression model alias: `strix-moe` with 300s timeout
+
+### Mumuni Agent Profile
+
+Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs:
+
+| Setting | Value | Notes |
+|---------|-------|-------|
+| `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) |
+| `model.provider` | `custom:litellm` | LiteLLM on CT116 |
+| `compression.model` | `strix-moe` | Stable alias — survives model swaps |
+| `aux.compression.model` | `strix-moe` | Compression auxiliary model |
+| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
+| `aux.web_extract.model` | `gpu-light` | Web extraction |
+| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
+| `context.max_context_window` | 262144 (256K) | Fixed 2026-07-16 (was 131072 — caused premature compression, WAL #1300) |
+| `compression.threshold` | 0.65 | Triggers at ~85K |
+| `compression.target_ratio` | 0.3 | Compresses to ~38K |
+| `compression.protect_last_n` | 40 | Preserves last 40 messages |
+| `memory.memory_char_limit` | 800 | Brief memory entries |
+| `personalities` | `creative` | Creative assistant personality |
+| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
+| Main model timeout | 300s | LiteLLM global timeout |
+| Compression model timeout | 300s | strix-moe timeout increased from 120s |
+
+### Agent Update Status (2026-07-15)
+
+| Agent | Host | Status |
+|-------|------|--------|
+| **Mumuni** | CT114 (.123) | ✅ Updated to stable aliases |
+| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
+| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
+| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
+| **Kagenz0** | CT105 | ❌ SSH unreachable — needs Zulip DM |
+
+All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap.

hermes-agent-baseline.prose.md

@@ -5,7 +5,7 @@ version: 1.0.0
 description: >
   Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
   configuration state, keys, workarounds, and audit procedure. When an agent's
-  configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
+  configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
 author: Abiba (pi agent)
 ---
 
@@ -26,11 +26,11 @@ done
 |-------|-----|------|-----|---------------|------------|----------|
 | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
 | Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
-| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
+| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
-| Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
+| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
-| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes |
+| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
 
-> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
+> **Note**: CT hostnames (tdunna→CT129, baggy→CT114) differ from agent identities (koby, koonimo).
 
 Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
 Keys are stored in Infisical vault (project=agents, env=production) and injected at
@@ -169,9 +169,9 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
 # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
 ```
 
-### For Koby (CT 111 / tdunna)
+### For Koby (CT 129 / tdunna)
 
-Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
+Koby runs Hermes on CT 129 (tdunna). Config files at `/root/.hermes/config.yaml`.
 Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
 
 **LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.

litellm-api-keys.prose.md

@@ -14,9 +14,15 @@ description: >
   expire/404. The .env fallback prevents agents from running without keys.
   Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
   
+  UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
+  token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
+  infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
+  Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
+  Abiba's key is now a proper agent key (NOT the master key — stale note removed).
+  
   Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
   Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
-  on CT 116. Last verified: 2026-07-12.
+  on CT 116. Last verified: 2026-07-16.
 ---
 
 ## Parameters
@@ -51,8 +57,8 @@ description: >
    - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
    - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
    - Duration is null (permanent) — inherited from litellm default_key_generate_params
-   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
+   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
-   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
+   - Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
    - Return the new key
 5. **If action == "rotate"**:
    - Generate new key with same alias (LiteLLM replaces the old key)
@@ -67,3 +73,147 @@ description: >
    - Test the key against LiteLLM /v1/models
    - Confirm key alias matches agent_name in LiteLLM key list
    - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
+
+## Production Vault Access Process (canonical, 2026-07-16)
+
+The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending —
+runs as user `jerome`, not systemd root, needs user-scope adaptation).
+
+### The canonical pattern
+
+1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`).
+2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`).
+   - Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project.
+   - Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).
+3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`):
+   ```bash
+   #!/bin/bash
+   export INFISICAL_API_URL="https://vault.sysloggh.net"
+   TOKEN=$(cat /root/.infisical-token)
+   LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs
+   while true; do
+     infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
+       --env=prod --domain=https://vault.sysloggh.net -- bash -c '
+         . /root/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3] safety net only
+         export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"
+         exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run
+       ' >> $LOG 2>&1
+     sleep 5   # restart on exit
+   done
+   ```
+4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth.
+5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation.
+6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot).
+7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault.
+
+### Why this is non-fail
+
+- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
+- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable.
+- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway.
+- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source.
+
+### Migration status (2026-07-16)
+
+| Agent | Host | Pattern | Vault key | Status |
+|-------|------|---------|-----------|--------|
+| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed |
+| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed |
+| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram |
+| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
+> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent.
+| tanko | .122 | infisical-gateway.sh + service token (migrated 2026-07-16) | TANKO_LITELLM_API_KEY | ✅ vault-backed |
+
+### Tanko migration (pending)
+
+Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in
+`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced).
+Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`)
+with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering
+enabled (`loginctl enable-linger jerome`) so the user service runs without a login session.
+
+### Koby migration lessons (2026-07-16)
+
+Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
+**Two mistakes I made that broke the agent:**
+1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
+   in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
+   inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
+2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
+   Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
+
+**How Koby actually connects (2026-07-16):**
+- Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
+  Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config).
+- Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026).
+  Allowed users: 6679773481. Home channel: 6679773481.
+- Both platforms now connect through the wrapper's env injection.
+
+**Golden rule for gateway restarts:** always `cat /proc/<pid>/environ` before killing the old
+process — captures the live env set. Especially important when migrating gateways between
+injection mechanisms.
+
+### Key rotation procedure (one vault operation with this standard)
+
+1. Generate new key: `POST /key/generate` (master key, admin).
+2. Update vault: `infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`.
+3. Update `.env` fallback: `echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`.
+4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
+5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
+
+## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
+
+**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
+In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace
+key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt
+repo only ships 0.38.0, so `apt upgrade` does not help.
+
+**Proper fix — Machine Identity (Infisical automation best practice):**
+Create a machine identity with READ+WRITE scope on the `agents` project (project_id=
+`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely.
+Then vault writes work from any host:
+```bash
+# Get a machine-identity access token
+TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \
+  -H 'Content-Type: application/json' \
+  -d '{"clientId":"<CLIENT_ID>","clientSecret":"<CLIENT_SECRET>"}' | jq -r .accessToken)
+# Write a secret via REST API v3
+ curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \
+  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
+  -d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
+# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
+```
+**2026-07-16 UPDATE — vault SYNCED + CLEANED.** The abiba service token (`st.8e848433…`, READ+WRITE)
+can write to the vault. All session-13 rotated keys in vault and validate 200 against LiteLLM.
+4 stale secrets deprecated, 5 personal creds flagged for separate project. Tanko migrated from
+hardcoded keys to `infisical-gateway.sh` wrapper. Koonimo Zulip key restored.
+
+**Machine Identity:** `8ddb9438-74fc-4ab6-bd74-929e7c47b53b` exists in Infisical UI.
+Client secret needed to use universal auth for vault writes from automation.
+
+**Service Token Inventory (2026-07-16):**
+| Token ID | Name | Permissions | Used By |
+|----------|------|-------------|---------|
+| `st.8e848433…` | tanko-gateway | READ+WRITE | Abiba, Koonimo |
+| `st.353699cd…` | tanko-agent | READ-only | Tanko |
+
+## Key Rotation Log
+
+| Date | Agent | Action | Notes |
+|------|-------|--------|-------|
+| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault (Tt2bUL…). Updated wrapper to inject ZULIP_API_KEY + ZULIP_EMAIL. Restarted gateway → Zulip connected as koonimo-bot@ (bot_id=17). 3 platforms now. |
+| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml key to infisical-gateway.sh wrapper + service token st.353699cd… (tanko-agent). Systemd user service updated, hardcoded key drop-ins removed. Verified LITELLM_API_KEY from vault, 3 platforms connected. |
+| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated: KAGENZ0_LITELLM_API_KEY, HERMES_OPENROUTER_KEY, LITELLM_API_KEY (generic duplicate), ZULIP_API_KEY (generic duplicate). 5 personal creds flagged for separate project. |
+| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
+| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
+| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
+
+## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
+
+- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
+- Used for /key/generate, /key/delete, /key/list (GET), DB queries
+- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
+It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
+The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference —
+see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference".
+- LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`

mumuni-delegation-prose-contract.prose.md

@@ -0,0 +1,290 @@
+---
+kind: pattern
+name: mumuni-delegation
+description: >
+  Mumuni-specific operating doctrine for task decomposition, worker
+  delegation, verification, and delivery. Defines when to delegate, which
+  worker to use for what, how to handle failures, and the kanban board
+  protocol. Enforces context-window discipline and separation of concerns.
+  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
+version: 1.0.0
+---
+
+## Maintains
+
+- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
+  `syslog-research`, `syslog-review`, `syslog-writer`)
+- Kanban board state at `~/.hermes/kanban/kanban.json`
+- Context window budget: ~65K tokens per request (131K total, 60% threshold)
+
+## Topology
+
+**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
+**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
+**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
+
+This contract is infrastructure-agnostic in terms of which nodes are used.
+Workers execute tasks on whatever infrastructure they're given — SSH to .6,
+.pm, .9, .12, or .15 depending on the task. The contract defines the
+**who** and **when** — not the **where**.
+
+## Why This Matters
+
+Without enforced delegation, the manager consumes the full iteration budget
+(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
+leaving no capacity for actual coordination. The result: context overflow
+(59K tokens in system prompt), iteration exhaustion, and degraded response
+quality. This contract exists because I blew through my budget checking
+Proxmox node status instead of delegating to `syslog-devops`.
+
+## Context Window Discipline
+
+**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
+**Volatile (MEMORY.md + USER.md): ~300 tokens.**
+**Total base: ~6,800 tokens per request.**
+
+The remaining budget is the conversation. Every tool call result adds to it.
+If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
+from multiple nodes), the context fills fast. That's why we delegate: workers
+process in isolation and return compact results.
+
+## Trigger Conditions
+
+Delegation is **mandatory** when any of these apply:
+
+| Condition | Threshold | Example |
+|-----------|-----------|---------|
+| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
+| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
+| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
+| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
+| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
+| Code builds or changes | Writing or modifying code | Scripts, configs, patches |
+| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
+
+**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
+`curl`, `hermes tools list` — these are decision-making tools. The manager
+reads them directly.
+
+## Data Source Integrity (CRITICAL)
+
+**Workers MUST use the data provided in their task context. They MUST NOT
+fetch their own data from external sources unless explicitly told to.**
+
+When a task says "Read file X and format it", the worker reads file X. It does
+not query a separate API, run its own diagnostics, or pull data from a different
+system. This is the #1 source of cross-worker inconsistency: one worker gathers
+SSH data, another queries the Proxmox API, and the report merges two incompatible
+datasets.
+
+**Rule:** If a worker needs additional data beyond what's in its task description,
+it asks the manager (via relay) — it doesn't go find it on its own.
+
+**This is a hard rule, not a recommendation.** Violating it produces the exact
+type of discrepancy the kanban pipeline exists to prevent: a review worker finds
+"5 nodes present" in the raw data but "5/5 online" in the report — even though
+one of those nodes was unreachable. The report lied because it used data the
+raw data never provided.
+
+## Worker Selection Matrix
+
+| Worker | Model | Toolsets | Role | Use When |
+|--------|-------|----------|------|----------|
+| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
+| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
+| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
+| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
+| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
+| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
+
+### Selection Rules
+
+1. **Match specialty first.** A code task → `syslog-code`. An infra task →
+   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
+2. **Research tasks with browser needs → `syslog-research`.** Other workers
+   don't have the browser toolset.
+3. **Verification → `syslog-review`.** Never deliver raw worker output.
+4. **Documentation/content → `syslog-writer`.** Let them own the prose.
+5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
+   (includes browser) and high reasoning effort.
+
+## Delegation Protocol
+
+### Step 1: Decompose
+
+Break the task into lanes. Each lane does ONE thing. Workers are independent —
+no lane depends on another's output mid-flight. If lanes depend on each other,
+dispatch sequentially.
+
+### Step 2: Dispatch
+
+Fire workers via `delegate_task`:
+
+**Critical: Pass the data, not just the goal.** When dispatching a worker that
+processes output from another worker, include the file path AND explicit
+instructions to use ONLY that source. Example:
+
+```
+delegate_task(
+    goal="Format the cluster check into a clean report",
+    context="Source data is at /tmp/proxmox-check-raw.md. Format ONLY the data
+    in that file. Do NOT query the Proxmox API or any other data source. Use the
+    file as your sole source of truth."
+)
+```
+
+**Parallel (independent lanes):**
+```
+delegate_task(
+    tasks=[
+        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
+        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
+    ]
+)
+```
+
+**Sequential (dependent lanes):**
+Dispatch lane 1 → wait for result → dispatch lane 2.
+
+### Step 3: Verify
+
+**MANDATORY for:**
+- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
+- Code builds and modifications
+- Research findings (web data, external sources)
+- Any output that will reach the user
+
+**Fire `syslog-review` to verify:**
+```
+delegate_task(
+    goal="Review the output of the devops worker. Verify the node status
+    report is accurate, check for inconsistencies, confirm all nodes were
+    reachable.",
+    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
+    Verify against live system."
+)
+```
+
+**If verification fails:**
+1. Send work back to original worker with review feedback
+2. Re-verify
+3. Max 2 re-verify cycles before escalating to Kwame
+
+### Step 4: Deliver
+
+Only verified results reach Kwame. Format per channel:
+- Telegram: Use `telegram-formatting` skill
+- Zulip: Use Zulip Markdown (CommonMark)
+- Email: Use `syslog-email` skill
+
+## Kanban Board Protocol
+
+**File:** `~/.hermes/kanban/kanban.json`
+
+```json
+{
+  "task_id": "unique-id",
+  "title": "Task description",
+  "created": "2026-07-09T01:00:00",
+  "status": "backlog|in_progress|review|done",
+  "lanes": [
+    {
+      "lane_id": "devops-check",
+      "worker": "syslog-devops",
+      "goal": "Check all 5 Proxmox nodes",
+      "status": "dispatched|completed|failed",
+      "output_file": "/tmp/node-report.md"
+    }
+  ]
+}
+```
+
+**Update the board on every state change.**
+
+## Failure Handling
+
+### Worker Timeouts
+
+- Child timeout: **900 seconds** (15 minutes)
+- Worker model `syslog-auto` is slow — it can hit the timeout limit with
+  22+ API calls
+- **If a worker times out:** Re-dispatch with a narrower scope. Break the
+  task into smaller pieces that fit in the timeout window.
+- **Avoid delegating sequential SSH hops** — each SSH connection adds latency
+  that compounds quickly. Prefer API-based or local approaches when possible.
+
+### Worker Selection Failures
+
+- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
+- `syslog-code` is best for code-level work (reading files, writing scripts)
+- `syslog-research` has the browser toolset — use for web research
+- `syslog-review` is the QA gate — always fire before delivery
+- **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
+- **Never nest delegation** (max_spawn_depth: 1)
+
+### Context Overflow
+
+- If a task requires >10K tokens of output, delegate the processing
+- Workers return compact summaries, not raw data dumps
+- Pass file paths and concrete goals — never dump raw data into context
+
+## Anti-patterns
+
+- ❌ Reading large files into your own context before deciding → delegate the read
+- ❌ Carrying SSH/grep/output results in your context → delegate the analysis
+- ❌ Doing work yourself and then "pretending" to delegate → the user can tell
+- ❌ Skipping verification → raw worker output never reaches the user
+- ❌ Delegating single tool calls → keep quick reads/writes at manager level
+- ❌ Firing more than 3 workers in parallel → hard limit
+- ❌ **Workers fetching their own data sources** → a writer worker that queries the Proxmox API when told to "format the raw file" is fabricating data. Use the input given, not external sources
+
+## Emergency Exception
+
+**In an emergency (server down, service must be restored immediately):**
+- Delegate the diagnosis (find the problem)
+- Execute the fix yourself (minimize handoff latency)
+- Verify the fix after delivery
+- Log the exception in the kanban board
+
+The emergency exception exists because the user needs the service back NOW,
+not after three worker round-trips. But it's an exception — not the rule.
+
+## What This Contract Doesn't Cover
+
+1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
+2. **SSH key management** — covered by existing SSH/Proxmox contracts
+3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
+4. **Cron job management** — covered by individual cron contracts
+5. **Infra verification** — covered by `verify-before-mutate` protocol
+
+## Verification
+
+Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
+
+```bash
+python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
+```
+
+## References
+
+- `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
+- `worker-profile-audit.md` (skill reference): Worker configuration audit notes
+- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
+- `verify-before-mutate` protocol: Infrastructure change verification
+- `hermes-config-template.prose.md`: Worker profile configuration
+
+## Success Criteria
+
+This contract succeeds when:
+
+1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
+2. **Workers do the work** — manager coordinates, doesn't execute
+3. **Verification before delivery** — all output passes through `syslog-review`
+4. **Kanban board is current** — every task has a lane, every lane has a status
+5. **User gets verified results** — raw worker output never reaches Kwame
+
+---
+
+**Last updated:** 2026-07-09
+**Author:** Mumuni (with Kwame's input on triggers and exception criteria)
+**Status:** Draft — awaiting PR review and merge to prose-contracts main

ra-h-os-custodianship-contract.prose.md

@@ -0,0 +1,210 @@
+---
+kind: pattern
+name: ra-h-os-custodianship-contract
+description: >
+  Operational standards for maintaining RA-H OS as a high-functioning shared
+  memory system across all agents. Defines rules for embedding consistency,
+  namespace discipline, staleness management, orphan prevention, agent
+  custodianship, and recovery protocols. Prevents knowledge graph degradation
+  and ensures reliable semantic search capabilities.
+---
+
+# RA-H OS Custodianship Contract — Shared Memory Protocol
+
+## Purpose
+This contract establishes the operational standards for maintaining RA-H OS as a high-functioning shared memory system across all agents. It defines the rules, responsibilities, and recovery protocols that prevent knowledge graph degradation and ensure consistent, reliable semantic search capabilities.
+
+## Core Principles
+
+### 1. Embedding Consistency
+**Rule:** All nodes MUST use the RA-H OS built-in embedding pipeline via `http://192.168.68.65:8080/v1/embeddings` for consistent vector representation.
+
+**Requirements:**
+- The embedding service at `192.168.68.65:8080` is the single source of truth for vector generation
+- No external embedding APIs (OpenAI, Cohere, etc.) may be used for RA-H OS nodes
+- All nodes must have `embedding_status: "chunked"` and `chunk_status: "chunked"` upon creation
+- If embedding fails, the node must be immediately flagged as `state: "unsearchable"` and logged
+
+### 2. Namespace Discipline
+**Rule:** All nodes MUST be categorized into one of three namespaces with strict metadata requirements.
+
+**Namespace Structure:**
+- `agent-private`: Isolated working notes for individual agents (Mumuni, Tanko, Okyeame, etc.)
+- `shared`: Policy files, registry nodes, collective knowledge
+- `syslogsolution`: Business nodes, client data, operational context
+
+**Metadata Requirements:**
+Every node MUST have these metadata fields:
+```json
+{
+  "agent_id": "<agent_name>",
+  "namespace": "<namespace>",
+  "state": "<state>",
+  "type": "<type>",
+  "owner": "<owner>",
+  "tenant": "<tenant>",
+  "visibility": "<visibility>",
+  "source": "<source>",
+  "captured_by": "<captured_by>"
+}
+```
+
+### 3. Staleness Management
+**Rule:** Nodes are categorized by type and have specific staleness thresholds.
+
+**Staleness Thresholds:**
+- `policy` type: 30 days (e.g., Shared Memory Policy, Agent Registry)
+- `registry` type: 30 days (e.g., Node State Management, Health Dashboard)
+- `template` type: 30 days (e.g., Agent SOUL.md)
+- `note` type: 60 days (e.g., business context, project notes)
+- `information` type: 90 days (e.g., reference documentation)
+- `idea` type: 30 days (e.g., brainstorming, experimental notes)
+
+**Transition Protocol:**
+- `active` → `stale`: After days since `updated_at` exceeds threshold
+- `stale` → `review_pending`: Daily cron check flags for human review
+- `review_pending` → `active`: Human reviews and updates content
+- `review_pending` → `archived`: After 15 days in review_pending state (automatic purge)
+
+### 4. Orphan Prevention
+**Rule:** No node should remain orphaned (0 edges) for more than 7 days.
+
+**Orphan Detection:**
+- Daily cron job identifies nodes with 0 incoming/outgoing edges
+- Orphans are flagged with `state: "review_pending"` and `metadata.orphan_detected: true`
+- 7-day grace period: Orphans must be either:
+  - Connected to relevant nodes via edges
+  - Merged into existing parent nodes
+  - Archived after 15 days in review_pending state
+
+### 5. Agent Custodianship
+**Rule:** Each agent is responsible for maintaining their own created nodes.
+
+**Responsibilities:**
+- **Mumuni**: Primary custodian for `syslogsolution` namespace, health dashboard, and all business nodes
+- **Tanko**: Custodian for `agent-private` namespace, personal notes, and fitness/creative work
+- **Okyeame**: Custodian for `shared` namespace, policy files, and collective knowledge nodes
+- **All Agents**: Must verify embedding status before reporting node creation as complete
+
+### 6. Embedding Failure Recovery
+**Rule:** When embedding pipeline is unavailable, nodes are blocked from creation or immediately flagged.
+
+**Recovery Protocol:**
+1. **Detection**: Daily health check monitors `http://192.168.68.65:8080/health` endpoint
+2. **Alert**: If embedding service is down, alert is sent to all agents via relay
+3. **Mitigation**: 
+   - New node creation is suspended
+   - Existing nodes with `chunk_status: "not_chunked"` are flagged as `unsearchable`
+4. **Recovery**: When service returns:
+   - All `not_chunked` nodes are queued for re-embedding
+   - Re-embedding status logged in `embedding_retry_log` table
+   - 5-minute retry window with exponential backoff (1s, 2s, 4s, 8s, 16s)
+
+### 7. Chunking Verification
+**Rule:** All node modifications must verify chunking status within 5 minutes.
+
+**Verification Protocol:**
+- After `updateNode` or `createNode`, query `chunks` table for `node_id`
+- If `chunk_status != "chunked"` after 5 minutes, log failure and flag node
+- Automated cleanup script runs every 6 hours to retry failed chunks
+- Maximum 3 retry attempts per node before escalation to human
+
+### 8. Graph Health Monitoring
+**Rule:** Health dashboard (Node #74) must reflect real-time graph status.
+
+**Dashboard Requirements:**
+- Total nodes, edges, orphans, stale nodes, chunk errors
+- Embedding service health status
+- Last successful embedding attempt
+- Alert thresholds:
+  - Orphan rate > 40%: Warning
+  - Stale node rate > 50%: Critical
+  - Chunk error rate > 30%: Critical
+  - Embedding service down: Critical (immediate alert)
+
+## Operational Procedures
+
+### Daily Maintenance (Automated)
+1. **Staleness Check**: Identify nodes past their threshold
+2. **Orphan Detection**: Flag nodes with 0 edges
+3. **Chunk Verification**: Retry failed chunks, log errors
+4. **Health Update**: Refresh Node #74 dashboard
+
+### Weekly Maintenance (Human Review)
+1. **Review Pending Nodes**: Examine flagged nodes
+2. **Edge Optimization**: Connect related orphaned nodes
+3. **Archive Cleanup**: Remove nodes past 15-day review period
+4. **Embedding Audit**: Verify all nodes have valid embeddings
+
+### Emergency Recovery
+1. **Embedding Service Down**: 
+   - Suspend node creation
+   - Alert all agents
+   - Investigate root cause (check `192.168.68.65:8080`)
+2. **Database Corruption**:
+   - Restore from latest PBS backup
+   - Verify chunk integrity
+   - Re-run embedding for affected nodes
+3. **Mass Orphan Creation**:
+   - Identify source agent/namespace
+   - Review recent changes
+   - Reconnect or archive affected nodes
+
+## Compliance & Enforcement
+
+### Audit Schedule
+- **Daily**: Automated health checks
+- **Weekly**: Human review of review_pending nodes
+- **Monthly**: Comprehensive graph audit (full schema validation)
+- **Quarterly**: Contract review and threshold adjustment
+
+### Violation Consequences
+- **First**: Alert to responsible agent
+- **Second**: Node placed in `review_pending` state
+- **Third**: Agent access suspended until remediation
+- **Fourth**: Escalation to human (Kwame)
+
+### Metrics for Success
+- Orphan rate: < 10%
+- Stale node rate: < 20%
+- Chunk error rate: < 5%
+- Embedding service uptime: > 99.9%
+- Node creation verification: 100%
+
+## Implementation Notes
+
+### Database Schema Requirements
+- `nodes` table: Standard fields plus `embedding_status`, `chunk_status`, `last_embedding_attempt`
+- `chunks` table: Must track `chunk_status` and `embedding_status`
+- `embedding_retry_log` table: Track retry attempts for failed chunks
+- `node_state_history` table: Log state transitions for audit trail
+
+### Cron Job Requirements
+- `ra-h-health-check`: Daily (4h interval)
+- `ra-h-staleness-detection`: Daily
+- `ra-h-orphan-detection`: Daily
+- `ra-h-chunk-retry`: Every 6 hours
+- `ra-h-review-cleanup`: Weekly (15-day archive)
+
+### Agent Onboarding
+All new agents must:
+1. Read this contract
+2. Understand namespace responsibilities
+3. Know how to verify embedding status
+4. Know the alert escalation path
+
+---
+
+## Approval
+This contract is effective immediately upon approval by the primary custodian (Mumuni) and system owner (Kwame).
+
+**Effective Date:** July 10, 2026  
+**Review Date:** October 10, 2026  
+**Primary Custodian:** Mumuni 🦅  
+**System Owner:** Jerome Tabiri
+
+---
+
+*Contract Version: 1.0*  
+*Last Updated: July 10, 2026*  
+*Storage: `/root/.hermes/skills/ra-h-os-custodianship-contract.prose.md`*