merge: resolve conflicts with master (delegation delete + gpu-fleet)

Reviewed-on: #18

cron-prompts-review.md

@@ -0,0 +1,612 @@
+# Cron Prompts Review — All 10 Scheduled Contracts
+
+Generated: 2026-07-13 20:59:18 ET
+
+---
+
+## hermes-key-enforcement
+
+**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 6 * * *
+
+```
+Contract Enforcement: hermes-key-enforcement
+
+Category: compliance
+Domain: hermes-agent
+Owner: abiba
+Schedule: Daily compliance scan at 6am ET
+
+This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: hermes-key-enforcement.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/hermes-key-enforcement/
+
+Postconditions to verify:
+[
+  {
+    "check": "no plaintext API keys in config",
+    "verify": "grep -rc 'api_key: sk-' /root/.hermes/config.yaml",
+    "expect": "0 matches"
+  },
+  {
+    "check": "api_key_env used for harness/litellm providers",
+    "verify": "grep -c 'api_key_env.*LITELLM_API_KEY' /root/.hermes/config.yaml",
+    "expect": "count > 0"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + pause
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/hermes-key-enforcement/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-key-enforcement, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## hermes-config-template
+
+**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 4 * * 1
+
+```
+Contract Enforcement: hermes-config-template
+
+Category: compliance
+Domain: hermes-agent
+Owner: abiba
+Schedule: Weekly config drift check Monday at 4am ET
+
+This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: hermes-config-template.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/hermes-config-template/
+
+Postconditions to verify:
+[
+  {
+    "check": "agent config template_version matches template file",
+    "verify": "grep -q 'template_version' /root/.hermes/config.yaml && diff <(grep 'template_version' /root/.hermes/config.yaml | cut -d: -f2 | xargs) <(grep 'template_version' /root/prose-contracts/hermes-config-template.prose.md | cut -d: -f2 | xargs) && echo match || echo mismatch",
+    "expect": "match"
+  },
+  {
+    "check": "config file is valid YAML",
+    "verify": "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
+    "expect": "valid"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/hermes-config-template/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-config-template, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## hermes-agent-baseline
+
+**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 5 * * 1
+
+```
+Contract Enforcement: hermes-agent-baseline
+
+Category: compliance
+Domain: hermes-agent
+Owner: abiba
+Schedule: Weekly baseline verification Monday at 5am ET
+
+This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: hermes-agent-baseline.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/hermes-agent-baseline/
+
+Postconditions to verify:
+[
+  {
+    "check": "Hermes agent process running",
+    "verify": "pgrep -f 'hermes' > /dev/null && echo running || echo stopped",
+    "expect": "running"
+  },
+  {
+    "check": "agent config file exists and valid YAML",
+    "verify": "test -f /root/.hermes/config.yaml && python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
+    "expect": "valid"
+  },
+  {
+    "check": "no uncommitted changes in hermes directory",
+    "verify": "cd /root/.hermes && git status --porcelain | wc -l",
+    "expect": "0"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/hermes-agent-baseline/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-agent-baseline, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## proxmox-monitor
+
+**Category:** monitoring | **Domain:** proxmox | **Owner:** abiba | **Schedule:** */15 * * * *
+
+```
+Contract Enforcement: proxmox-monitor
+
+Category: monitoring
+Domain: proxmox
+Owner: abiba
+Schedule: Every 15 minutes
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: proxmox-monitor.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/proxmox-monitor/
+
+Postconditions to verify:
+[
+  {
+    "check": "all Proxmox nodes reachable",
+    "verify": "curl -sf http://192.168.68.10:8006/api2/json/status | jq '.status'",
+    "expect": "healthy"
+  },
+  {
+    "check": "no VMs in crashed state",
+    "verify": "pvesh get /nodes -output-format=json | jq '.[] | select(.status==\"Crashed\")'",
+    "expect": "empty"
+  },
+  {
+    "check": "backups running on schedule",
+    "verify": "pbs-info --check",
+    "expect": "last_backup < 24h ago"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/proxmox-monitor/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=proxmox-monitor, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## gpu-monitor
+
+**Category:** monitoring | **Domain:** gpu | **Owner:** abiba | **Schedule:** */15 * * * *
+
+```
+Contract Enforcement: gpu-monitor
+
+Category: monitoring
+Domain: gpu
+Owner: abiba
+Schedule: Every 15 minutes — polls all GPU subsystems
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: gpu-monitor.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/gpu-monitor/
+
+Postconditions to verify:
+[
+  {
+    "check": "GPU metrics accessible",
+    "verify": "curl -sf http://localhost:9100/gpu-data",
+    "expect": "200 OK, populated data"
+  },
+  {
+    "check": "dashboard serving",
+    "verify": "curl -sf http://localhost:9100/gpu-fleet.html",
+    "expect": "200 OK, HTML returned"
+  },
+  {
+    "check": "health endpoint responsive",
+    "verify": "curl -sf http://localhost:9100/health",
+    "expect": "200 OK"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/gpu-monitor/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=gpu-monitor, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## infrastructure-monitoring
+
+**Category:** monitoring | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** */30 * * * *
+
+```
+Contract Enforcement: infrastructure-monitoring
+
+Category: monitoring
+Domain: infrastructure
+Owner: abiba
+Schedule: Every 30 minutes
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: infrastructure-monitoring.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/infrastructure-monitoring/
+
+Postconditions to verify:
+[
+  {
+    "check": "Proxmox API reachable",
+    "verify": "curl -sf http://192.168.68.10:8006/api2/json",
+    "expect": "200 OK"
+  },
+  {
+    "check": "Zulip API reachable",
+    "verify": "curl -sf https://chat.sysloggh.net/api/v1/me",
+    "expect": "200 OK"
+  },
+  {
+    "check": "LiteLLM proxy reachable",
+    "verify": "curl -sf http://192.168.68.116/litellm/v1/models",
+    "expect": "200 OK"
+  },
+  {
+    "check": "Gitea API reachable",
+    "verify": "curl -sf https://git.sysloggh.net/api/v1/version",
+    "expect": "200 OK"
+  },
+  {
+    "check": "SearXNG reachable",
+    "verify": "curl -sf http://192.168.68.17:8080",
+    "expect": "200 OK"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/infrastructure-monitoring/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-monitoring, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## zulip-health
+
+**Category:** monitoring | **Domain:** zulip | **Owner:** abiba | **Schedule:** */15 * * * *
+
+```
+Contract Enforcement: zulip-health
+
+Category: monitoring
+Domain: zulip
+Owner: abiba
+Schedule: Every 15 minutes — monitors all Zulip-connected agents
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: zulip-health.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/zulip-health/
+
+Postconditions to verify:
+[
+  {
+    "check": "bot registration active",
+    "verify": "curl -sf https://chat.sysloggh.net/api/v1/me | jq '.user_id'",
+    "expect": "bot_id present"
+  },
+  {
+    "check": "DM delivery working",
+    "verify": "curl -sf https://chat.sysloggh.net/api/v1/users/me/is-online",
+    "expect": "online: true"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/zulip-health/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=zulip-health, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## litellm-health
+
+**Category:** monitoring | **Domain:** litellm | **Owner:** abiba | **Schedule:** */10 * * * *
+
+```
+Contract Enforcement: litellm-health
+
+Category: monitoring
+Domain: litellm
+Owner: abiba
+Schedule: Every 10 minutes — LiteLLM proxy health
+
+This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
+
+Steps:
+1. Load contract from prose-contracts/main (file: litellm-health.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/litellm-health/
+
+Postconditions to verify:
+[
+  {
+    "check": "LiteLLM proxy reachable",
+    "verify": "curl -sf http://192.168.68.116/litellm/v1/models",
+    "expect": "200 OK, models returned"
+  },
+  {
+    "check": "router deprecated, nginx routes work",
+    "verify": "curl -sf https://litellm.sysloggh.net/v1/models",
+    "expect": "200 OK (via nginx)"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba, mumuni → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/litellm-health/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=litellm-health, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## memory-audit-maintenance
+
+**Category:** maintenance | **Domain:** memory | **Owner:** mumuni | **Schedule:** 0 3 * * *
+
+```
+Contract Enforcement: memory-audit-maintenance
+
+Category: maintenance
+Domain: memory
+Owner: mumuni
+Schedule: Daily at 3am ET
+
+This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: memory-audit-maintenance.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/memory-audit-maintenance/
+
+Postconditions to verify:
+[
+  {
+    "check": "memory files below 80% capacity",
+    "verify": "wc -l ~/.hermes/memories/*.md",
+    "expect": "total lines < threshold"
+  },
+  {
+    "check": "no stale entries",
+    "verify": "grep -r 'STALE' ~/.hermes/memories/",
+    "expect": "0 matches"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify mumuni → action: relay_alert
+- CRITICAL: notify mumuni, abiba → action: relay_alert
+- FATAL: notify mumuni, abiba, kwame → action: relay_alert + human_required
+
+Circuit Breaker:
+- Max retries: 3 per 60.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/memory-audit-maintenance/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=memory-audit-maintenance, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+---
+
+## infrastructure-update
+
+**Category:** maintenance | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** 0 2 * * 0
+
+```
+Contract Enforcement: infrastructure-update
+
+Category: maintenance
+Domain: infrastructure
+Owner: abiba
+Schedule: Weekly system updates Sunday at 2am ET
+
+This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
+
+Steps:
+1. Load contract from prose-contracts/main (file: infrastructure-update.prose.md)
+2. Verify prerequisites (connectivity, tools, deps)
+3. Execute contract per SOP
+4. Run postconditions from contract registry
+5. Generate receipt with status (pass/fail/escalated)
+6. If any postcondition fails, escalate per contract escalation tiers
+7. Log to ~/.hermes/runs/infrastructure-update/
+
+Postconditions to verify:
+[
+  {
+    "check": "all services running after update",
+    "verify": "systemctl list-units --state=running",
+    "expect": "all critical services"
+  }
+]
+
+Escalation Tiers:
+- INFO: notify nobody → action: log_to_receipt
+- WARNING: notify abiba → action: relay_alert
+- CRITICAL: notify abiba, mumuni → action: relay_alert
+- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
+
+Circuit Breaker:
+- Max retries: 1 per 120.0min window
+- On trip: escalate_to_fatal
+
+Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
+Receipt storage: ~/.hermes/runs/infrastructure-update/receipt-{timestamp}.json
+Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-update, status=<status>
+
+If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
+
+IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
+```
+
+
+---
+*End of review*

delegation-prose-contract.prose.md

@@ -1,256 +0,0 @@
----
-kind: pattern
-name: delegation-prose-contract
-description: >
-  Manager (Mumuni) operating doctrine for task decomposition, worker
-  delegation, verification, and delivery. Defines when to delegate, which
-  worker to use for what, how to handle failures, and the kanban board
-  protocol. Enforces context-window discipline and separation of concerns.
-  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
-version: 1.0.0
----
-
-## Maintains
-
-- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
-  `syslog-research`, `syslog-review`, `syslog-writer`)
-- Kanban board state at `~/.hermes/kanban/kanban.json`
-- Context window budget: ~65K tokens per request (131K total, 60% threshold)
-
-## Topology
-
-**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
-**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
-**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
-
-This contract is infrastructure-agnostic in terms of which nodes are used.
-Workers execute tasks on whatever infrastructure they're given — SSH to .6,
-.pm, .9, .12, or .15 depending on the task. The contract defines the
-**who** and **when** — not the **where**.
-
-## Why This Matters
-
-Without enforced delegation, the manager consumes the full iteration budget
-(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
-leaving no capacity for actual coordination. The result: context overflow
-(59K tokens in system prompt), iteration exhaustion, and degraded response
-quality. This contract exists because I blew through my budget checking
-Proxmox node status instead of delegating to `syslog-devops`.
-
-## Context Window Discipline
-
-**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
-**Volatile (MEMORY.md + USER.md): ~300 tokens.**
-**Total base: ~6,800 tokens per request.**
-
-The remaining budget is the conversation. Every tool call result adds to it.
-If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
-from multiple nodes), the context fills fast. That's why we delegate: workers
-process in isolation and return compact results.
-
-## Trigger Conditions
-
-Delegation is **mandatory** when any of these apply:
-
-| Condition | Threshold | Example |
-|-----------|-----------|---------|
-| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
-| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
-| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
-| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
-| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
-| Code builds or changes | Writing or modifying code | Scripts, configs, patches |
-| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
-
-**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
-`curl`, `hermes tools list` — these are decision-making tools. The manager
-reads them directly.
-
-## Worker Selection Matrix
-
-| Worker | Model | Toolsets | Role | Use When |
-|--------|-------|----------|------|----------|
-| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
-| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
-| `syslog-email` | strix-moe | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
-| `syslog-research` | strix-moe | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
-| `syslog-review` | strix-moe | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
-| `syslog-writer` | strix-moe | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
-
-### Selection Rules
-
-1. **Match specialty first.** A code task → `syslog-code`. An infra task →
-   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
-2. **Research tasks with browser needs → `syslog-research`.** Other workers
-   don't have the browser toolset.
-3. **Verification → `syslog-review`.** Never deliver raw worker output.
-4. **Documentation/content → `syslog-writer`.** Let them own the prose.
-5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
-   (includes browser) and high reasoning effort.
-
-## Delegation Protocol
-
-### Step 1: Decompose
-
-Break the task into lanes. Each lane does ONE thing. Workers are independent —
-no lane depends on another's output mid-flight. If lanes depend on each other,
-dispatch sequentially.
-
-### Step 2: Dispatch
-
-Fire workers via `delegate_task`:
-
-**Parallel (independent lanes):**
-```
-delegate_task(
-    tasks=[
-        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
-        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
-    ]
-)
-```
-
-**Sequential (dependent lanes):**
-Dispatch lane 1 → wait for result → dispatch lane 2.
-
-### Step 3: Verify
-
-**MANDATORY for:**
-- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
-- Code builds and modifications
-- Research findings (web data, external sources)
-- Any output that will reach the user
-
-**Fire `syslog-review` to verify:**
-```
-delegate_task(
-    goal="Review the output of the devops worker. Verify the node status
-    report is accurate, check for inconsistencies, confirm all nodes were
-    reachable.",
-    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
-    Verify against live system."
-)
-```
-
-**If verification fails:**
-1. Send work back to original worker with review feedback
-2. Re-verify
-3. Max 2 re-verify cycles before escalating to Kwame
-
-### Step 4: Deliver
-
-Only verified results reach Kwame. Format per channel:
-- Telegram: Use `telegram-formatting` skill
-- Zulip: Use Zulip Markdown (CommonMark)
-- Email: Use `syslog-email` skill
-
-## Kanban Board Protocol
-
-**File:** `~/.hermes/kanban/kanban.json`
-
-```json
-{
-  "task_id": "unique-id",
-  "title": "Task description",
-  "created": "2026-07-09T01:00:00",
-  "status": "backlog|in_progress|review|done",
-  "lanes": [
-    {
-      "lane_id": "devops-check",
-      "worker": "syslog-devops",
-      "goal": "Check all 5 Proxmox nodes",
-      "status": "dispatched|completed|failed",
-      "output_file": "/tmp/node-report.md"
-    }
-  ]
-}
-```
-
-**Update the board on every state change.**
-
-## Failure Handling
-
-### Worker Timeouts
-
-- Child timeout: **900 seconds** (15 minutes)
-- Worker model `syslog-auto` is slow — it can hit the timeout limit with
-  22+ API calls
-- **If a worker times out:** Re-dispatch with a narrower scope. Break the
-  task into smaller pieces that fit in the timeout window.
-- **Avoid delegating sequential SSH hops** — each SSH connection adds latency
-  that compounds quickly. Prefer API-based or local approaches when possible.
-
-### Worker Selection Failures
-
-- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
-- `syslog-code` is best for code-level work (reading files, writing scripts)
-- `syslog-research` has the browser toolset — use for web research
-- `syslog-review` is the QA gate — always fire before delivery
-- **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
-- **Never nest delegation** (max_spawn_depth: 1)
-
-### Context Overflow
-
-- If a task requires >10K tokens of output, delegate the processing
-- Workers return compact summaries, not raw data dumps
-- Pass file paths and concrete goals — never dump raw data into context
-
-## Anti-patterns
-
-- ❌ Reading large files into your own context before deciding → delegate the read
-- ❌ Carrying SSH/grep/output results in your context → delegate the analysis
-- ❌ Doing work yourself and then "pretending" to delegate → the user can tell
-- ❌ Skipping verification → raw worker output never reaches the user
-- ❌ Delegating single tool calls → keep quick reads/writes at manager level
-- ❌ Firing more than 3 workers in parallel → hard limit
-
-## Emergency Exception
-
-**In an emergency (server down, service must be restored immediately):**
-- Delegate the diagnosis (find the problem)
-- Execute the fix yourself (minimize handoff latency)
-- Verify the fix after delivery
-- Log the exception in the kanban board
-
-The emergency exception exists because the user needs the service back NOW,
-not after three worker round-trips. But it's an exception — not the rule.
-
-## What This Contract Doesn't Cover
-
-1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
-2. **SSH key management** — covered by existing SSH/Proxmox contracts
-3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
-4. **Cron job management** — covered by individual cron contracts
-5. **Infra verification** — covered by `verify-before-mutate` protocol
-
-## Verification
-
-Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
-
-```bash
-python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
-```
-
-## References
-
-- `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
-- `worker-profile-audit.md` (skill reference): Worker configuration audit notes
-- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
-- `verify-before-mutate` protocol: Infrastructure change verification
-- `hermes-config-template.prose.md`: Worker profile configuration
-
-## Success Criteria
-
-This contract succeeds when:
-
-1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
-2. **Workers do the work** — manager coordinates, doesn't execute
-3. **Verification before delivery** — all output passes through `syslog-review`
-4. **Kanban board is current** — every task has a lane, every lane has a status
-5. **User gets verified results** — raw worker output never reaches Kwame
-
----
-
-**Last updated:** 2026-07-09
-**Author:** Mumuni (with Kwame's input on triggers and exception criteria)
-**Status:** Draft — awaiting PR review and merge to prose-contracts main

gpu-fleet.prose.md

@@ -7,7 +7,7 @@ description: >
   saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
   UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe,
   gpu-dense, gpu-light. These never change — only the underlying model does.
-  Strix Halo: ornith-1.0-35b → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
+  Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
   RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
   RTX 5070 context: 131K → 256K. VRAM: 88% (10.8/12.2GB).
   Compression timeout: 300s (was 120s). Mumuni context: 128K (was 256K).
@@ -186,7 +186,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
 5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
-   - gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (ornith-1.0-35b does NOT exist — legacy name, do not use)
+   - gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (strix-moe does NOT exist — legacy name, do not use)
    - global request_timeout: 300s, nginx proxy_read_timeout: 600s
 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
 7. Check port conflicts: verify only one llama-server on :8080 per host
@@ -314,7 +314,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
 | `personalities` | `creative` | Creative assistant personality |
 | Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
 | Main model timeout | 300s | LiteLLM global timeout |
-| Compression model timeout | 300s | ornith timeout increased from 120s |
+| Compression model timeout | 300s | strix-moe timeout increased from 120s |
 
 ### Agent Update Status (2026-07-15)
 

hermes-agent-baseline.prose.md

@@ -5,7 +5,7 @@ version: 1.0.0
 description: >
   Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
   configuration state, keys, workarounds, and audit procedure. When an agent's
-  configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
+  configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
 author: Abiba (pi agent)
 ---
 
@@ -26,11 +26,11 @@ done
 |-------|-----|------|-----|---------------|------------|----------|
 | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
 | Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
-| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
+| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
-| Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
+| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
-| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes |
+| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
 
-> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
+> **Note**: CT hostnames (tdunna→CT129, baggy→CT114) differ from agent identities (koby, koonimo).
 
 Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
 Keys are stored in Infisical vault (project=agents, env=production) and injected at
@@ -169,9 +169,9 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
 # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
 ```
 
-### For Koby (CT 111 / tdunna)
+### For Koby (CT 129 / tdunna)
 
-Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
+Koby runs Hermes on CT 129 (tdunna). Config files at `/root/.hermes/config.yaml`.
 Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
 
 **LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.

hermes-config-template.prose.md

@@ -143,23 +143,25 @@ compression:
 
 # ─── Auxiliary Tasks (CONSISTENCY RULE) ───
 # All auxiliary services MUST use identical model, base_url, and api_key_env:
-#   model: gemma-4-12b
+#   model: gpu-light                 # stable alias (NOT raw "gemma-4-12b")
 #   base_url: http://192.168.68.116/v1
 #   api_key_env: LITELLM_API_KEY
 # Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
-# gemma-4-12b is a lightweight 12B model on the RTX 5070, freeing the Strix Halo
+# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
-# for agent reasoning.
+# Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
+# NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
+# in agent configs — use the stable aliases so model swaps don't break agents.
 auxiliary:
   vision:
     provider: harness
-    model: gemma-4-12b
+    model: gpu-light            # stable alias for RTX 5070 (was raw gemma-4-12b)
     base_url: http://192.168.68.116/v1
     api_key_env: LITELLM_API_KEY
     timeout: 60
     download_timeout: 30
   web_extract:
     provider: harness
-    model: gemma-4-12b
+    model: gpu-light            # stable alias for RTX 5070
     base_url: http://192.168.68.116/v1
     api_key_env: LITELLM_API_KEY
     timeout: 30
@@ -170,10 +172,19 @@ auxiliary:
     api_key_env: LITELLM_API_KEY
     timeout: 300                 # gpu-fleet: 300s for large-history summarization (was 60)
 
+# ─── Delegation / Heavy Aux (use gpu-dense = RTX 3090) ───
+# delegation.model and x_search.model use gpu-dense (NOT raw qwen3.6-27B-code).
+
+delegation:
+  model: gpu-dense              # stable alias for RTX 3090 (was raw qwen3.6-27B-code)
+  provider: harness
+  base_url: http://192.168.68.116/v1
+  api_key_env: LITELLM_API_KEY
+
 # ─── Custom Provider ───
 custom_providers:
   - name: harness
-    model: <agent_model>
+    model: syslog-auto            # weighted pool (default)
     base_url: http://192.168.68.116/v1
     api_key_env: LITELLM_API_KEY
     api_mode: chat_completions

hermes-key-enforcement.prose.md

@@ -72,6 +72,11 @@ External providers are **explicitly exempt** and may use hardcoded keys:
 
 ## Standard Pattern
 
+> **Canonical vault process (2026-07-16):** see `litellm-api-keys` § Production Vault Access Process.
+> All agents MUST use the `infisical-gateway.sh` wrapper (live vault injection). Hardcoded systemd
+> drop-ins / config.yaml keys are DEPRECATED — they rot on rotation (root cause of the 2026-07-16 401 storm).
+> 4/5 agents migrated; tanko (user jerome) pending.
+
 ```yaml
 # ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
 model:

litellm-api-keys.prose.md

@@ -14,9 +14,15 @@ description: >
   expire/404. The .env fallback prevents agents from running without keys.
   Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
   
+  UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
+  token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
+  infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
+  Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
+  Abiba's key is now a proper agent key (NOT the master key — stale note removed).
+  
   Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
   Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
-  on CT 116. Last verified: 2026-07-12.
+  on CT 116. Last verified: 2026-07-16.
 ---
 
 ## Parameters
@@ -68,6 +74,97 @@ description: >
    - Confirm key alias matches agent_name in LiteLLM key list
    - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
 
+## Production Vault Access Process (canonical, 2026-07-16)
+
+The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending —
+runs as user `jerome`, not systemd root, needs user-scope adaptation).
+
+### The canonical pattern
+
+1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`).
+2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`).
+   - Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project.
+   - Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).
+3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`):
+   ```bash
+   #!/bin/bash
+   export INFISICAL_API_URL="https://vault.sysloggh.net"
+   TOKEN=$(cat /root/.infisical-token)
+   LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs
+   while true; do
+     infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
+       --env=prod --domain=https://vault.sysloggh.net -- bash -c '
+         . /root/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3] safety net only
+         export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"
+         exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run
+       ' >> $LOG 2>&1
+     sleep 5   # restart on exit
+   done
+   ```
+4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth.
+5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation.
+6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot).
+7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault.
+
+### Why this is non-fail
+
+- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
+- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable.
+- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway.
+- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source.
+
+### Migration status (2026-07-16)
+
+| Agent | Host | Pattern | Vault key | Status |
+|-------|------|---------|-----------|--------|
+| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed |
+| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed |
+| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram |
+| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
+> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent.
+<<<<<<< HEAD
+| tanko | .122 | infisical-gateway.sh + service token (migrated 2026-07-16) | TANKO_LITELLM_API_KEY | ✅ vault-backed |
+=======
+| tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |
+>>>>>>> origin/master
+
+### Tanko migration (pending)
+
+Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in
+`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced).
+Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`)
+with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering
+enabled (`loginctl enable-linger jerome`) so the user service runs without a login session.
+
+### Koby migration lessons (2026-07-16)
+
+Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
+**Two mistakes I made that broke the agent:**
+1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
+   in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
+   inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
+2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
+   Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
+
+**How Koby actually connects (2026-07-16):**
+- Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
+  Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config).
+- Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026).
+  Allowed users: 6679773481. Home channel: 6679773481.
+- Both platforms now connect through the wrapper's env injection.
+
+**Golden rule for gateway restarts:** always `cat /proc/<pid>/environ` before killing the old
+process — captures the live env set. Especially important when migrating gateways between
+injection mechanisms.
+
+### Key rotation procedure (one vault operation with this standard)
+
+1. Generate new key: `POST /key/generate` (master key, admin).
+2. Update vault: `infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`.
+3. Update `.env` fallback: `echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`.
+4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
+5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
+
 ## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
 
 **Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
@@ -90,19 +187,47 @@ TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth
   -d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
 # OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
 ```
+<<<<<<< HEAD
+**2026-07-16 UPDATE — vault SYNCED + CLEANED.** The abiba service token (`st.8e848433…`, READ+WRITE)
+can write to the vault. All session-13 rotated keys in vault and validate 200 against LiteLLM.
+4 stale secrets deprecated, 5 personal creds flagged for separate project. Tanko migrated from
+hardcoded keys to `infisical-gateway.sh` wrapper. Koonimo Zulip key restored.
+
+**Machine Identity:** `8ddb9438-74fc-4ab6-bd74-929e7c47b53b` exists in Infisical UI.
+Client secret needed to use universal auth for vault writes from automation.
+
+**Service Token Inventory (2026-07-16):**
+| Token ID | Name | Permissions | Used By |
+|----------|------|-------------|---------|
+| `st.8e848433…` | tanko-gateway | READ+WRITE | Abiba, Koonimo |
+| `st.353699cd…` | tanko-agent | READ-only | Tanko |
+=======
 Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings →
 Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity
 and store its credentials in the vault itself (or a root-only file).**
 
 **Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The
-infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `MUMUNI_LITELLM_API_KEY` overrides
+infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `<AGENT>_LITELLM_API_KEY`
-the stale vault value. This is fully functional and contract-sanctioned — the vault sync above is
+overrides a stale vault value.
-only for consistency so .env and vault never drift.
+
+**2026-07-16 UPDATE — vault is now SYNCED.** The abiba service token (`st.8e848433…`, READ+WRITE)
+can write to the vault, so the session-13 rotated keys (mumuni `sk-OzuWsoX2…`, koby `sk-BqRRMboTI…`,
+koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `KOBY_LITELLM_API_KEY` /
+`KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again.
+Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix
+so the shared service token isn't reused across hosts — but it is no longer blocking.
+>>>>>>> origin/master
 
 ## Key Rotation Log
 
 | Date | Agent | Action | Notes |
 |------|-------|--------|-------|
+<<<<<<< HEAD
+| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault (Tt2bUL…). Updated wrapper to inject ZULIP_API_KEY + ZULIP_EMAIL. Restarted gateway → Zulip connected as koonimo-bot@ (bot_id=17). 3 platforms now. |
+| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml key to infisical-gateway.sh wrapper + service token st.353699cd… (tanko-agent). Systemd user service updated, hardcoded key drop-ins removed. Verified LITELLM_API_KEY from vault, 3 platforms connected. |
+| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated: KAGENZ0_LITELLM_API_KEY, HERMES_OPENROUTER_KEY, LITELLM_API_KEY (generic duplicate), ZULIP_API_KEY (generic duplicate). 5 personal creds flagged for separate project. |
+=======
+>>>>>>> origin/master
 | 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
 | 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
 | 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
@@ -111,5 +236,8 @@ only for consistency so .env and vault never drift.
 
 - Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
 - Used for /key/generate, /key/delete, /key/list (GET), DB queries
-- **Known violation:** Abiba's LITELLM_API_KEY IS the master key (should be a dedicated `abiba` virtual key). TODO: generate `abiba` virtual key and stop using master key directly.
+- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
+It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
+The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference —
+see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference".
 - LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`

litellm-self-heal.prose.md

@@ -69,7 +69,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
 
 - `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200).
 - `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively.
-- Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. Note `qwen3.6-35B-udq4` is NOT in the agent-key allowlist — agents must use `strix-moe` to reach the AMD model.
+- Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. As of 2026-07-16 the `baggy`/`koby`/`mumuni`/`abiba-pi` keys ALSO include `qwen3.6-35B-udq4`; `abiba-pi` additionally includes `deepseek-v4-pro` (cloud fallback). `kagenz0`/`koonimo`/`pi-agents-unified` have the standard 6 only. Agents should still use the stable alias `strix-moe` (not the raw `qwen3.6-35B-udq4`) so model swaps don't break them.
+- **NEVER use `litellm_proxy_master_key` (the `sk-litellm-...` master key) for inference.** It is for admin endpoints only (`/key/list`, `/key/generate`, `/key/info`). All inference — agent traffic, health-check model tests, monitor scripts — uses agent-specific keys. The health-check script's model tests use a dedicated `monitor` agent key stored at `/etc/litellm-monitor.env` on CT 116 (root-only, `chmod 600`); `/key/list` is the only call that legitimately uses `$MASTER_KEY`.
 
 ## Model Fallback Chains (LiteLLM)