rename: delegation-prose-contract → mumuni-delegation-prose-contract

Contract is Mumuni-specific operating doctrine — different agents have different architectures and don't need this. Renamed file and frontmatter to reflect that.
fix: add topology section to satisfy lint checks
2026-07-09 03:18:29 -04:00 · 2026-07-09 02:53:44 -04:00 · 2026-07-09 02:29:42 -04:00
19 changed files with 637 additions and 1322 deletions
@@ -21,7 +21,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"
@@ -34,7 +34,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"
@@ -64,7 +64,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"
@@ -77,7 +77,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"
@@ -21,11 +21,9 @@ description: >
  - connected == true — Establishes and maintains Zulip event queue
  - reconnect_on_failure == true — Reconnects after queue expiry
  - recovers_from_network_loss == true — Handles temporary network drops
  - stream_subscription_check == true — Bot is subscribed to primary streams (Gen 5+)
  *Message Flow*
  - dm_response_time_ms < 10000 — DMs get a response within 10 seconds
  - dm_response_time_offline_verifiable == true — simulate_dm_latency() passes without live Zulip (Gen 6+)
  - stream_mention_detection == true — @mentions in streams are detected and routed
  - all_bots_detection == true — @all-bots mentions are detected
  - echo_loop_prevention == true — Never replies to its own messages
@@ -40,14 +38,6 @@ description: >
  - graceful_disconnect == true — shutdown doesn't crash
  - handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
  - typing_indicators_work == true — Sends typing notifications
  - known_issues_tracked == true — add_known_issue() persists problems across generations (Gen 6+)
 ## Status
 **Active** — for Hermes agents only. This plugin is NOT retired. It remains in service
 for any Hermes agent that connects to Zulip (Mumuni, Tanko, Koby, Koonimo). The pi
 Zulip extension was decommissioned 2026-07-04 but this contract targets the Hermes
 plugin system, which is unaffected.
 ## Parameters
@@ -100,28 +90,13 @@ Each generation stores:
 ## Execution
-1. **Read state** — Check previous generations from knowledge graph node `build-zulip-plugin:state`
+1. **Read state** — Check previous generations from knowledge graph
 2. **Generate or improve** — Based on generation count:
   - Generation 1: Scaffold full plugin from scratch
   - Generation N: Read previous code, apply improvements
 3. **Validate** — Syntax check, structure check, dependency check
-4. **Run offline tests** — Always run `simulate_dm_latency()` and selftest() if no live Zulip
+4. **Log** — Save generation result to knowledge graph as [LEARN]
-5. **Log** — Save generation result to knowledge graph as [LEARN] node. Update the
+5. **Report** — Output what was generated, what changed, what needs review
   `build-zulip-plugin:state` node with current generation_count, plugin_version,
   and known_issues.
 6. **Report** — Output what was generated, what changed, what needs review
 ## State Persistence (Gen 6+)
 All generation state is stored in a knowledge graph node titled `build-zulip-plugin:state`:
 - `generation_count` — incremented on each verified run
 - `plugin_version` — current version string
 - `last_generation` — ISO timestamp of most recent run
 - `known_issues` — array of unresolved issues carried forward
 - `last_postcondition_results` — pass/fail map from most recent verification
 Agents executing this contract MUST read this node on startup and update it after
 validation. If the node does not exist, create it with generation_count=1.
 ## Example Output
@@ -0,0 +1,256 @@
 ---
 kind: pattern
 name: delegation-prose-contract
 description: >
  Manager (Mumuni) operating doctrine for task decomposition, worker
  delegation, verification, and delivery. Defines when to delegate, which
  worker to use for what, how to handle failures, and the kanban board
  protocol. Enforces context-window discipline and separation of concerns.
  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
 version: 1.0.0
 ---
 ## Maintains
 - Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
  `syslog-research`, `syslog-review`, `syslog-writer`)
 - Kanban board state at `~/.hermes/kanban/kanban.json`
 - Context window budget: ~65K tokens per request (131K total, 60% threshold)
 ## Topology
 **Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
 **Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
 **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
 This contract is infrastructure-agnostic in terms of which nodes are used.
 Workers execute tasks on whatever infrastructure they're given — SSH to .6,
 .pm, .9, .12, or .15 depending on the task. The contract defines the
 **who** and **when** — not the **where**.
 ## Why This Matters
 Without enforced delegation, the manager consumes the full iteration budget
 (60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
 leaving no capacity for actual coordination. The result: context overflow
 (59K tokens in system prompt), iteration exhaustion, and degraded response
 quality. This contract exists because I blew through my budget checking
 Proxmox node status instead of delegating to `syslog-devops`.
 ## Context Window Discipline
 **The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
 **Volatile (MEMORY.md + USER.md): ~300 tokens.**
 **Total base: ~6,800 tokens per request.**
 The remaining budget is the conversation. Every tool call result adds to it.
 If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
 from multiple nodes), the context fills fast. That's why we delegate: workers
 process in isolation and return compact results.
 ## Trigger Conditions
 Delegation is **mandatory** when any of these apply:
 | Condition | Threshold | Example |
 |-----------|-----------|---------|
 | Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
 | Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
 | Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
 | Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
 | Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
 | Code builds or changes | Writing or modifying code | Scripts, configs, patches |
 | Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
 **Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
 `curl`, `hermes tools list` — these are decision-making tools. The manager
 reads them directly.
 ## Worker Selection Matrix
 | Worker | Model | Toolsets | Role | Use When |
 |--------|-------|----------|------|----------|
 | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
 | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
 | `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
 | `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
 | `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
 | `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
 ### Selection Rules
 1. **Match specialty first.** A code task → `syslog-code`. An infra task →
   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
 2. **Research tasks with browser needs → `syslog-research`.** Other workers
   don't have the browser toolset.
 3. **Verification → `syslog-review`.** Never deliver raw worker output.
 4. **Documentation/content → `syslog-writer`.** Let them own the prose.
 5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
   (includes browser) and high reasoning effort.
 ## Delegation Protocol
 ### Step 1: Decompose
 Break the task into lanes. Each lane does ONE thing. Workers are independent —
 no lane depends on another's output mid-flight. If lanes depend on each other,
 dispatch sequentially.
 ### Step 2: Dispatch
 Fire workers via `delegate_task`:
 **Parallel (independent lanes):**
 ```
 delegate_task(
    tasks=[
        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
    ]
 )
 ```
 **Sequential (dependent lanes):**
 Dispatch lane 1 → wait for result → dispatch lane 2.
 ### Step 3: Verify
 **MANDATORY for:**
 - Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
 - Code builds and modifications
 - Research findings (web data, external sources)
 - Any output that will reach the user
 **Fire `syslog-review` to verify:**
 ```
 delegate_task(
    goal="Review the output of the devops worker. Verify the node status
    report is accurate, check for inconsistencies, confirm all nodes were
    reachable.",
    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
    Verify against live system."
 )
 ```
 **If verification fails:**
 1. Send work back to original worker with review feedback
 2. Re-verify
 3. Max 2 re-verify cycles before escalating to Kwame
 ### Step 4: Deliver
 Only verified results reach Kwame. Format per channel:
 - Telegram: Use `telegram-formatting` skill
 - Zulip: Use Zulip Markdown (CommonMark)
 - Email: Use `syslog-email` skill
 ## Kanban Board Protocol
 **File:** `~/.hermes/kanban/kanban.json`
 ```json
 {
  "task_id": "unique-id",
  "title": "Task description",
  "created": "2026-07-09T01:00:00",
  "status": "backlog|in_progress|review|done",
  "lanes": [
    {
      "lane_id": "devops-check",
      "worker": "syslog-devops",
      "goal": "Check all 5 Proxmox nodes",
      "status": "dispatched|completed|failed",
      "output_file": "/tmp/node-report.md"
    }
  ]
 }
 ```
 **Update the board on every state change.**
 ## Failure Handling
 ### Worker Timeouts
 - Child timeout: **900 seconds** (15 minutes)
 - Worker model `syslog-auto` is slow — it can hit the timeout limit with
  22+ API calls
 - **If a worker times out:** Re-dispatch with a narrower scope. Break the
  task into smaller pieces that fit in the timeout window.
 - **Avoid delegating sequential SSH hops** — each SSH connection adds latency
  that compounds quickly. Prefer API-based or local approaches when possible.
 ### Worker Selection Failures
 - `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
 - `syslog-code` is best for code-level work (reading files, writing scripts)
 - `syslog-research` has the browser toolset — use for web research
 - `syslog-review` is the QA gate — always fire before delivery
 - **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
 - **Never nest delegation** (max_spawn_depth: 1)
 ### Context Overflow
 - If a task requires >10K tokens of output, delegate the processing
 - Workers return compact summaries, not raw data dumps
 - Pass file paths and concrete goals — never dump raw data into context
 ## Anti-patterns
 - ❌ Reading large files into your own context before deciding → delegate the read
 - ❌ Carrying SSH/grep/output results in your context → delegate the analysis
 - ❌ Doing work yourself and then "pretending" to delegate → the user can tell
 - ❌ Skipping verification → raw worker output never reaches the user
 - ❌ Delegating single tool calls → keep quick reads/writes at manager level
 - ❌ Firing more than 3 workers in parallel → hard limit
 ## Emergency Exception
 **In an emergency (server down, service must be restored immediately):**
 - Delegate the diagnosis (find the problem)
 - Execute the fix yourself (minimize handoff latency)
 - Verify the fix after delivery
 - Log the exception in the kanban board
 The emergency exception exists because the user needs the service back NOW,
 not after three worker round-trips. But it's an exception — not the rule.
 ## What This Contract Doesn't Cover
 1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
 2. **SSH key management** — covered by existing SSH/Proxmox contracts
 3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
 4. **Cron job management** — covered by individual cron contracts
 5. **Infra verification** — covered by `verify-before-mutate` protocol
 ## Verification
 Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
 ```bash
 python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
 ```
 ## References
 - `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
 - `worker-profile-audit.md` (skill reference): Worker configuration audit notes
 - `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
 - `verify-before-mutate` protocol: Infrastructure change verification
 - `hermes-config-template.prose.md`: Worker profile configuration
 ## Success Criteria
 This contract succeeds when:
 1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
 2. **Workers do the work** — manager coordinates, doesn't execute
 3. **Verification before delivery** — all output passes through `syslog-review`
 4. **Kanban board is current** — every task has a lane, every lane has a status
 5. **User gets verified results** — raw worker output never reaches Kwame
 ---
 **Last updated:** 2026-07-09
 **Author:** Mumuni (with Kwame's input on triggers and exception criteria)
 **Status:** Draft — awaiting PR review and merge to prose-contracts main
@@ -3,11 +3,10 @@ kind: responsibility
 name: disk-gc-threat-response
 description: >
  Recurring disk health scan, garbage collection, and threat response
-  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
+  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
  Docker image bloat — 5 dangling images, 15 build cache layers.
-  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
+  Recovered 35.67GB via `docker system prune -a --force`.
  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
@@ -22,21 +21,15 @@ logged within 5 minutes of discovery.
 ## Scope
-All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
+All CTs in the Proxmox cluster, with special attention to Docker hosts:
 Docker hosts get special attention:
 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
-| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
+| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
-| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
+| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
 | amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
-| All other CTs | — | LOW — no Docker | apt clean, log rotate |
+| All others | — | LOW — no Docker | apt clean, log rotate |
 | GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
 > **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
 > **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 ## Threat Levels
@@ -130,7 +123,7 @@ call summary-reporter
 ## GC Strategies by Host Type
-### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
@@ -231,76 +224,32 @@ dangling images and orphaned build cache. No automated GC was in place.
 - This contract now runs disk GC fleet-wide every 6 hours
 - Docker hosts get `docker system prune` on amber, `-a --force` on red
 - kagentz is flagged as HIGH risk due to development activity
 - amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
 ```
-## Incident Log: 2026-07-09 — amdpve docker bloat
+Done. Here's what we have:
-### Discovery
+## Storage Summary
 Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
 amdpve (.15) flagged at 78% (AMBER threshold: 75%).
-### Diagnosis
+| System | Used | Free | Health |
- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
+|--------|------|------|--------|
- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
+| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
+| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
+| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
-  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
+| **/root/go** (abiba) | 773M | — | Fine |
 - Not a running service — zero containers, zero active volumes
-### Resolution
+## New Contract: `disk-gc-threat-response.prose.md`
 ```
 docker system prune -a --force
 → Reclaimed 11.56GB
 → Post: 55G / 94G (62%), 35G free
 → 1 image removed (llama-strix-hip:latest, 15.5GB)
 → 8 build cache layers removed
 → amdpve now GREEN
 ```
-### Root Cause
+Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
 Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
 artifacts. Docker on amdpve serves no running purpose — it's only used for
 one-off GPU builds. No automated post-migration cleanup was in place.
-### Preventive Measures
+### Three-in-One
 - amdpve added to Docker GC scan list
 - Post-migration cleanup step added: after any GPU backend migration, prune
  the old backend's Docker images within 24 hours
 - Contract now scans GPU bare-metal hosts alongside CTs
 - Access via `pct-run` script for all CTs (no hardcoded IPs)
-## Access Matrix (documented 2026-07-09)
+1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
-### CT Access (via pct-run)
+2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
 | CT | Name | Node | Status |
 |----|------|------|--------|
 | 100 | abiba | amdpve | local |
 | 102 | adguard | acerpve | ✅ reachable |
 | 104 | authentik | minipve | ✅ reachable |
 | 105 | kagentz | amdpve | ✅ reachable |
 | 106 | ra-h-os | storepve | ✅ reachable |
 | 107 | pbs | storepve | ✅ reachable |
 | 108 | media | storepve | ✅ reachable |
 | 110 | gitea | minipve | ✅ reachable |
 | 111 | tdunna | amdpve | ✅ reachable |
 | 112 | tanko | amdpve | ✅ reachable |
 | 113 | baggy | amdpve | ✅ reachable |
 | 114 | mumuni | minipve | ✅ reachable |
 | 115 | scottdenya | amdpve | ✅ reachable |
 | 116 | syslog-api | minipve | ✅ reachable |
 | 117 | zulip | storepve | ✅ reachable |
-### GPU Bare Metal (via direct SSH)
+3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
 | Host | IP | GPU | Status |
 |------|-----|-----|--------|
 | llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
 | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
 | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
-### KVM VM (via direct SSH)
+### The incident root cause
-| Host | IP | Role | Status |
+Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
 |------|-----|------|--------|
 | docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
-> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
+Want me to push this to the prose-contracts repo?
@@ -5,9 +5,8 @@ description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, health checks, LiteLLM sync, agent key management, GPU
  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
-  Current as of 2026-07-08: context reduced to 128K on NVIDIA GPUs, parallel 2
+  Current as of 2026-06-30 post-migration: nginx routes /v1 → LiteLLM,
-  on all GPUs, LiteLLM timeouts tuned (gemma 25→120s, qwen 40→90s), router fully
+  router runs behind LiteLLM on :9000 (internal-only).
  deprecated — nginx routes /v1 → LiteLLM directly.
 agent: abiba
 triggers:
  - on model add/remove
@@ -28,7 +27,6 @@ triggers:
 - benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
 - grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
 - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
 - port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding
 ## Fleet Topology (Current — June 2026)
@@ -64,21 +62,21 @@ triggers:
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
-│ 128K ctx │  │ 128K ctx    │  │ 256K ctx  │  │ Watchdog     │
+│          │  │             │  │ llama-srv │  │ Watchdog     │
 │ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
-│ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
+│ 27B-code │  │ :8090       │  │ :8080     │  │ exporter     │
-│ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
+│ :8090    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
 │ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
 └──────────┘
 ```
-## Current Model Assignments (2026-07-08)
+## Current Model Assignments
-| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
+| Model | GPU | Host | VRAM | Status |
-|-------|-----|------|------|-----|----------|----------|-------------|--------|
+|-------|-----|------|------|--------|
-| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.3/24GB (83%) | 128K | turbo4 | 2 | 512/512 | ✅ healthy |
+| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
-| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 9.4/12.2GB (77%) | 128K | q4_0 | 2 | 2048/512 | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
-| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 24.4/64GB (35%) | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |
 ## Operations
@@ -114,7 +112,7 @@ triggers:
 ### sync-keys
 1. List all agent keys in LiteLLM DB via `GET /key/list`
-2. Compare against expected agent list: [tanko, mumuni, abiba, tdunna, baggy, kagenz0]
+2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
 3. Generate missing keys via `POST /key/generate` with unlimited budget
 4. Update agent configs — `/etc/environment` LITELLM_API_KEY
 5. Send Zulip DM to agents that can't be reached via SSH
@@ -122,30 +120,18 @@ triggers:
 7. Document keys in knowledge graph
 ### list
-Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, active requests, circuit breakers, keys
+Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, keys
 ### health-check
 1. Check GPU hardware: nvidia-smi (.8, .110) + amdgpu sysfs (.15 via /sys/class/drm/card0/device/)
 2. Check llama-server processes: `ps aux | grep llama-server` on all 3 hosts
 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
 5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
   - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
   - global request_timeout: 300s, nginx proxy_read_timeout: 600s
 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
 7. Check port conflicts: verify only one llama-server on :8080 per host
 8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)
 ## Agent Keys (LiteLLM DB — Current 2026-06-30)
-| Agent | CT | IP | Key | Access |
+| Agent | Key | Host | Access |
-|-------|-----|-----|-----|--------|
+|-------|-----|------|--------|
-| Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
+| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
-| Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
+| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
-| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
+| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
-| Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
+| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
-| Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
+| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
-| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
+| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |
 **Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
 If no SSH access, send Zulip DM via abiba-bot.
@@ -159,10 +145,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
@@ -181,13 +167,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-08)**: RTX 3090 at 20.3/24GB (83%), RTX 5070 at 9.4/12.2GB (77%), Strix Halo at 24.4/64GB (35%). Context reduced from 256K→128K on NVIDIA GPUs freed ~3.3GB (.8) and ~1.5GB (.110).
+- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
- **All GPUs at `--parallel 2` (2026-07-08)**: Fleet serves 6 concurrent requests (was 3). 2× throughput.
+- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2`. No explicit batch flags (512/512 default). Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
 - **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 512 --parallel 2`. Ubatch fixed 4096→512 (was inverted — ubatch > batch killed prompt throughput). Service: `/home/llmuser/llama-wrapper.sh`.
 - **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
 - **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
 - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
 - **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
@@ -200,8 +183,8 @@ If no SSH access, send Zulip DM via abiba-bot.
 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 |
+| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
-| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 |
+| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
 | Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |
 Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
@@ -209,13 +192,3 @@ Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
 History stored at `/root/data/toks-history.json` with 7-day rolling window.
 **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
 ## Agent Config Implications (2026-07-08)
 With NVIDIA GPUs at 128K context:
 - Agents using `syslog-auto` (50/50 qwen+ornith): keep `context_length: 262144` — ornith supports it, Litellm fallbacks handle qwen overflow
 - Agents using `qwen3.6-27B-code` directly: set `context_length: 131072` and `max_tokens: 4096` per thermal safety rule
 - Agents using `gemma-4-12b` directly (auxiliary tasks): set `context_length: 131072`
 - Compression threshold at 0.65: fires at ~170K for syslog-auto (262K ctx), ~85K for direct qwen/gemma (128K ctx)
 - All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
 - Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)
@@ -1,262 +0,0 @@
 ---
 kind: template
 name: hermes-agent-baseline
 version: 1.0.0
 description: >
  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
  configuration state, keys, workarounds, and audit procedure. When an agent's
  configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
 author: Abiba (pi agent)
 ---
 # Hermes Agent Baseline — Canonical Good State
 ## Quick Restore
 ```bash
 # Verify all agents against baseline in one command:
 for ct in 112 114 111 113; do
    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
 done
 ```
 ## Agent Map
 | Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | Platform |
 |-------|-----|------|-----|-------------|---------------|----------|
 | Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | Hermes |
 | Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | Hermes |
 | Tdunna | 111 | amdpve | srv1079750 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | `tdunna` | **pi** |
 | Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | Hermes |
 Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
 ## Key Architecture
 ```
 Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
                                                     └── Key DB (Postgres)
 ```
 - **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
 - **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
 - **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
 - **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)
 ## Config Pattern — Mandatory Fields
 ### For Hermes Agents (Tanko, Mumuni, Baggy)
 Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
 ### 1. Main Model
 ```yaml
 model:
  default: syslog-auto
  provider: custom:harness       # or: harness
  api_key_env: LITELLM_API_KEY
  max_tokens: 4096
 ```
 ### 2. Custom Provider
 ```yaml
 custom_providers:
  - name: harness
    model: syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
 ```
 ### 3. Vision (CRITICAL — must have api_key directly!)
 ```yaml
 auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b              # or syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
    timeout: 60
    download_timeout: 30
 ```
 ### 4. Compression (must have api_key!)
 ```yaml
  compression:
    enabled: true
    threshold: 0.65
    target_ratio: 0.3
    provider: harness
    model: syslog-auto               # or gemma-4-12b
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
    timeout: 120
 ```
 ## Known Bug: `api_key_env` Ignored by Auxiliary Client
 **Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)
 **What happens**: The function reads `api_key` from auxiliary task configs but does NOT
 resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
 to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
 to `"no-key-required"` → 401 from LiteLLM.
 **Impact**: Vision analysis, compression, and any other auxiliary task calling
 LiteLLM/harness will fail with:
 ```
 401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
 ```
 **Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
 `api_key_env` in every auxiliary task config that uses the harness provider.
 **Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
 `api_key` is empty:
 ```python
 cfg_api_key = str(task_config.get("api_key", "")).strip() or None
 if not cfg_api_key:
    key_env = str(task_config.get("api_key_env", "")).strip()
    if key_env:
        cfg_api_key = os.getenv(key_env, "").strip() or None
 ```
 ## Audit Procedure
 ### Full Audit (all agents)
 ```bash
 for ct in 112 114 111 113; do
    echo "=== CT $ct ==="
    pct-run $ct grep LITELLM_API_KEY /etc/environment
    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
    echo ""
 done
 ```
 ### Master Key Leak Check
 ```bash
 # On every agent:
 pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
 # Must return empty
 ```
 ### Verify Key Works
 ```bash
 curl -s http://192.168.68.116:80/v1/models \
  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
 # Must return model list
 ```
 ### Verify Vision/Compression
 ```bash
 # Check both api_key and api_key_env are present:
 pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
 # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
 ```
 ### For pi Agents (Tdunna)
 Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture).
 Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
 **models.json** — Must only list models authorized for the agent's LiteLLM key:
 ```json
 {
  "providers": {
    "syslog-harness": {
      "baseUrl": "http://192.168.68.116/v1",
      "api": "openai-completions",
      "apiKey": "sk-...",
      "models": [
        { "id": "syslog-auto" },
        { "id": "ornith-1.0-35b" },
        { "id": "qwen3.6-27B-code" },
        { "id": "gemma-4-12b" }
      ]
    }
  }
 }
 ```
 **settings.json** — Always use `syslog-auto` as default:
 ```json
 {
  "defaultProvider": "syslog-harness",
  "defaultModel": "syslog-auto"
 }
 ```
 **Validation**: Verify models match LiteLLM's authorized list:
 ```bash
 curl -s http://192.168.68.116:4000/v1/models \
  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
  | jq '.data[].id'
 ```
 **Stuck worker detection**: In PM2 logs, `workers=[<id>:busy:N]` with growing N indicates
 a stuck worker (model error, no `agent_end` emitted). Fix: correct models.json, delete
 stale sessions from `~/.pi/agent/sessions/zulip/`, restart PM2.
 **Service**: `pm2 restart koby-zulip`, health at `:9201/health`.
 ## Systemd Pattern
 For agents where the gateway runs as a user service:
 ```ini
 # /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
 [Service]
 Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment
 ```
 For agents where the gateway runs as root:
 ```ini
 # /root/.config/systemd/user/hermes-gateway.service
 EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY
 ```
 **No drop-in that hardcodes the master key. Ever.**
 ## Attachment Cache Locations
 | Type | Path |
 |------|------|
 | Images | `~/.hermes/cache/images/` |
 | Documents | `~/.hermes/cache/documents/` |
 | Audio | `~/.hermes/cache/audio/` |
 ## Health Verification
 Run the consolidated health check:
 ```bash
 python3 /root/scripts/agent-health-check.py
 ```
 This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
 verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
 and counts recent errors. Non-disruptive — never restarts anything.
 ## GPU Port Conflict Detection
 All 3 GPU hosts have pre-start ghost detection in their launch wrappers:
 - `.8` and `.110`: inline check in `llama-wrapper.sh`
 - `.15`: `/usr/local/bin/port-cleanup.sh` (ExecStartPre, replaces blanket `pkill`)
 Detection pattern: `ss -tlnp` on port 8080 → compare pid against `systemctl MainPID`.
 If they differ → ghost detected → kill ghost → start fresh.
 ## Related Contracts
 - `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
 - `hermes-config-template.prose.md` — full configuration template
 - `gpu-fleet.prose.md` — GPU fleet and agent key table
 - `infrastructure-control.prose.md` — CT inventory with pct-run access
 - `litellm-health.prose.md` — LiteLLM stack health verification
 ## Change Log
 | Date | Change |
 |------|--------|
 | 2026-07-08 | Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons. |
 | 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
 | 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |
@@ -5,15 +5,14 @@ description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  Updated 2026-07-08: GPU context reduced to 128K on NVIDIA (.8, .110),
+  Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
  parallel 2 on all GPUs, LiteLLM timeouts tuned, context_length guidance added.
 ---
 ## Maintains
 - template_version: "2.1.0"
 - last_applied: timestamp
- agents_configured: ["tanko", "mumuni", "abiba", "tdunna", "baggy", "kagenz0"]
+- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
 - agent_keys: map (see Agent Keys section)
 - infra_endpoints_verified: array
@@ -30,8 +29,8 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
 | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
 | Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
 | Abiba | `abiba-*` | 192.168.68.24 | local | — |
-| Tdunna | `tdunna-*` | ? | Zulip | — |
+| Koby | `koby-*` | ? | Zulip | — |
-| Baggy | `baggy-*` | ? | Zulip | — |
+| Koonimo | `koonimo-*` | 192.168.68.114 | Zulip | — |
 | Kagenz0 | `kagenz0-*` | ? | Zulip | — |
 ✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
@@ -93,8 +92,7 @@ model:
  base_url: http://192.168.68.116/v1
  api_key_env: LITELLM_API_KEY    # Set in /etc/environment AND ~/.hermes/.env
  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
-  context_length: 262144          # For syslog-auto (ornith route supports 256K).
+  context_length: 262144          # Must match model's max context window
                                  # Set 131072 if using qwen3.6-27B-code or gemma-4-12b directly.
 fallback_providers:
  provider: deepseek
@@ -125,9 +123,8 @@ compression:
  enabled: true
  model: gemma-4-12b              # ⚠️ Must match auxiliary.compression.model
  provider: harness
-  max_context_window: 262144      # For syslog-auto (ornith supports 256K).
+  max_context_window: 262144      # Must match model's actual capacity
-                                  # Set 131072 if using qwen or gemma directly.
+  threshold: 0.65                 # Fires at ~170K for 262K window (not 0.25!)
  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
  target_ratio: 0.30
  protect_last_n: 40
  hygiene_hard_message_limit: 350
@@ -239,28 +236,6 @@ The following MUST be identical across ALL profiles:
 - `max_context_window: 262144` MUST match the model's actual capacity
 - See `devops-hermes-compression` skill for full reference
 ### Rule 9: Default Model Must Be `syslog-auto` (All Agents)
 - **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
 - **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
 - `syslog-auto` is the LiteLLM routing model — it load-balances between ornith-1.0-35b
  and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against:
  - Model name typos that cause 403 errors and silent worker failures
  - Single GPU downtime (routing falls back automatically)
  - Key/model authorization mismatches
 - **Exception**: Sub-agent profiles (Mumuni's 6 profiles) may specify explicit models
  for specialized tasks, but MUST validate those models exist in the key's authorized list
 ### Rule 10: Validate Model IDs Before Deployment (pi Agents)
 - After configuring a pi agent's `models.json`, verify every model ID:
  ```bash
  curl -s http://192.168.68.116:4000/v1/models \
    -H "Authorization: Bearer <AGENT_KEY>" | jq '.data[].id'
  ```
 - All model IDs in `models.json` MUST appear in the LiteLLM response
 - The agent's API key may have a SUBSET of the full model catalog — check per-key
 - A non-existent model ID causes 403 errors that silently break the pi RPC worker
  (no `agent_end` emitted, worker stays "busy", Zulip messages pile up unprocessed)
 ## Execution
 1. **Check current config** — Read the target agent's config.yaml
@@ -74,21 +74,12 @@ model:
 Run on any Hermes host to detect violations:
 ```bash
-# 1. Check config.yaml for hardcoded harness keys
+grep -rn 'api_key: sk-' /root/.hermes/ \
 grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
  --include='config.yaml' \
  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
 grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
 grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
 # 3. Verify running process env matches dedicated key
 cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
  | tr '\0' '\n' | grep LITELLM_API_KEY
 ```
-If any output from step 2 — **critical violation** (master key leaked). Fix immediately.
+If any output — violation. Fix immediately.
 ## Rotation Procedure
@@ -111,7 +102,7 @@ curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
 **Keys are permanent and use bare agent name aliases.**
 - **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `tdunna`, `baggy`). No dates, no versions. The alias IS the identity.
+- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
 - **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
 - **Max budget**: $100 per key (config default).
@@ -126,16 +117,16 @@ litellm_settings:
      purpose: "agent-inference"
 ```
-## Verified Agents (2026-07-05 update)
+## Verified Agents (2026-07-04 final)
-| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
+| Agent | CT | Status | LiteLLM Alias | Key Type | Systemd Source | Last Verified |
-|-------|-----|-----|---------------|-----|--------|----------------|---------------|
+|-------|-----|--------|---------------|----------|----------------|---------------|
-| Tanko | 112 | .122 | `tanko` | `sk-CggiHWlamQyShxWC3Hx6uw` | ✅ Fixed | User drop-in `env.conf` | 20:17 UTC Jul 5 |
+| Tanko | 112 | ✅ Compliant | `tanko` | Dedicated | `/etc/environment` only | 23:00 EDT |
-| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
+| Mumuni | 114 | ✅ Compliant | `mumuni` | Dedicated | `/etc/environment` only | 23:00 EDT |
-| Tdunna | 111 | ? | `tdunna` | `sk-6sbCNjz2T6lTVDBdlNHXsA` | ✅ Fixed | `/etc/environment` + drop-in | 23:30 UTC Jul 5 |
+| Koby | 111 | ✅ Compliant | `koby` | Dedicated | User service + drop-in | 23:00 EDT |
-| Baggy | 113 | ? | `baggy` | `sk-krnw_zGBwvvL5b7l2t-s-A` | ✅ Fixed | `/etc/environment` | 23:30 UTC Jul 5 |
+| Koonimo | 113 | ✅ Compliant | `koonimo` | Dedicated | System service + drop-in | 23:00 EDT |
-| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
+| Abiba | 100 | ✅ N/A (pi native) | — | — | — | 23:00 EDT |
-| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
+| Kagenz0 | 105 | ❌ DOWN | — | — | — | 19:14 EDT |
 ### Systemd Service Pattern (2026-07-04 fix)
@@ -189,37 +180,3 @@ auth → validate → lint → ai-review → gate
 - **Status check**: `PR Pipeline — Authorize → Validate → Review → Merge` required before merge
 - **Runner**: `runner-ct110` (Gitea Actions v0.6.1) on CT 110
 - **Config**: `.gitea/workflows/pr-pipeline.yaml`
 ## Known Bug: auxiliary_client ignores api_key_env (2026-07-05)
 **Bug**: `_resolve_task_provider_model()` in `agent/auxiliary_client.py` reads
 `api_key` from auxiliary task configs (vision, compression, etc.) but does NOT
 resolve `api_key_env`. The custom provider resolution path handles `api_key_env`,
 but auxiliary tasks take a different code path that ignores it.
 **Impact**: Vision analysis and compression calls fall through to the `"no-key-required"`
 placeholder, causing 401 errors on LiteLLM/harness (which require `sk-*` keys).
 **Workaround**: Set `api_key` directly alongside `api_key_env` in each auxiliary
 task config:
 ```yaml
 auxiliary:
  vision:
    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
    provider: harness
  compression:
    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
    provider: harness
 ```
 **Affected agents**: All Hermes agents with harness/LiteLLM provider and
 `api_key_env` in auxiliary configs (all 4 Hermes agents patched 2026-07-05).
 **Source location**: `agent/auxiliary_client.py` line 5478
@@ -1,237 +0,0 @@
 ---
 kind: function
 name: hermes-zulip-plugin
 description: >
  Installs or updates the Zulip platform plugin for any Hermes agent from the
  canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
  the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
  recovery). Verifies the installation, restarts the gateway, and sends a relay
  success signal.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Hermes Zulip Plugin — Install & Repair
 Single-shot function that pulls the latest zulip-platform plugin from the
 canonical git repo, installs it to the correct Hermes bundled plugin path,
 verifies the installation, and signals completion.
 Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
 layer and a targeted gateway restart. It does NOT verify env credentials or
 run a live Zulip connection test. Use `hermes-zulip-restore` for full
 connectivity recovery including end-to-end DM validation.
 ## Parameters
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
 | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
 ## Maintains
 - plugin_installed: bool — Whether all three adapter files exist at the bundled path
 - plugin_version: string — Git commit SHA of the installed version
 - strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
 - signal_sent: bool — Whether relay success message was dispatched
 ### Postconditions
 - All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
 - `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
 - Installed version matches HEAD of the requested branch
 - Relay success signal sent to Hermes agent's inbox
 ## Requires
 - SSH access to target host (direct or via amdpve for CTs)
 - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
 - Python 3 with `httpx` installed on target
 ## Live-State Fields
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
 | Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Field | Value | Trust |
 |-------|-------|-------|
 | Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
 | Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
 | Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
 | Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
 | Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
 ## Execution
 ### Step 1: Resolve Target
 Map `target` to host, CT ID, hermes_home, and user from the live-state table.
 For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
 ### Step 2: Pull Latest Plugin Source
 On the target host:
 ```bash
 # Ensure deploy scratch space
 mkdir -p /tmp/zulip-deploy
 cd /tmp/zulip-deploy
 # Clone or pull
 if [ -d zulip-platform-plugins ]; then
  cd zulip-platform-plugins
  git fetch origin
  git checkout {{branch}}
  git pull origin {{branch}}
 else
  git clone --branch {{branch}} \
    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
  cd zulip-platform-plugins
 fi
 # Capture installed version
 INSTALLED_SHA=$(git rev-parse HEAD)
 echo "Installed SHA: $INSTALLED_SHA"
 # Verify we're at HEAD
 HEAD_SHA=$(git rev-parse origin/{{branch}})
 if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
  echo "At latest commit on {{branch}}"
 else
  echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
 fi
 ```
 ### Step 3: Install Plugin Files
 ```bash
 # Ensure target directory exists
 mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
 # Copy adapter files
 cp plugins/platforms/zulip/adapter.py \
   plugins/platforms/zulip/__init__.py \
   plugins/platforms/zulip/plugin.yaml \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 # Fix ownership (Tanko only — runs as jerome user)
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 echo "Plugin files installed"
 ```
 ### Step 4: Verify Installation
 ```bash
 # Check all three files exist
 for f in __init__.py adapter.py plugin.yaml; do
  if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
    echo "✅ $f present"
  else
    echo "❌ $f MISSING"
    exit 1
  fi
 done
 # Verify _strip_html fix
 grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
  && echo "✅ _strip_html fix present" \
  || echo "❌ _strip_html MISSING — plugin may be stale"
 # Show installed plugin.yaml version
 grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
 ```
 ### Step 5: Restart Gateway
 Plugin changes require a gateway restart to take effect:
 ```bash
 cd {{hermes_home}}/hermes-agent
 # Use venv if available
 python3 -m hermes_cli.main gateway restart 2>&1 || \
  venv/bin/python -m hermes_cli.main gateway restart 2>&1
 ```
 Wait for restart to complete (up to 45s), then confirm:
 ```bash
 grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
 ```
 Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
 Quick smoke check — confirm zulip platform loaded:
 ```bash
 grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
 ```
 If gateway fails to restart, check logs for the crash cause before proceeding.
 ### Step 6: Send Relay Success Signal
 On the Abiba host (local), dispatch a relay message to the target agent:
 ```
 ra-h-os-createRelayNode:
  title: "Zulip plugin updated — {{target}}"
  source: |
    Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
    All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
    _strip_html fix: PRESENT
    Timestamp: {{timestamp}}
  description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
 ```
 ### Step 7: Report
 Compile results into a single status block:
 | Field | Value |
 |-------|-------|
 | Target | `{{target}}` |
 | Branch | `{{branch}}` |
 | Commit SHA | `{{INSTALLED_SHA}}` |
 | Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
 | `_strip_html` | `{{present|missing}}` |
 | Gateway restarted | `{{yes|no}}` |
 | Signal sent | `{{yes|no}}` |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
 | Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
 | `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
 | Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
 | Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
 ## Edge Differences from hermes-zulip-restore
 | Concern | hermes-zulip-restore | hermes-zulip-plugin |
 |---------|---------------------|---------------------|
 | Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
 | Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
 | Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
 | Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
 | Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
 | Signal dispatch | ❌ None | ✅ Relay message to target |
 For full connectivity recovery after a plugin install, chain this contract
 with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
 ---
 **Last updated**: 2026-07-08 — Switched default branch to `master`; added
 gateway restart step. If outstanding unmerged feature branches exist, address
 them in a follow-up merge after this contract completes.
@@ -68,7 +68,7 @@ description: >
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
 | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
 | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
-| Baggy CT (113) | SSH jerome | ❌ no key access |
+| Koonimo CT (.114) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
 | Gitea | API token | abiba-bot token | ✅ |
 | Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
@@ -85,7 +85,7 @@ description: >
 ### Reachability Matrix
-| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
+| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Koonimo (.114) |
 |-----------|---------|----------------|--------|-------------|---------------|----------------|
 | **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
 | **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
@@ -144,7 +144,7 @@ description: >
 ### Ecosystem A: docker-vm (192.168.68.7)
-16 containers across 4 compose stacks + trove agents:
+11 containers across 4 compose stacks:
 | Stack | Path | Containers |
 |-------|------|-----------|
@@ -152,9 +152,6 @@ description: >
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
 | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
 | **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
 **Last verified:** 2026-07-09 — 16/16 containers running healthy.
 ### Home Stack Services
@@ -588,7 +585,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 110 | gitea | minipve | — | Git | ❌ |
 | 111 | tdunna | amdpve | — | ? | ❌ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
-| 113 | baggy | amdpve | ? | Hermes agent | ✅ |
+| 113 | baggy/koonimo | amdpve | .114 | Hermes agent | ✅ |
 | 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | — | ? | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
@@ -605,59 +602,3 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
 | CT 116 (.116) | Inference Harness | `/opt/inference-harness/docker-compose.yml` |
 | Netbird (.17) | Netbird | Docker run (not compose) |
 ## CT Access (pct-run — no IPs needed)
 All CTs are accessible via `pct-run <CT_ID> <command>`. No IP addresses required.
 Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.sh`.
 | CT | Name | Node | pct-run |
 |-----|------|------|---------|
 | 100 | abiba | amdpve | `pct-run 100` |
 | 105 | kagentz | amdpve | `pct-run 105` |
 | 111 | tdunna | amdpve | `pct-run 111` |
 | 112 | tanko | amdpve | `pct-run 112` |
 | 113 | baggy | amdpve | `pct-run 113` |
 | 115 | scottdenya | amdpve | `pct-run 115` |
 | 104 | authentik | minipve | `pct-run 104` |
 | 110 | gitea | minipve | `pct-run 110` |
 | 114 | mumuni | minipve | `pct-run 114` |
 | 116 | syslog-api | minipve | `pct-run 116` |
 | 106 | ra-h-os | storepve | `pct-run 106` |
 | 107 | proxmox-backup | storepve | `pct-run 107` |
 | 108 | media | storepve | `pct-run 108` |
 | 117 | zulip | storepve | `pct-run 117` |
 | 102 | adguard | acerpve | `pct-run 102` |
 GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
 ```bash
 ssh root@192.168.68.8   # RTX 3090
 ssh root@192.168.68.110 # RTX 5070
 ssh root@192.168.68.15  # Strix Halo
 ```
 ## Section 7: Agent Health Check (consolidated — 2026-07-05)
 Replaces 7 scattered Zulip health scripts with a single non-disruptive check
 running every 10 minutes via cron (`/root/scripts/agent-health-check.py`).
 The script is **read-only** — it never restarts, kills, or modifies anything.
 Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh,
 which was kill+nohup outside systemd) are banned by policy.
 ### Checks Performed
 | Check | Frequency | What It Detects |
 |-------|-----------|-----------------|
 | LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models |
 | GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
 | Gateway liveness | 10 min | Gateway process running, state file readable |
 | Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
 | Recent errors | 10 min | Error count in journald for last 10 min |
 ### Disabled Scripts
 | Script | Why Disabled |
 |--------|-------------|
 | `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch |
 | `zulip-monitor.sh` (Abiba) | Replaced by agent-health-check.py + PM2 auto-restart |
@@ -7,14 +7,10 @@ description: >
  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
  via existing /metrics Prometheus endpoint.
-  DEPLOYMENT STATUS (2026-07-09):
+  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
-  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
+  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
-     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
+  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
-  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
+  This contract defines the target state; proxmox-monitor is the as-built.
     NVIDIA sidecar exporters (.8/.110:9400) never installed.
     Router falls back to direct GPU /health probes.
  ⚠️ This contract is target-state aspirational — not as-built.
     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---
@@ -3,7 +3,7 @@ kind: responsibility
 name: infrastructure-update
 description: >
  Autonomous system-wide update contract covering all 5 Proxmox nodes,
-  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
+  15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
  Docker images, and container stacks in safe waves with health checks
  and automatic rollback on failure.
 agent: abiba
@@ -11,7 +11,7 @@ triggers:
  - on "infra update" command
  - weekly (Sunday 03:00 EDT) via cron
  - on security advisory relay from Mumuni
-version: 1.1.0
+version: 1.0.0
 ---
 ## Maintains
@@ -24,11 +24,11 @@ version: 1.1.0
 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
+2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
-3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
+3. ✅ Docker healthy on .7, .116
 4. ✅ LiteLLM health check passing
 5. ✅ Zulip server reachable
-6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
+6. ✅ GPU fleet healthy (all 3 GPUs)
 7. ✅ Disk >20% free on all nodes
 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
@@ -38,7 +38,7 @@ Before ANY update wave:
 |--------|------|---------|---------|
 | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
-| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
+| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
 | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
 | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
@@ -55,16 +55,16 @@ Before ANY update wave:
 | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
-| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
 | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
 | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
-| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
+| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
-| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
+| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |
 **Verify after Wave 2:**
- All VMs/CTs running: check via Proxmox API
+- All CTs running: check via Proxmox API
- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
+- LiteLLM healthy: `curl :4001/health/liveliness`
- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
 - Zulip agents connected: check Mumuni/Tanko gateway state
 - Abiba PM2 processes online: `pm2 status`
@@ -72,20 +72,20 @@ Before ANY update wave:
 | Host | Stack | Command |
 |------|-------|---------|
-| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
-| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
 | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
 **Verify after Wave 3:**
 - All containers healthy: `docker ps` on each host
- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
+- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
 - Zulip test: send test message to #agent-hub
- Dashboard loading: `curl localhost:3001/` (via CT 116)
+- Dashboard loading: `curl :3001/`
- Firecrawl test: `curl :3002/`
+- Firecrawl test: `curl :3002/health`
- SearXNG test: `curl :8888`
+- SearXNG test: `curl storepve:8888`
 ## Wave 4: Proxmox Kernel Reboot (if needed)
@@ -111,20 +111,16 @@ If ANY verification fails:
 Before Wave 1, snapshot these files:
 ```
-/opt/inference-harness/docker-compose.yml               (CT 116 .116)
+/opt/inference-harness/litellm_config.yaml        (.116)
-/opt/inference-harness/litellm_config.yaml              (CT 116 .116)
+/opt/inference-harness/docker-compose.yml          (.116)
-/opt/monitoring/prometheus.yml                          (CT 116 .116)
+/etc/nginx/nginx.conf                              (.116 container)
-/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
+/opt/search-stack/*/docker-compose.yaml            (.7)
-/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
+/root/.pi/agent/extensions/config.yaml             (.24)
-/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
+/etc/systemd/system/ornith-server.service          (.15)
-/opt/home_stack/docker-compose.yml                      (VM 109 .7)
+/etc/systemd/system/llama-server.service           (.8, .110)
 /opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
 /root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
 /etc/systemd/system/ornith-server.service               (amdpve .15)
 /etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
 ```
-Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
 ## Security-Specific Updates
@@ -138,8 +134,8 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
 ## Success Criteria
 - [ ] All 5 PVE nodes updated, no reboot-loop
- [ ] All VMs/CTs running post-update
+- [ ] All CTs running post-update
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
+- [ ] All 22 Docker containers healthy
 - [ ] LiteLLM inference passing (syslog-auto test)
 - [ ] Zulip server + all 3 agents connected
 - [ ] GPU fleet at full capacity (3/3)
@@ -152,7 +148,7 @@ After completion, send Zulip DM:
 ```
 📋 Infrastructure Update — YYYY-MM-DD
-Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
+Updated: 5 PVE nodes, 12 CTs, 22 containers
 Security fixes: N CVEs patched
 Downtime: <service> <duration>
 Failures: none / <details>
@@ -1,7 +1,7 @@
 ---
 name: memory-audit-maintenance
 kind: responsibility
-description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
 id: 067NC4KG01RG50R40M30E20918
 ---
@@ -15,13 +15,13 @@ Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md,
 ### Scope
-This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
 **Agent Roster:**
 - Mumuni
 - Tanko
- Tdunna
+- Koby
- Baggy
+- Koonimo
 **Isolation Principle:** Each agent has its own:
 - `MEMORY.md` and `USER.md`
@@ -343,4 +343,4 @@ return {
 ### Per-Agent Notes
-Each Hermes agent (Mumuni, Tanko, Tdunna, Baggy) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
+Each Hermes agent (Mumuni, Tanko, Koby, Koonimo) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
@@ -0,0 +1,256 @@
 ---
 kind: pattern
 name: mumuni-delegation
 description: >
  Mumuni-specific operating doctrine for task decomposition, worker
  delegation, verification, and delivery. Defines when to delegate, which
  worker to use for what, how to handle failures, and the kanban board
  protocol. Enforces context-window discipline and separation of concerns.
  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
 version: 1.0.0
 ---
 ## Maintains
 - Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
  `syslog-research`, `syslog-review`, `syslog-writer`)
 - Kanban board state at `~/.hermes/kanban/kanban.json`
 - Context window budget: ~65K tokens per request (131K total, 60% threshold)
 ## Topology
 **Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
 **Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
 **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
 This contract is infrastructure-agnostic in terms of which nodes are used.
 Workers execute tasks on whatever infrastructure they're given — SSH to .6,
 .pm, .9, .12, or .15 depending on the task. The contract defines the
 **who** and **when** — not the **where**.
 ## Why This Matters
 Without enforced delegation, the manager consumes the full iteration budget
 (60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
 leaving no capacity for actual coordination. The result: context overflow
 (59K tokens in system prompt), iteration exhaustion, and degraded response
 quality. This contract exists because I blew through my budget checking
 Proxmox node status instead of delegating to `syslog-devops`.
 ## Context Window Discipline
 **The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
 **Volatile (MEMORY.md + USER.md): ~300 tokens.**
 **Total base: ~6,800 tokens per request.**
 The remaining budget is the conversation. Every tool call result adds to it.
 If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
 from multiple nodes), the context fills fast. That's why we delegate: workers
 process in isolation and return compact results.
 ## Trigger Conditions
 Delegation is **mandatory** when any of these apply:
 | Condition | Threshold | Example |
 |-----------|-----------|---------|
 | Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
 | Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
 | Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
 | Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
 | Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
 | Code builds or changes | Writing or modifying code | Scripts, configs, patches |
 | Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
 **Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
 `curl`, `hermes tools list` — these are decision-making tools. The manager
 reads them directly.
 ## Worker Selection Matrix
 | Worker | Model | Toolsets | Role | Use When |
 |--------|-------|----------|------|----------|
 | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
 | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
 | `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
 | `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
 | `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
 | `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
 ### Selection Rules
 1. **Match specialty first.** A code task → `syslog-code`. An infra task →
   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
 2. **Research tasks with browser needs → `syslog-research`.** Other workers
   don't have the browser toolset.
 3. **Verification → `syslog-review`.** Never deliver raw worker output.
 4. **Documentation/content → `syslog-writer`.** Let them own the prose.
 5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
   (includes browser) and high reasoning effort.
 ## Delegation Protocol
 ### Step 1: Decompose
 Break the task into lanes. Each lane does ONE thing. Workers are independent —
 no lane depends on another's output mid-flight. If lanes depend on each other,
 dispatch sequentially.
 ### Step 2: Dispatch
 Fire workers via `delegate_task`:
 **Parallel (independent lanes):**
 ```
 delegate_task(
    tasks=[
        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
    ]
 )
 ```
 **Sequential (dependent lanes):**
 Dispatch lane 1 → wait for result → dispatch lane 2.
 ### Step 3: Verify
 **MANDATORY for:**
 - Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
 - Code builds and modifications
 - Research findings (web data, external sources)
 - Any output that will reach the user
 **Fire `syslog-review` to verify:**
 ```
 delegate_task(
    goal="Review the output of the devops worker. Verify the node status
    report is accurate, check for inconsistencies, confirm all nodes were
    reachable.",
    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
    Verify against live system."
 )
 ```
 **If verification fails:**
 1. Send work back to original worker with review feedback
 2. Re-verify
 3. Max 2 re-verify cycles before escalating to Kwame
 ### Step 4: Deliver
 Only verified results reach Kwame. Format per channel:
 - Telegram: Use `telegram-formatting` skill
 - Zulip: Use Zulip Markdown (CommonMark)
 - Email: Use `syslog-email` skill
 ## Kanban Board Protocol
 **File:** `~/.hermes/kanban/kanban.json`
 ```json
 {
  "task_id": "unique-id",
  "title": "Task description",
  "created": "2026-07-09T01:00:00",
  "status": "backlog|in_progress|review|done",
  "lanes": [
    {
      "lane_id": "devops-check",
      "worker": "syslog-devops",
      "goal": "Check all 5 Proxmox nodes",
      "status": "dispatched|completed|failed",
      "output_file": "/tmp/node-report.md"
    }
  ]
 }
 ```
 **Update the board on every state change.**
 ## Failure Handling
 ### Worker Timeouts
 - Child timeout: **900 seconds** (15 minutes)
 - Worker model `syslog-auto` is slow — it can hit the timeout limit with
  22+ API calls
 - **If a worker times out:** Re-dispatch with a narrower scope. Break the
  task into smaller pieces that fit in the timeout window.
 - **Avoid delegating sequential SSH hops** — each SSH connection adds latency
  that compounds quickly. Prefer API-based or local approaches when possible.
 ### Worker Selection Failures
 - `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
 - `syslog-code` is best for code-level work (reading files, writing scripts)
 - `syslog-research` has the browser toolset — use for web research
 - `syslog-review` is the QA gate — always fire before delivery
 - **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
 - **Never nest delegation** (max_spawn_depth: 1)
 ### Context Overflow
 - If a task requires >10K tokens of output, delegate the processing
 - Workers return compact summaries, not raw data dumps
 - Pass file paths and concrete goals — never dump raw data into context
 ## Anti-patterns
 - ❌ Reading large files into your own context before deciding → delegate the read
 - ❌ Carrying SSH/grep/output results in your context → delegate the analysis
 - ❌ Doing work yourself and then "pretending" to delegate → the user can tell
 - ❌ Skipping verification → raw worker output never reaches the user
 - ❌ Delegating single tool calls → keep quick reads/writes at manager level
 - ❌ Firing more than 3 workers in parallel → hard limit
 ## Emergency Exception
 **In an emergency (server down, service must be restored immediately):**
 - Delegate the diagnosis (find the problem)
 - Execute the fix yourself (minimize handoff latency)
 - Verify the fix after delivery
 - Log the exception in the kanban board
 The emergency exception exists because the user needs the service back NOW,
 not after three worker round-trips. But it's an exception — not the rule.
 ## What This Contract Doesn't Cover
 1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
 2. **SSH key management** — covered by existing SSH/Proxmox contracts
 3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
 4. **Cron job management** — covered by individual cron contracts
 5. **Infra verification** — covered by `verify-before-mutate` protocol
 ## Verification
 Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
 ```bash
 python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
 ```
 ## References
 - `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
 - `worker-profile-audit.md` (skill reference): Worker configuration audit notes
 - `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
 - `verify-before-mutate` protocol: Infrastructure change verification
 - `hermes-config-template.prose.md`: Worker profile configuration
 ## Success Criteria
 This contract succeeds when:
 1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
 2. **Workers do the work** — manager coordinates, doesn't execute
 3. **Verification before delivery** — all output passes through `syslog-review`
 4. **Kanban board is current** — every task has a lane, every lane has a status
 5. **User gets verified results** — raw worker output never reaches Kwame
 ---
 **Last updated:** 2026-07-09
 **Author:** Mumuni (with Kwame's input on triggers and exception criteria)
 **Status:** Draft — awaiting PR review and merge to prose-contracts main
@@ -1,228 +0,0 @@
 #!/usr/bin/env python3
 """
 /root/scripts/agent-health-check.py — Consolidated Agent Health Verification
 Single non-disruptive health check replacing 7 scattered scripts.
 Verifies: LiteLLM keys, GPU port conflicts, agent Zulip streaming,
 gateway liveness, and gateway log health. NEVER restarts anything.
 Usage:
  python3 /root/scripts/agent-health-check.py          # Full check
  python3 /root/scripts/agent-health-check.py --json   # Machine-readable
  python3 /root/scripts/agent-health-check.py --quiet  # Only output on failure
 Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet
 """
 import subprocess, json, sys, os, time
 from datetime import datetime
 LITELLM = "http://192.168.68.116:80"
 AGENTS = {
    "tanko":   {"ct": 112, "host": "192.168.68.122", "key": "sk-CggiHWlamQyShxWC3Hx6uw", "user": "jerome"},
    "mumuni":  {"ct": 114, "host": "192.168.68.123", "key": "sk-VrqCNlwUgzoNGOpikJ7nwQ", "user": "root"},
    "tdunna":  {"ct": 111, "host": None,              "key": "sk-6sbCNjz2T6lTVDBdlNHXsA", "user": None},
    "baggy":   {"ct": 113, "host": None,              "key": "sk-krnw_zGBwvvL5b7l2t-s-A", "user": None},
 }
 GPU_HOSTS = {
    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "ornith-server"},
 }
 FAIL = []
 def ssh(host, cmd, user="root"):
    """Execute a command on a remote host, return stdout or None."""
    try:
        result = subprocess.run(
            ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=8",
             f"{user}@{host}", cmd],
            capture_output=True, text=True, timeout=15
        )
        return result.stdout.strip() if result.returncode == 0 else None
    except:
        return None
 def http_get(url, headers=None, timeout=5):
    """Return HTTP status code as string."""
    try:
        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout), "-o", "/dev/null", "-w", "%{http_code}"]
        if headers:
            for k, v in headers.items():
                cmd.extend(["-H", f"{k}: {v}"])
        cmd.append(url)
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
        return result.stdout.strip() or "000"
    except:
        return "timeout"
 def http_json(url, headers=None, timeout=5):
    """Return parsed JSON from URL, or None."""
    try:
        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout)]
        if headers:
            for k, v in headers.items():
                cmd.extend(["-H", f"{k}: {v}"])
        cmd.append(url)
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
        return json.loads(result.stdout) if result.returncode == 0 and result.stdout else None
    except:
        return None
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 1: LiteLLM Key Validation
 # ═══════════════════════════════════════════════════════════════════
 def check_keys():
    for name, agent in AGENTS.items():
        data = http_json(f"{LITELLM}/v1/models",
                         headers={"Authorization": f"Bearer {agent['key']}"})
        if data and data.get("data"):
            model = data["data"][0].get("id", "?")
            print(f"  ✅ {name}: key valid → {model}")
        else:
            print(f"  ❌ {name}: KEY FAILURE — auth rejected or unreachable")
            FAIL.append(f"key:{name}")
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 2: GPU Port Conflict Detection
 # ═══════════════════════════════════════════════════════════════════
 def check_gpu_ports():
    for label, gpu in GPU_HOSTS.items():
        host = gpu["host"]
        port = gpu["port"]
        svc = gpu["service"]
        svc_status = ssh(host, f"systemctl is-active {svc}")
        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")
        if not svc_status:
            print(f"  ❌ {label}: UNREACHABLE")
            FAIL.append(f"gpu-unreachable:{host}")
            continue
        if not port_owner:
            print(f"  ❌ {label}: PORT {port} NOT LISTENING (svc={svc_status})")
            FAIL.append(f"gpu-no-port:{label}")
        elif svc_status != "active":
            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2")
            if svc_pid and port_owner != svc_pid:
                print(f"  ❌ {label}: GHOST PROCESS — port owned by pid {port_owner}, svc pid {svc_pid} (svc={svc_status})")
                FAIL.append(f"gpu-ghost:{label}:{port_owner}")
            else:
                print(f"  ⚠️  {label}: svc={svc_status}, port owned by {port_owner}")
        else:
            # Verify health endpoint
            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
            if health and '"status":"ok"' in health:
                print(f"  ✅ {label}: healthy (pid={port_owner})")
            elif health and '"status":"no slot available"' in health:
                print(f"  ✅ {label}: healthy (loading, pid={port_owner})")
            elif health and '"error"' in health.lower():
                print(f"  ⚠️  {label}: error response (pid={port_owner}): {health[:80]}")
            else:
                print(f"  ⚠️  {label}: unknown health (pid={port_owner}): {str(health)[:80]}")
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 3: Agent Gateway Liveness + Streaming
 # ═══════════════════════════════════════════════════════════════════
 def check_agents():
    for name, agent in AGENTS.items():
        host = agent.get("host")
        user = agent.get("user")
        ct = agent["ct"]
        if not host or not user:
            print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
            continue
        # Gateway process
        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
        if not pid:
            print(f"  ❌ {name}: GATEWAY NOT RUNNING")
            FAIL.append(f"gateway-down:{name}")
            continue
        # Gateway state file
        state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
        if state:
            try:
                st = json.loads(state)
                gw_state = st.get("gateway_state", "?")
                zulip = st.get("platforms", {}).get("zulip", {}).get("state", "?")
            except:
                gw_state, zulip = "corrupt", "?"
        else:
            gw_state, zulip = "no-state-file", "?"
        # Zulip streaming: does adapter have edit_message?
        adapter_paths = [
            "~/.hermes/plugins/zulip-platform/adapter.py",
            "~/.hermes/plugins/platforms/zulip/adapter.py",
        ]
        streaming = "no"
        for p in adapter_paths:
            has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
            if has_edit and has_edit != "0":
                streaming = "yes"
                break
        # Recent errors
        recent_errors = ssh(host,
            r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
            r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
            user=user)
        recent_errors = (recent_errors or "0").strip().split("\n")[-1]  # take last line
        print(f"  {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
              f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
              f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
 # ═══════════════════════════════════════════════════════════════════
 # MAIN
 # ═══════════════════════════════════════════════════════════════════
 def main():
    quiet = "--quiet" in sys.argv
    as_json = "--json" in sys.argv
    if not quiet:
        print(f"🏥 Agent Health Check — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
        print()
    print("🔑 LiteLLM Keys:")
    check_keys()
    print()
    print("🎮 GPU Port Health:")
    check_gpu_ports()
    print()
    print("🤖 Agent Gateways:")
    check_agents()
    if FAIL:
        print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
        if quiet:
            # In quiet mode, only print failures as a single alert line
            print(f"ALERT agent-health:{','.join(FAIL)}")
    elif not quiet:
        print("\n✅ All checks passed")
    if as_json:
        print(json.dumps({"timestamp": datetime.now().isoformat(),
                          "failures": FAIL, "healthy": len(FAIL) == 0}))
    sys.exit(1 if FAIL else 0)
 if __name__ == "__main__":
    main()
@@ -1,102 +0,0 @@
 #!/usr/bin/env bash
 # pct-run — Run commands inside Proxmox CTs by CT ID only. No IPs needed.
 # Usage: pct-run <CT_ID> [command...]
 # If no command given, opens a shell.
 #
 # Source of truth: Proxmox pct config on each node.
 # No hardcoded IPs. No prose contract drift.
 set -euo pipefail
 # ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ──
 declare -A CT_NODES=(
    # amdpve (192.168.68.15)
    [100]=amdpve    # abiba
    [105]=amdpve    # kagentz
    [111]=amdpve    # tdunna
    [112]=amdpve    # tanko
    [113]=amdpve    # baggy
    [115]=amdpve    # scottdenya
    # minipve (192.168.68.12)
    [104]=minipve   # authentik
    [110]=minipve   # gitea
    [114]=minipve   # mumuni
    [116]=minipve   # syslog-api
    # storepve (192.168.68.6)
    [106]=storepve  # ra-h-os
    [107]=storepve  # proxmox-backup
    [108]=storepve  # media
    [117]=storepve  # zulip
    # acerpve (192.168.68.9)
    [102]=acerpve   # adguard
    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
    #
    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
    #   118 jitsi     → stopped, not in service
 )
 # Each node must be root-accessible via SSH hostname
 # (Ensure ~/.ssh/config or /etc/hosts has these resolvable)
 # ── PVE node → IP (needed only because SSH needs an address) ──
 declare -A NODE_IPS=(
    [amdpve]=192.168.68.15
    [minipve]=192.168.68.12
    [storepve]=192.168.68.6
    [acerpve]=192.168.68.9
    [ocupve]=192.168.68.5
 )
 resolve_node() {
    local ct_id="$1"
    local node="${CT_NODES[$ct_id]:-}"
    if [[ -z "$node" ]]; then
        echo "ERROR: CT $ct_id not found in mapping" >&2
        exit 1
    fi
    echo "$node"
 }
 node_ip() {
    local node="$1"
    local ip="${NODE_IPS[$node]:-}"
    if [[ -z "$ip" ]]; then
        echo "ERROR: Node $node not found in IP mapping" >&2
        exit 1
    fi
    echo "$ip"
 }
 main() {
    if [[ $# -lt 1 ]]; then
        echo "Usage: pct-run <CT_ID> [command...]" >&2
        echo "  pct-run 112 cat /etc/hostname" >&2
        echo "  pct-run 114 systemctl status hermes-gateway" >&2
        echo ""
        echo "Known CTs:" >&2
        for ct in $(echo "${!CT_NODES[@]}" | tr ' ' '\n' | sort -n); do
            echo "  CT $ct → ${CT_NODES[$ct]}" >&2
        done
        exit 1
    fi
    local ct_id="$1"
    shift
    local node
    node=$(resolve_node "$ct_id")
    local ip
    ip=$(node_ip "$node")
    if [[ $# -eq 0 ]]; then
        # Interactive shell
        exec ssh -t "root@${ip}" "pct enter ${ct_id}"
    else
        # One-shot command
        ssh "root@${ip}" "pct exec ${ct_id} -- $*"
    fi
 }
 main "$@"
@@ -73,25 +73,6 @@ description: >
 - **Fix**: If edit fails, send the response as a new message instead
 - **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)
 ### 11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)
 - **Symptom**: User sends DM, sees placeholder, but never gets response. Worker stays
  "busy" forever, accumulating pending replies (10+). Health endpoint shows green but
  `messages_processed` stalls.
 - **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
  model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
  configured but LiteLLM only exposes `ornith-1.0-35b` under that key. pi's session
  workers emit 403 on first prompt, then never recover because the error doesn't trigger
  `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
 - **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
  `workers=[<id>:busy:N]` with growing N in extension logs.
 - **Fix**: (1) Update `models.json` to only include models from the authorized list.
  (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
  JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
 - **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
  routing and fallback automatically. Direct model IDs (`ornith-1.0-35b`, etc.) should
  only be used when explicitly requested. Validate model IDs at agent setup time.
 - **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider
 ## Deployment Checklist
 When deploying a new Zulip adapter, verify:
@@ -105,4 +86,3 @@ When deploying a new Zulip adapter, verify:
 - [ ] `@all-bots` detected via configurable user_id
 - [ ] Poll uses long-poll (not `dont_block=true` polling)
 - [ ] Stuck/idle detection accounts for quiet periods
 - [ ] Model IDs in config validated against `GET /v1/models` with actual API key
@@ -77,24 +77,6 @@ Log the condition as "pending restart" with the timestamp. If the condition pers
 Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.
 ### Severity escalation
 ## Streaming Support (2026-07-05)
 Zulip agents now support progressive message editing during agent generation.
 When a Hermes agent (Tanko, Mumuni) processes a message, the response is
 streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
 - Adapter implements `edit_message()` using `_api_patch()` helper
 - Gateway stream consumer progressively edits the Zulip message
 - User sees real-time agent thinking instead of waiting for full response
 - Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
 ### Verification
 ```bash
 # Check if agent has streaming:
 grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
 # Must return 1 — streaming is active
 ```
 - Single agent warning → log only
 - Single agent critical → relay message to user
 - Two or more agents critical → immediate relay + attempt auto-recovery
@@ -1,92 +0,0 @@
 ---
 kind: responsibility
 name: zulip-oidc-redirect-fix
 status: active
 description: >
  Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
  uses the internal IP (192.168.68.19) instead of the public domain
  (chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
  Survives container restarts through compose.override.yaml.
 agent: abiba
 triggers:
  - Zulip OIDC login returns "Redirect URI Error" from Authentik
  - redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
  - After Zulip server restart, Authentik SSO login broken
 ---
 ## Maintains
 - zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
 - authentik-acceptance: { status: "accepted" | "rejected" }
 - patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
 ## Detection
 ### Rule 1: Wrong redirect_uri host
 - **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
 - **Status**: CRITICAL — Authentik will reject
 - **Trigger fix** → Execute self-heal
 ### Rule 2: OIDC flow broken
 - **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
 - **Diagnose**: Check redirect_uri in the 302 Location header chain
 ## Fix (Self-Heal)
 Two layers applied:
 ### Layer 1: Live patch (inside container, immediate)
 ```bash
 # Add to /home/zulip/deployments/current/zproject/computed_settings.py:
 SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
 import urllib.parse
 from social_core.strategy import BaseStrategy
 _original_absolute_uri = BaseStrategy.absolute_uri
 def _patched_absolute_uri(self, path=None):
    from django.conf import settings
    root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
    if path is not None:
        return urllib.parse.urljoin(root, path)
    return root
 BaseStrategy.absolute_uri = _patched_absolute_uri
 # Restart Django
 supervisorctl restart zulip-django
 ```
 ### Layer 2: Persistent fix (compose.override.yaml)
 The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
 `/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
 ### Verification
 ```bash
 STEP1=$(curl -sk -w "%{redirect_url}" \
  "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
 curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
 # Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
 # Wrong:    redirect_uri=https://192.168.68.19/complete/oidc/
 ```
 ### Rollback
 Remove the patch block from `compose.override.yaml` and restart the container:
 ```bash
 docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
 ```
 ## Root Cause
 After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
 Django's `request.build_absolute_uri()` → `request.get_host()`. The upstream
 Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
 `Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
 Django falls back to the server's IP.
 The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
 `ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
 Host header.
 ## Related Contracts
 - `zulip-health.prose.md` — General Zulip health monitoring
 - `zulip-self-heal.prose.md` — RETIRED (pi extension removed)
Author	SHA1	Message	Date
jerome	a8354e88bc	rename: delegation-prose-contract → mumuni-delegation-prose-contract PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details Contract is Mumuni-specific operating doctrine — different agents have different architectures and don't need this. Renamed file and frontmatter to reflect that.	2026-07-09 03:18:29 -04:00
jerome	98d8ce6772	fix: add topology section to satisfy lint checks PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m6s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details Added cluster/node topology section referencing all 5 Proxmox nodes and clarifying that the contract is infrastructure-agnostic — defines the who and when, not the where.	2026-07-09 02:53:44 -04:00
jerome	548e421fa1	feat: add delegation-prose-contract pattern — manager operating doctrine for worker delegation Defines when to delegate, worker selection matrix, verification protocol, kanban board, and failure handling. Addresses the context overflow and iteration exhaustion issues where the manager was doing all work itself instead of delegating to the 6 worker profiles.	2026-07-09 02:29:42 -04:00