prose: review, fix, and consolidate infrastructure contracts

- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.

disk-gc-threat-response.prose.md

@@ -0,0 +1,255 @@
+---
+kind: responsibility
+name: disk-gc-threat-response
+description: >
+  Recurring disk health scan, garbage collection, and threat response
+  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
+  Docker image bloat — 5 dangling images, 15 build cache layers.
+  Recovered 35.67GB via `docker system prune -a --force`.
+id: 067NV8KJ03ZG71S44N41F31022
+version: 1.0.0
+---
+
+# Disk GC & Threat Response
+
+## Goal
+
+Every container in the fleet stays below 80% disk usage through regular inspection
+and automated garbage collection. Critical breaches are detected, remediated, and
+logged within 5 minutes of discovery.
+
+## Scope
+
+All CTs in the Proxmox cluster, with special attention to Docker hosts:
+
+| Host | CT | Disk Risk | GC Strategy |
+|------|----|-----------|-------------|
+| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
+| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
+| All others | — | LOW — no Docker | apt clean, log rotate |
+
+## Threat Levels
+
+| Level | Threshold | Response | Escalation |
+|-------|-----------|----------|------------|
+| **GREEN** | < 75% | Log only | None |
+| **AMBER** | 75-84% | Warn via Zulip DM | GC scheduled for next run |
+| **RED** | 85-94% | Immediate GC attempt | Zulip DM + channel alert |
+| **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame |
+| **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame |
+
+## Requires
+
+- SSH access to all Docker hosts (matrix from infrastructure-control pattern)
+- Proxmox API access from abiba (token already provisioned)
+- Zulip bot credentials for alerting
+
+## Maintains
+
+Per-CT disk state, GC history, and threat resolution ledger. Each entry carries
+the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed
+bytes. Postcondition: no CT runs above 85% for more than one scan cycle without
+documented reason.
+
+### disk-health
+Current disk state for every CT: `pct_used`, `pct_avail`, `rootfs_size`, last GC
+timestamp, and active threat level.
+
+### gc-history
+Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed,
+and whether threat was resolved.
+
+### threat-log
+Active and resolved threat entries with severity, timestamps, remediation applied,
+and escalation trail.
+
+## Continuity
+
+- self-driven: full fleet scan every 6 hours
+- May also be invoked manually: `prose run disk-gc-threat-response`
+- Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates
+
+## Shape
+
+- `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
+- `delegates`:
+  - `disk-scanner`: per-CT disk check (pct exec df or SSH)
+  - `gc-docker`: docker system prune execution
+  - `gc-system`: apt clean, log rotate, tmp cleanup
+  - `alerter`: Zulip notification dispatch
+- `prohibited`: deleting user data, removing running containers, force-killing
+  production services
+
+## Runtime
+
+- `timeout`: 300 seconds per CT (GC may take time on large docker hosts)
+- `retry`: 2 attempts for SSH failures before marking a CT unreachable
+
+## Execution
+
+```prose
+let fleet = call disk-scanner
+  scope: all
+
+let threats = []
+for ct in fleet:
+  if ct.usage_pct >= 95:
+    push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct }
+  else if ct.usage_pct >= 85:
+    push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
+  else if ct.usage_pct >= 75:
+    push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }
+
+-- sort by severity descending
+sort threats by pct desc
+
+for threat in threats:
+  let result = call gc-executor
+    ct: threat.ct
+    level: threat.level
+    strategy: lookup-gc-strategy(threat.ct)
+  
+  call alerter
+    threat: threat
+    result: result
+
+call summary-reporter
+  fleet: fleet
+  threats: threats
+```
+
+## GC Strategies by Host Type
+
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+
+```bash
+# Phase 1: Safe prune (won't touch running containers' images)
+docker system prune -f
+
+# Phase 2: Aggressive (if still > 85% after Phase 1)
+docker system prune -a --force
+
+# Phase 3: Emergency (if still > 95%)
+docker system prune -a --force --volumes
+docker builder prune --all --force
+
+# Verification after each phase
+df -h /
+docker system df
+```
+
+### Non-Docker CTs
+
+```bash
+# Package cache
+apt-get clean
+apt-get autoremove --yes
+
+# Log rotation
+journalctl --vacuum-size=100M
+find /var/log -type f -name "*.log" -mtime +30 -delete
+
+# Temp files
+find /tmp -type f -mtime +7 -delete
+find /var/tmp -type f -mtime +30 -delete
+
+# Snap (if installed)
+snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do
+  snap remove "$snap" --revision="$rev"
+done
+```
+
+### Special Cases
+
+| CT | Special GC |
+|----|-----------|
+| 116 (syslog-api) | Prometheus retention: check `--storage.tsdb.retention.time` |
+| 100 (abiba) | Go module cache: `go clean -cache -modcache` if > 500MB |
+| 106 (ra-h-os) | Check relay DB size, enforce TTL |
+| 109 (docker-vm) | Check `/media/storage` and `/media/mediastore` mounts first |
+
+## Alert Templates
+
+### AMBER (75-84%)
+```
+⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+Next scheduled GC will attempt cleanup. No immediate action needed.
+```
+
+### RED (85-94%)
+```
+🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%.
+Status: {resolved|still elevated — {reason}}
+```
+
+### CRITICAL (≥95%)
+```
+🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%.
+{service_status} — {manual_action if needed}
+@Kwame — container nearly full, resolved: {yes_no}
+```
+
+## Incident Log: 2026-07-04 — kagentz docker bloat
+
+### Discovery
+Kwame noticed kagentz was full, asked Abiba to investigate.
+
+### Diagnosis
+- CT 105 (kagentz, amdpve): 49G used / 59G total (87%)
+- `/var/lib` = 55G (93% of disk)
+- Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable
+- 1 stopped container, 1 unused network, 15 build cache layers
+
+### Resolution
+```
+docker system prune -a --force
+→ Reclaimed 35.67GB
+→ Post: 16G / 59G (27%), 41G free
+→ 5 dangling images removed (kagentz-bridge + old builds)
+→ 1 stopped container removed
+→ 1 unused network removed (kagentz-bridge_default)
+→ 15 build cache layers removed
+```
+
+### Root Cause
+Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates
+dangling images and orphaned build cache. No automated GC was in place.
+
+### Preventive Measures
+- This contract now runs disk GC fleet-wide every 6 hours
+- Docker hosts get `docker system prune` on amber, `-a --force` on red
+- kagentz is flagged as HIGH risk due to development activity
+```
+
+Done. Here's what we have:
+
+## Storage Summary
+
+| System | Used | Free | Health |
+|--------|------|------|--------|
+| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
+| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
+| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
+| **/root/go** (abiba) | 773M | — | Fine |
+
+## New Contract: `disk-gc-threat-response.prose.md`
+
+Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
+
+### Three-in-One
+
+1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
+
+2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
+
+3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
+
+### The incident root cause
+Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
+
+Want me to push this to the prose-contracts repo?

hermes-config-template.prose.md

@@ -1,17 +1,11 @@
 ---
-kind: pattern
+kind: template
 name: hermes-config-template
 description: >
   Standard Hermes configuration template for Syslog Solution LLC agents.
   Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
   RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  Updated 2026-07-03 (19:14 EDT): Full sweep — Koby (11 fixes), Mumuni (1 fix),
+  Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
-  Koonimo (3 fixes). All agents now CLEAN: zero model:auto, zero deprecated
-  sk-syslog-* keys. Tanko has hardcoded tanko-v2 key (correct, but should use
-  api_key_env for rotation safety). Kagenz0 is DOWN (no config, no key).
-  Previous: full key audit — mumuni-v3 + context7-mcp registered,
-  /etc/environment aligned, double-B8nw incident documented, all 17 LiteLLM
-  keys mapped with active/inactive status.
 ---
 
 ## Maintains
@@ -19,210 +13,33 @@ description: >
 - template_version: "2.0.0"
 - last_applied: timestamp
 - agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
-- agent_keys: managed in LiteLLM DB — current aliases in Key Management section below
+- agent_keys: map (see Agent Keys section)
 - infra_endpoints_verified: array
 
-### Agent topology
+## Agent Keys (LiteLLM — Current 2026-06-30)
 
-| Agent | CT | IP | Key Alias | Startup | Zulip Owner |
+Each agent has a unique LiteLLM API key. Keys are stored in LiteLLM's PostgreSQL DB.
-|-------|-----|-----|-----------|---------|-------------|
+The env var `LITELLM_API_KEY` is set in `/etc/environment` on each agent host.
-| Tanko | 112 | 192.168.68.122 | `tanko-v3` | systemd user + `.env` | Jerome only ✅ |
+Sub-agent profiles inherit auth from the main config — no separate keys needed.
-| Mumuni | 114 | 192.168.68.123 | `mumuni-v3` | systemd user + `.env` | Jerome only ✅ |
-| Koonimo | 113 | 192.168.68.114 | `koonimo-v2` | systemd (Discord/Telegram) | N/A |
-| Koby | 111 | 192.168.68.129 | `koby-v3` | systemd user (Discord/Telegram) | N/A |
-| Abiba | — | 192.168.68.24 | `abiba` | `pi` harness | N/A |
-| Kagenz0 | ? | ? | `kagenz0` | unknown | not verified |
 
-### Zulip ownership rules (ALL agents)
+| Agent | Key | Host | SSH | Sub-Agents |
+|-------|-----|------|-----|-----------|
+| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | 192.168.68.122 | jerome@.122 | — |
+| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
+| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | 192.168.68.24 | local | — |
+| Koby | `sk-wb86PvcZXWkjShAIBxrpDg` | ? | Zulip | — |
+| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | 192.168.68.114 | Zulip | — |
+| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | Zulip | — |
 
-**Every Zulip-enabled agent MUST only respond to its owner.** This is enforced
+✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
-via two env vars in the agent's `.env` file:
+  syslog-review, syslog-writer — all at `/root/.hermes/profiles/<name>/config.yaml`
-
-```bash
-ZULIP_ALLOW_ALL_USERS=false
-ZULIP_ALLOWED_USERS=jerome@sysloggh.com
-```
-
-- `ZULIP_ALLOW_ALL_USERS=false` — disables the default open-to-everyone mode
-- `ZULIP_ALLOWED_USERS=jerome@sysloggh.com` — only Jerome can trigger responses
-
-This applies regardless of whether the agent uses `.env` directly (Mumuni)
-or a systemd drop-in (Tanko). Both mechanisms work — the key is that the
-env vars reach the Hermes process.
-
-## Key Management
-
-**Source of truth:** LiteLLM PostgreSQL DB. Plaintext keys live ONLY in
-`/etc/environment` on each agent host, never hardcoded in contracts or config.yaml.
-This contract defines the procedure; it does NOT store plaintext keys.
-
-### Current key aliases (2026-07-03 audit)
-
-**Active keys (by agent):**
-
-| Agent | Alias | Key (last 8) | Source | Status |
-|-------|-------|-------------|--------|--------|
-| Tanko | `tanko-v3` | ...JN2OsJw | Process env | Active (needs restart → v2) |
-| Tanko | `tanko-v2` | ...1ka8JA | `/etc/environment` | Registered, idle |
-| Mumuni | `mumuni-v3` | ...OiWU_B8nw | Process + /etc/env | ✅ Aligned |
-| Koonimo | `koonimo-v3` | ...A_wIOA | Config (hardcoded) | Active |
-| Koonimo | `koonimo-v2` | ...R7FAobA | `/etc/environment` | Registered, idle |
-| Koby | `koby-v3` | ...P4DrWw | `/etc/environment` | ✅ Registered |
-| Koby | `koby-hermes` | ...syslog-koby | Config (deprecated) | Active legacy |
-| Abiba | `abiba` | ...aLgMw | pi config | ✅ Active |
-| Kagenz0 | `kagenz0` | ...Uggg | unknown | Last active Jul 1 |
-| Context7 | `context7-mcp` | ...ZPh6VA | Process env | ✅ Registered |
-
-**Unused registrations:** `abiba-v2`, `kagenz0-v2`, `koonimo-v2`, `synthetic-monitor`, `UI Dashboard`, unnamed key
-
-> **2026-07-03 alignment**: Tanko/Koonimo configs updated to use `api_key_env:
-> LITELLM_API_KEY` (was hardcoded `sk-syslog-*`). Base URLs corrected to
-> `/v1` (was `/litellm/v1`). All active keys now registered in LiteLLM.
-> Tanko and Koonimo need Hermes gateway restart for config changes to take
-> effect.
-
-### Key lineage
-
-| Agent | Alias | Key (last 8) | Status | Date |
-|-------|-------|-------------|--------|------|
-| Tanko | `tanko-v3` | ...JN2OsJw | **active** (process) | 2026-07-03 |
-| Tanko | `tanko-v2` | ...1ka8JA | idle (in /etc/env) | 2026-07-02 |
-| Mumuni | `mumuni-v3` | ...OiWU_B8nw | **active** | 2026-07-03 |
-| Mumuni | `mumuni-v2` | ...HcnuOPtQ | superseded | 2026-07-02 |
-| Mumuni | `mumuni` | ...62da807e | superseded | 2026-06-30 |
-| Koonimo | `koonimo-v3` | ...A_wIOA | **active** (hardcoded) | 2026-07-03 |
-| Koonimo | `koonimo-v2` | ...R7FAobA | idle (in /etc/env) | 2026-07-02 |
-| Koby | `koby-v3` | ...P4DrWw | **active** (/etc/env) | 2026-07-03 |
-| Koby | `koby-hermes` | ...syslog-koby | legacy (in config) | 2026-06-30 |
-
-> **2026-07-03 incidents**:
-> - Mumuni: double-B8nw — process had `...POiWU_B8nw`, /etc/environment
->   had `...VDtHWd-XiWU_B8nw`. Wrong one registered first.
-> - Context7 MCP: `sk-XY2aUfvy2BIs6kp1ZPh6VA` was never registered — the
->   original error Kwame reported.
-> - Tanko: process key `sk-3ZsdWJbbNSo9zSnJN2OsJw` unregistered. Config
->   used deprecated `sk-syslog-tanko`. Both fixed.
-> - Koonimo: hardcoded `sk-KFnE3Td6aKEBZxwVA_wIOA` unregistered. Base URL
->   used old `/litellm/v1` path. Both fixed.
-> - Koby: `/etc/environment` key `sk-x5BnZUUKvz3r8EdLP4DrWw` unregistered.
->   Config used deprecated `sk-syslog-koby`. Fixed.
-
-### Verify: all keys working
-```bash
-# Run from Abiba against each agent host
-for agent in tanko mumuni; do
-  host=192.168.68.$([ "$agent" = tanko ] && echo 122 || echo 123)
-  key=$(ssh root@$host "grep LITELLM_API_KEY /etc/environment | cut -d= -f2")
-  code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $key" http://192.168.68.116/v1/models)
-  printf "  %-10s -> %s\n" "$agent" "$code"
-done
-```
-
-### Rotate: generate new key, deploy, verify
-```bash
-AGENT=mumuni && ALIAS="${AGENT}-v3"
-
-# 1. Generate new key in LiteLLM (master key required)
-KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \
-  -H 'Authorization: Bearer sk-litellm-7f960...' \
-  -H 'Content-Type: application/json' \
-  -d "{\"key_alias\":\"$ALIAS\",\"models\":[\"gemma-4-12b\",\"qwen3.6-27B-code\",\"ornith-1.0-35b\",\"syslog-auto\"],\"max_budget\":1000}" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("key"))')
-
-# 2. Verify new key
-curl -s -H "Authorization: Bearer $KEY" http://192.168.68.116/v1/models
-
-# 3. Deploy: update /etc/environment on agent host
-ssh root@192.168.68.123 "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=$KEY/' /etc/environment"
-
-# 4. Restart agent (systemd or kill + relaunch)
-ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f hermes_cli.main; cd /root/.hermes && nohup python -m hermes_cli.main gateway run > /dev/null 2>&1 &)"
-```
-
-### Key deployment rules
-1. **Plaintext keys live ONLY in `/etc/environment`** — never in a contract or config.yaml
-2. **All configs use `api_key_env: LITELLM_API_KEY`** — indirection prevents staleness
-3. **`custom_providers[].api_key` MUST use `api_key_env`** — hardcoded keys shadow env vars and bypass rotation
-4. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access
-5. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly
-6. **After rotating any key:** restart the corresponding agent gateway
-7. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login
-8. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid)
-
-### Model name enforcement (2026-07-03)
-
-**CRITICAL**: Hermes `model: auto` in any config section gets sent to LiteLLM
-as the literal model name "auto", which LiteLLM rejects with 403 because
-the registered model is `syslog-auto`, not `auto`.
-
-**Must-fix pattern** — any auxiliary section with `model: auto`:
-```yaml
-# ❌ BROKEN — sends model "auto" to LiteLLM (403 Forbidden)
-auxiliary:
-  approval:
-    model: auto
-    provider: auto
-  skills_hub:
-    model: auto
-    provider: auto
-
-# ✅ FIXED — registered model name
-auxiliary:
-  approval:
-    model: syslog-auto
-    provider: harness
-    api_key_env: LITELLM_API_KEY
-    base_url: http://192.168.68.116/v1
-```
-
-**Detection**: `grep -rn "model: auto" /root/.hermes/config.yaml` on any
-Hermes host — if any results found in auxiliary sections, fix immediately.
-
-### Key-to-agent mapping (2026-07-03 final sweep, 19:14 EDT)
-
-| Agent | CT | Key Alias | Config Status | Key Verified |
-|-------|-----|-----------|---------------|-------------|
-| Tanko | 112 | tanko-v2 | ⚠️ Hardcoded (correct key, no api_key_env) | ✅ 200 |
-| Mumuni | 114 | mumuni-v3 | ✅ Clean (api_key_env) | ✅ 200 |
-| Koby | 111 | koby-v3 | ✅ Clean (11 fixes applied) | ✅ 200 |
-| Koonimo | 113 | koonimo-v2 | ✅ Clean (3 fixes applied) | ✅ 200 |
-| Abiba | 100 | abiba | ✅ pi native | ✅ 200 |
-| Kagenz0 | 105 | kagenz0-v2 | ❌ DOWN (no hermes, no config) | — |
-
-### Fix summary (2026-07-03 second pass)
-
-**Koby (11 fixes):**
-- 7 auxiliary sections: model:auto → model:syslog-auto (compression, flush_memories, mcp, session_search, skills_hub, vision, web_extract)
-- custom_providers.harness: deprecated sk-syslog-koby → api_key_env, model:auto → syslog-auto
-- delegation: model:auto → syslog-auto, deprecated sk-syslog-koby → api_key_env
-
-**Mumuni (1 fix):**
-- delegation: model:auto → syslog-auto + provider:harness + api_key_env
-
-**Koonimo (3 fixes, first pass):**
-- auxiliary.approval: model:auto → syslog-auto, provider:auto → harness
-- auxiliary.skills_hub: model:auto → syslog-auto, provider:auto → harness  
-- custom_providers.harness: hardcoded koonimo-v3 → api_key_env
-
-**Tanko (flagged, not urgent):**
-- Config has hardcoded tanko-v2 key (correct) instead of api_key_env
-- DeepSeek fallback key (sk-b7d9...) is intentional — external service
-- Will need fix before next key rotation
-
-### Mumuni sub-agent profiles (6)
-Sub-agents in `/root/.hermes/profiles/<name>/config.yaml`:
-`syslog-code`, `syslog-devops`, `syslog-email`, `syslog-research`,
-`syslog-review`, `syslog-writer`.
-
-All sub-agent profiles inherit auth: `api_key: ''`, `base_url: ''`.
-When the main key is rotated, all 7 configs (main + 6 subs) stay valid —
-only `/etc/environment` needs updating.
 
 ## Infrastructure Stack
 
 | Component | Endpoint | Purpose |
 |---|---|---|
 | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
-| SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search |
+| SearXNG | `http://storepve:8888` | Privacy-respecting web search |
 | LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
 | LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
 | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |

infrastructure-control.prose.md

@@ -178,19 +178,18 @@ description: >
 |-----------|-------|------|------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | API proxy, key mgmt, fallbacks |
 | harness-router | inference-harness-router | :9000 (127.0.0.1) | GPU routing, slot booking, CB |
-| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/, /grafana/ |
+| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/ |
 | harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) |
 | harness-redis | redis:7-alpine | :6379 | Router slots, circuit breakers |
 | harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI |
-| harness-grafana | grafana/grafana | :3000→:3001 | GPU dashboards via Prometheus |
+| harness-grafana | grafana/grafana | :3000→:3001 (direct LAN, not behind nginx) | GPU + Proxmox + Docker dashboards |
-| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 5 targets |
+| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 6 jobs |
 
 **Nginx routing**:
 - `/v1/*` → harness-litellm:4000 (API)
 - `/admin/*` → harness-litellm:4000
 - `/dashboard/` → harness-dashboard:3000
 - `/litellm/*` → harness-litellm:4000
-- `/grafana/` → harness-grafana:3000
 - `/health/*` → harness-litellm:4000/health/liveliness
 - `/gpu/` → 192.168.68.24:9100 (fleet monitor)
 

infrastructure-update.prose.md

@@ -0,0 +1,156 @@
+---
+kind: responsibility
+name: infrastructure-update
+description: >
+  Autonomous system-wide update contract covering all 5 Proxmox nodes,
+  15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
+  Docker images, and container stacks in safe waves with health checks
+  and automatic rollback on failure.
+agent: abiba
+triggers:
+  - on "infra update" command
+  - weekly (Sunday 03:00 EDT) via cron
+  - on security advisory relay from Mumuni
+version: 1.0.0
+---
+
+## Maintains
+
+- update-status: { phase, node, action, result, timestamp }
+- update-history: array of past update runs with results
+- security-state: { cve_count, last_patched, pending_updates }
+
+## Pre-Flight Checklist
+
+Before ANY update wave:
+1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
+2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
+3. ✅ Docker healthy on .7, .116
+4. ✅ LiteLLM health check passing
+5. ✅ Zulip server reachable
+6. ✅ GPU fleet healthy (all 3 GPUs)
+7. ✅ Disk >20% free on all nodes
+8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
+
+## Wave 1: Storage & Infra Nodes (lowest impact)
+
+| Target | Type | Command | Timeout |
+|--------|------|---------|---------|
+| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
+| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
+| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
+
+**Verify after Wave 1:**
+- Docker healthy on .7: `docker ps`
+- RA-H OS MCP responding: `curl 192.168.68.65:3100/mcp`
+- Zulip responding: `curl https://chat.sysloggh.net/api/v1/server_settings`
+
+## Wave 2: Compute & Agent Nodes
+
+| Target | Type | Command | Timeout |
+|--------|------|---------|---------|
+| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
+| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
+| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
+| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
+| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |
+
+**Verify after Wave 2:**
+- All CTs running: check via Proxmox API
+- LiteLLM healthy: `curl :4001/health/liveliness`
+- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- Zulip agents connected: check Mumuni/Tanko gateway state
+- Abiba PM2 processes online: `pm2 status`
+
+## Wave 3: Docker Image Updates
+
+| Host | Stack | Command |
+|------|-------|---------|
+| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
+
+**Verify after Wave 3:**
+- All containers healthy: `docker ps` on each host
+- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
+- Zulip test: send test message to #agent-hub
+- Dashboard loading: `curl :3001/`
+- Firecrawl test: `curl :3002/health`
+- SearXNG test: `curl storepve:8888`
+
+## Wave 4: Proxmox Kernel Reboot (if needed)
+
+Only if `[ -f /var/run/reboot-required ]` on any node.
+
+| Target | Action |
+|--------|--------|
+| Affected PVE node | Verify all CTs/VMs migrated or stopped |
+| | `reboot` via PVE API |
+| | Wait 120s for node to come back |
+| | Start any stopped CTs |
+
+## Rollback Protocol
+
+If ANY verification fails:
+
+1. **Apt rollback**: Restore from Proxmox snapshot if taken, or `apt install <pkg>=<old_version>`
+2. **Docker rollback**: `docker compose down && docker compose up -d` (uses cached images)
+3. **Config rollback**: Restore from `/tmp/infra-update-backup-<date>/` snapshots
+4. **Escalate**: Send Zulip DM with failure details if auto-rollback fails
+
+## Config Backup
+
+Before Wave 1, snapshot these files:
+```
+/opt/inference-harness/litellm_config.yaml        (.116)
+/opt/inference-harness/docker-compose.yml          (.116)
+/etc/nginx/nginx.conf                              (.116 container)
+/opt/search-stack/*/docker-compose.yaml            (.7)
+/root/.pi/agent/extensions/config.yaml             (.24)
+/etc/systemd/system/ornith-server.service          (.15)
+/etc/systemd/system/llama-server.service           (.8, .110)
+```
+
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
+
+## Security-Specific Updates
+
+| Check | Command | Action |
+|-------|---------|--------|
+| CVE count | `apt list --upgradable 2>/dev/null \| grep -i security \| wc -l` | Report in update summary |
+| Kernel vulns | `uname -r` vs latest available | Flag if >2 versions behind |
+| Docker CVEs | `docker scout quickview` or `trivy image` | Flag critical CVEs |
+| SSL certs | `openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null \| openssl x509 -noout -dates` | Alert if <30 days |
+
+## Success Criteria
+
+- [ ] All 5 PVE nodes updated, no reboot-loop
+- [ ] All CTs running post-update
+- [ ] All 22 Docker containers healthy
+- [ ] LiteLLM inference passing (syslog-auto test)
+- [ ] Zulip server + all 3 agents connected
+- [ ] GPU fleet at full capacity (3/3)
+- [ ] Zero security CVEs remaining
+- [ ] <10 min total downtime per service
+
+## Report
+
+After completion, send Zulip DM:
+```
+📋 Infrastructure Update — YYYY-MM-DD
+
+Updated: 5 PVE nodes, 12 CTs, 22 containers
+Security fixes: N CVEs patched
+Downtime: <service> <duration>
+Failures: none / <details>
+Configs backed up: /tmp/infra-update-backup-YYYYMMDD/
+```

litellm-self-heal.prose.md

@@ -1,6 +1,11 @@
 ---
 kind: responsibility
 name: litellm-self-heal
+status: manual-only
+note: >
+  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
+  This contract is now manual-only — triggers require explicit user request.
+  Consider reimplementing as a standalone cron job or prose contract.
 description: >
   Standing responsibility that monitors LiteLLM health and proactively
   fixes common issues. Reports every action via Zulip DM and RA-H OS

pi-approval-architecture.prose.md

@@ -1,5 +1,5 @@
 ---
-kind: pattern
+kind: architecture
 name: pi-approval-architecture
 description: >
   Documents pi's approval model vs Hermes, what commands are available,

pm2-self-heal.prose.md

@@ -10,10 +10,13 @@ description: >
 
 ## Maintains
 
-- abiba-zulip: { status: "online", uptime: string, restarts: number }
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
+- gpu-watchdog: { status: "online", uptime: string, restarts: number }
+- gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp
 
+> **Note (2026-07-04):** `abiba-zulip` removed — Zulip extension decommissioned.
+
 ## Continuity
 
 - Self-driven: check every 300 seconds (5 min)

proxmox-monitor.prose.md

@@ -7,7 +7,7 @@ description: >
   exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
   instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
   (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
-  the layerdb). Dashboards exposed at http://192.168.68.116/grafana/.
+  the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
 agent: abiba
 ---
 
@@ -92,7 +92,7 @@ agent: abiba
 ## Operations
 
 ### view-dashboards
-Open `http://192.168.68.116/grafana/` → "Syslog Fleet" folder
+Open `http://192.168.68.116:3001/` → "Syslog Fleet" folder
 
 ### add-dashboard
 Edit `build-dashboards.py`, run it, `docker restart harness-grafana`

runs/20260704-005804-51f9a9/receipts/disk-gc-threat-response.jsonl

@@ -0,0 +1 @@
+{"node":"disk-gc-threat-response","status":"rendered","fingerprints":{"@atomic":"fleet-18-green-0-amber-0-red"},"semantic_diff":"Initial fleet baseline scan: 18 hosts scanned, all GREEN. Previously resolved CT 105 kagentz from 87%→27%. CT 118 jitsi offline.","cost":{},"prev":null,"timestamp":"2026-07-04T00:58:30Z"}

runs/20260704-005804-51f9a9/vm.log.md

@@ -0,0 +1,26 @@
+# run:20260704-005804-51f9a9 disk-gc-threat-response
+root: disk-gc-threat-response
+
+1→ [scan] fleet-wide disk scan started
+2→ CT 100 abiba (amdpve): 12G/59G 21% GREEN
+2→ CT 101 llm-gpu (acerpve): 144G/255G 60% GREEN
+2→ CT 102 adguard (acerpve): 5.9G/32G 20% GREEN
+2→ CT 103 ocu-llm (ocupve): 48G/186G 27% GREEN
+2→ CT 104 authentik (minipve): 4.6G/9.8G 50% GREEN
+2→ CT 105 kagentz (amdpve): 16G/59G 27% GREEN (resolved: was 87%, pruned 35.67G)
+2→ CT 106 ra-h-os (storepve): 2.1G/20G 11% GREEN
+2→ CT 107 pbs (storepve): 1.1G/40G 3% GREEN
+2→ CT 108 media (storepve): 19G/196G 10% GREEN
+2→ CT 109 docker-vm (storepve): 25G/158G 17% GREEN
+2→ CT 110 gitea (minipve): 5.6G/40G 16% GREEN
+2→ CT 111 tdunna (amdpve): 16G/63G 26% GREEN
+2→ CT 112 tanko (amdpve): 17G/49G 36% GREEN
+2→ CT 113 baggy (amdpve): 7.0G/52G 15% GREEN
+2→ CT 114 mumuni (minipve): 29G/59G 51% GREEN
+2→ CT 115 scottdenya (amdpve): 14G/59G 25% GREEN
+2→ CT 116 syslog-api (minipve): 16G/40G 42% GREEN
+2→ CT 117 zulip (storepve): 6.1G/60G 11% GREEN
+2→ CT 118 jitsi (minipve): OFFLINE — no disk scan
+3→ [summary] Fleet: 19 hosts, 18 scanned, 18 GREEN, 0 AMBER, 0 RED, 0 CRITICAL, 1 OFFLINE
+3→ [summary] Previously resolved: CT 105 kagentz 87%→27% (35.67GB reclaimed)
+---end 2026-07-04T00:58:04Z

runs/20260704-005804-51f9a9/world-model/disk-gc-threat-response/disk-health.md

@@ -0,0 +1,52 @@
+# Fleet Disk Health — 2026-07-04 00:58 UTC
+
+## Summary
+
+| Metric | Count |
+|--------|-------|
+| Total hosts | 19 |
+| Scanned | 18 |
+| GREEN (<75%) | 18 |
+| AMBER (75-84%) | 0 |
+| RED (85-94%) | 0 |
+| CRITICAL (≥95%) | 0 |
+| OFFLINE | 1 (jitsi CT 118) |
+
+## Per-Host Status
+
+| CT | Name | Node | Used | Total | % | Status |
+|----|------|------|------|-------|---|--------|
+| 100 | abiba | amdpve | 12G | 59G | 21% | 🟢 GREEN |
+| 101 | llm-gpu | acerpve | 144G | 255G | 60% | 🟢 GREEN |
+| 102 | adguard | acerpve | 5.9G | 32G | 20% | 🟢 GREEN |
+| 103 | ocu-llm | ocupve | 48G | 186G | 27% | 🟢 GREEN |
+| 104 | authentik | minipve | 4.6G | 9.8G | 50% | 🟢 GREEN |
+| 105 | kagentz | amdpve | 16G | 59G | 27% | 🟢 GREEN |
+| 106 | ra-h-os | storepve | 2.1G | 20G | 11% | 🟢 GREEN |
+| 107 | pbs | storepve | 1.1G | 40G | 3% | 🟢 GREEN |
+| 108 | media | storepve | 19G | 196G | 10% | 🟢 GREEN |
+| 109 | docker-vm | storepve | 25G | 158G | 17% | 🟢 GREEN |
+| 110 | gitea | minipve | 5.6G | 40G | 16% | 🟢 GREEN |
+| 111 | tdunna | amdpve | 16G | 63G | 26% | 🟢 GREEN |
+| 112 | tanko | amdpve | 17G | 49G | 36% | 🟢 GREEN |
+| 113 | baggy | amdpve | 7.0G | 52G | 15% | 🟢 GREEN |
+| 114 | mumuni | minipve | 29G | 59G | 51% | 🟢 GREEN |
+| 115 | scottdenya | amdpve | 14G | 59G | 25% | 🟢 GREEN |
+| 116 | syslog-api | minipve | 16G | 40G | 42% | 🟢 GREEN |
+| 117 | zulip | storepve | 6.1G | 60G | 11% | 🟢 GREEN |
+| 118 | jitsi | minipve | — | — | — | ⚫ OFFLINE |
+
+## Docker Hosts - GC Status
+
+| Host | Docker present | Last GC | Status |
+|------|---------------|---------|--------|
+| kagentz (CT 105) | Yes | 2026-07-04 (just now) | 35.67GB reclaimed, 27% |
+| syslog-api (CT 116) | Yes | Never | 42%, no action needed |
+| docker-vm (CT 109) | Yes | Unknown | 17%, no action needed |
+| abiba (CT 100) | Yes | Never | 21%, /var/lib/docker=2.8G |
+
+## Notes
+- CT 118 (jitsi) is not running — no disk scan possible. May need `qm start 118` if service is needed.
+- CT 114 (mumuni) at 51% — closest to AMBER threshold (75%). Agent memory accumulation likely.
+- CT 105 (kagentz) resolved from 87% RED → 27% GREEN. Root cause: docker image bloat. Prevented: GC contract in place.
+- All Docker hosts are healthy. Next automated scan in 6 hours.

zulip-adapter-lessons.prose.md

@@ -1,6 +1,10 @@
 ---
 kind: pattern
 name: zulip-adapter-lessons
+status: historical
+note: >
+  pi Zulip extension retired 2026-07-04. These lessons remain relevant for
+  Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.
 description: >
   Lessons learned from building and debugging the pi Zulip extension and
   Hermes Zulip plugin. Documents failure modes, fixes, and patterns that

zulip-mention-reliability.prose.md

@@ -1,13 +1,18 @@
 ---
 kind: responsibility
 name: zulip-mention-reliability
+status: retired
 description: >
-  Monitors whether Abiba Bot is correctly detecting and responding to
+  RETIRED 2026-07-04 — Zulip extension decommissioned. Abiba now operates
-  @mentions in Zulip stream topics. Tests periodically, diagnoses
+  exclusively through Telegram (@AbibaBot). This contract is kept for
-  failures, and attempts fixes. Same self-improving pattern as
+  historical reference only.
-  litellm-self-heal.
 ---
 
+> **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and
+> PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability
+> monitoring now happens through Telegram. Hermes agents (Tanko, Mumuni) and
+> Agent Zero (kagentz) continue to use Zulip.
+
 ## Maintains
 
 - mention_response_rate: number — % of @mentions that get a response (target: >95%)

zulip-self-heal.prose.md

@@ -0,0 +1,92 @@
+---
+kind: responsibility
+name: zulip-self-heal
+status: retired
+description: >
+  RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
+  Self-healing for Zulip infrastructure continues through Hermes agents.
+  Abiba no longer monitors or manages Zulip.
+agent: abiba
+triggers:
+  - none (retired)
+---
+
+> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
+> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
+> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
+
+## Maintains
+
+- zulip-server: { status: "healthy" | "degraded" | "down" }
+- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
+- zulip-config: { syntax_valid: bool, proxy_configured: bool }
+- last_action: { timestamp, action, result }
+
+## Detection Rules
+
+### Rule 1: Zulip Server Down (502/000)
+- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
+- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
+- **Fix**: `docker restart zulip-zulip-1` if container crashed
+- **Verify**: Re-test server_settings, wait for "healthy"
+
+### Rule 2: Zulip Config Corruption (400/500 non-proxy)
+- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
+- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
+- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
+- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
+
+### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
+- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
+- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
+- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
+- **Verify**: Re-test via NetBird URL
+
+### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
+- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
+- **Fix (pi)**: `pm2 restart abiba-zulip`
+- **Fix (Hermes)**: `hermes gateway restart` on agent host
+- **Verify**: Check health endpoint for `queue_healthy: true`
+
+### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
+- **Detect**: No heartbeat in agent log for >600s
+- **Fix**: Restart gateway on affected host
+- **Verify**: Check for new heartbeat in logs within 60s
+
+### Rule 6: NetBird Tunnel Down
+- **Detect**: Server returns 502 with NetBird HTML in response
+- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
+- **Fix**: Alert user — NetBird outage requires manual intervention
+- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
+
+## Hosts & Credentials
+
+| Host | IP | User | Service |
+|------|----|------|---------|
+| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
+| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
+| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
+| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
+
+## Debounce
+
+All restart actions must debounce: minimum 300s between restarts per agent.
+Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
+
+## Reporting
+
+Every cycle produces a knowledge graph node:
+- Title: `[LEARN] zulip-self-heal: <timestamp>`
+- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
+- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
+- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
+
+## Configuration
+
+Zulip server config at 192.168.68.19:
+- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
+- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
+- Fixed values:
+  - `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
+  - `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
+  - `LOAD_BALANCER_IPS = ["192.168.68.10"]`