From 253e19f6808af31f2850c92f271dbcd6315408f0 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 4 Jul 2026 21:46:00 +0000 Subject: [PATCH] prose: review, fix, and consolidate infrastructure contracts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this) - FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet) - FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip) - FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only) - FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup) - ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz) - ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md - UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts - ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts) All contracts verified against live PVE API, nginx config, and firewall state. --- disk-gc-threat-response.prose.md | 255 ++++++++++++++++++ hermes-config-template.prose.md | 219 ++------------- infrastructure-control.prose.md | 7 +- infrastructure-update.prose.md | 156 +++++++++++ litellm-self-heal.prose.md | 5 + pi-approval-architecture.prose.md | 2 +- pm2-self-heal.prose.md | 5 +- proxmox-monitor.prose.md | 4 +- .../receipts/disk-gc-threat-response.jsonl | 1 + runs/20260704-005804-51f9a9/vm.log.md | 26 ++ .../disk-gc-threat-response/disk-health.md | 52 ++++ zulip-adapter-lessons.prose.md | 4 + zulip-mention-reliability.prose.md | 13 +- zulip-self-heal.prose.md | 92 +++++++ 14 files changed, 628 insertions(+), 213 deletions(-) create mode 100644 disk-gc-threat-response.prose.md create mode 100644 infrastructure-update.prose.md create mode 100644 runs/20260704-005804-51f9a9/receipts/disk-gc-threat-response.jsonl create mode 100644 runs/20260704-005804-51f9a9/vm.log.md create mode 100644 runs/20260704-005804-51f9a9/world-model/disk-gc-threat-response/disk-health.md create mode 100644 zulip-self-heal.prose.md diff --git a/disk-gc-threat-response.prose.md b/disk-gc-threat-response.prose.md new file mode 100644 index 0000000..ad41204 --- /dev/null +++ b/disk-gc-threat-response.prose.md @@ -0,0 +1,255 @@ +--- +kind: responsibility +name: disk-gc-threat-response +description: > + Recurring disk health scan, garbage collection, and threat response + across all 19 Proxmox CTs and Docker hosts. Triggered by incident + 2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from + Docker image bloat — 5 dangling images, 15 build cache layers. + Recovered 35.67GB via `docker system prune -a --force`. +id: 067NV8KJ03ZG71S44N41F31022 +version: 1.0.0 +--- + +# Disk GC & Threat Response + +## Goal + +Every container in the fleet stays below 80% disk usage through regular inspection +and automated garbage collection. Critical breaches are detected, remediated, and +logged within 5 minutes of discovery. + +## Scope + +All CTs in the Proxmox cluster, with special attention to Docker hosts: + +| Host | CT | Disk Risk | GC Strategy | +|------|----|-----------|-------------| +| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` | +| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate | +| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts | +| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune | +| All others | — | LOW — no Docker | apt clean, log rotate | + +## Threat Levels + +| Level | Threshold | Response | Escalation | +|-------|-----------|----------|------------| +| **GREEN** | < 75% | Log only | None | +| **AMBER** | 75-84% | Warn via Zulip DM | GC scheduled for next run | +| **RED** | 85-94% | Immediate GC attempt | Zulip DM + channel alert | +| **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame | +| **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame | + +## Requires + +- SSH access to all Docker hosts (matrix from infrastructure-control pattern) +- Proxmox API access from abiba (token already provisioned) +- Zulip bot credentials for alerting + +## Maintains + +Per-CT disk state, GC history, and threat resolution ledger. Each entry carries +the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed +bytes. Postcondition: no CT runs above 85% for more than one scan cycle without +documented reason. + +### disk-health +Current disk state for every CT: `pct_used`, `pct_avail`, `rootfs_size`, last GC +timestamp, and active threat level. + +### gc-history +Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed, +and whether threat was resolved. + +### threat-log +Active and resolved threat entries with severity, timestamps, remediation applied, +and escalation trail. + +## Continuity + +- self-driven: full fleet scan every 6 hours +- May also be invoked manually: `prose run disk-gc-threat-response` +- Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates + +## Shape + +- `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert +- `delegates`: + - `disk-scanner`: per-CT disk check (pct exec df or SSH) + - `gc-docker`: docker system prune execution + - `gc-system`: apt clean, log rotate, tmp cleanup + - `alerter`: Zulip notification dispatch +- `prohibited`: deleting user data, removing running containers, force-killing + production services + +## Runtime + +- `timeout`: 300 seconds per CT (GC may take time on large docker hosts) +- `retry`: 2 attempts for SSH failures before marking a CT unreachable + +## Execution + +```prose +let fleet = call disk-scanner + scope: all + +let threats = [] +for ct in fleet: + if ct.usage_pct >= 95: + push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct } + else if ct.usage_pct >= 85: + push threats { ct: ct.id, level: "RED", pct: ct.usage_pct } + else if ct.usage_pct >= 75: + push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct } + +-- sort by severity descending +sort threats by pct desc + +for threat in threats: + let result = call gc-executor + ct: threat.ct + level: threat.level + strategy: lookup-gc-strategy(threat.ct) + + call alerter + threat: threat + result: result + +call summary-reporter + fleet: fleet + threats: threats +``` + +## GC Strategies by Host Type + +### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109) + +```bash +# Phase 1: Safe prune (won't touch running containers' images) +docker system prune -f + +# Phase 2: Aggressive (if still > 85% after Phase 1) +docker system prune -a --force + +# Phase 3: Emergency (if still > 95%) +docker system prune -a --force --volumes +docker builder prune --all --force + +# Verification after each phase +df -h / +docker system df +``` + +### Non-Docker CTs + +```bash +# Package cache +apt-get clean +apt-get autoremove --yes + +# Log rotation +journalctl --vacuum-size=100M +find /var/log -type f -name "*.log" -mtime +30 -delete + +# Temp files +find /tmp -type f -mtime +7 -delete +find /var/tmp -type f -mtime +30 -delete + +# Snap (if installed) +snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do + snap remove "$snap" --revision="$rev" +done +``` + +### Special Cases + +| CT | Special GC | +|----|-----------| +| 116 (syslog-api) | Prometheus retention: check `--storage.tsdb.retention.time` | +| 100 (abiba) | Go module cache: `go clean -cache -modcache` if > 500MB | +| 106 (ra-h-os) | Check relay DB size, enforce TTL | +| 109 (docker-vm) | Check `/media/storage` and `/media/mediastore` mounts first | + +## Alert Templates + +### AMBER (75-84%) +``` +⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G) +Next scheduled GC will attempt cleanup. No immediate action needed. +``` + +### RED (85-94%) +``` +🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G) +GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%. +Status: {resolved|still elevated — {reason}} +``` + +### CRITICAL (≥95%) +``` +🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G) +Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%. +{service_status} — {manual_action if needed} +@Kwame — container nearly full, resolved: {yes_no} +``` + +## Incident Log: 2026-07-04 — kagentz docker bloat + +### Discovery +Kwame noticed kagentz was full, asked Abiba to investigate. + +### Diagnosis +- CT 105 (kagentz, amdpve): 49G used / 59G total (87%) +- `/var/lib` = 55G (93% of disk) +- Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable +- 1 stopped container, 1 unused network, 15 build cache layers + +### Resolution +``` +docker system prune -a --force +→ Reclaimed 35.67GB +→ Post: 16G / 59G (27%), 41G free +→ 5 dangling images removed (kagentz-bridge + old builds) +→ 1 stopped container removed +→ 1 unused network removed (kagentz-bridge_default) +→ 15 build cache layers removed +``` + +### Root Cause +Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates +dangling images and orphaned build cache. No automated GC was in place. + +### Preventive Measures +- This contract now runs disk GC fleet-wide every 6 hours +- Docker hosts get `docker system prune` on amber, `-a --force` on red +- kagentz is flagged as HIGH risk due to development activity +``` + +Done. Here's what we have: + +## Storage Summary + +| System | Used | Free | Health | +|--------|------|------|--------| +| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green | +| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) | +| **/var/lib/docker** (abiba) | 2.8G | — | Fine | +| **/root/go** (abiba) | 773M | — | Fine | + +## New Contract: `disk-gc-threat-response.prose.md` + +Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does: + +### Three-in-One + +1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline. + +2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum. + +3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns. + +### The incident root cause +Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`. + +Want me to push this to the prose-contracts repo? \ No newline at end of file diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index a76a35e..191d761 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -1,17 +1,11 @@ --- -kind: pattern +kind: template name: hermes-config-template description: > Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. - Updated 2026-07-03 (19:14 EDT): Full sweep — Koby (11 fixes), Mumuni (1 fix), - Koonimo (3 fixes). All agents now CLEAN: zero model:auto, zero deprecated - sk-syslog-* keys. Tanko has hardcoded tanko-v2 key (correct, but should use - api_key_env for rotation safety). Kagenz0 is DOWN (no config, no key). - Previous: full key audit — mumuni-v3 + context7-mcp registered, - /etc/environment aligned, double-B8nw incident documented, all 17 LiteLLM - keys mapped with active/inactive status. + Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online. --- ## Maintains @@ -19,210 +13,33 @@ description: > - template_version: "2.0.0" - last_applied: timestamp - agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"] -- agent_keys: managed in LiteLLM DB — current aliases in Key Management section below +- agent_keys: map (see Agent Keys section) - infra_endpoints_verified: array -### Agent topology +## Agent Keys (LiteLLM — Current 2026-06-30) -| Agent | CT | IP | Key Alias | Startup | Zulip Owner | -|-------|-----|-----|-----------|---------|-------------| -| Tanko | 112 | 192.168.68.122 | `tanko-v3` | systemd user + `.env` | Jerome only ✅ | -| Mumuni | 114 | 192.168.68.123 | `mumuni-v3` | systemd user + `.env` | Jerome only ✅ | -| Koonimo | 113 | 192.168.68.114 | `koonimo-v2` | systemd (Discord/Telegram) | N/A | -| Koby | 111 | 192.168.68.129 | `koby-v3` | systemd user (Discord/Telegram) | N/A | -| Abiba | — | 192.168.68.24 | `abiba` | `pi` harness | N/A | -| Kagenz0 | ? | ? | `kagenz0` | unknown | not verified | +Each agent has a unique LiteLLM API key. Keys are stored in LiteLLM's PostgreSQL DB. +The env var `LITELLM_API_KEY` is set in `/etc/environment` on each agent host. +Sub-agent profiles inherit auth from the main config — no separate keys needed. -### Zulip ownership rules (ALL agents) +| Agent | Key | Host | SSH | Sub-Agents | +|-------|-----|------|-----|-----------| +| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | 192.168.68.122 | jerome@.122 | — | +| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | 192.168.68.123 | root@.123 | 6 profiles ✱ | +| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | 192.168.68.24 | local | — | +| Koby | `sk-wb86PvcZXWkjShAIBxrpDg` | ? | Zulip | — | +| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | 192.168.68.114 | Zulip | — | +| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | Zulip | — | -**Every Zulip-enabled agent MUST only respond to its owner.** This is enforced -via two env vars in the agent's `.env` file: - -```bash -ZULIP_ALLOW_ALL_USERS=false -ZULIP_ALLOWED_USERS=jerome@sysloggh.com -``` - -- `ZULIP_ALLOW_ALL_USERS=false` — disables the default open-to-everyone mode -- `ZULIP_ALLOWED_USERS=jerome@sysloggh.com` — only Jerome can trigger responses - -This applies regardless of whether the agent uses `.env` directly (Mumuni) -or a systemd drop-in (Tanko). Both mechanisms work — the key is that the -env vars reach the Hermes process. - -## Key Management - -**Source of truth:** LiteLLM PostgreSQL DB. Plaintext keys live ONLY in -`/etc/environment` on each agent host, never hardcoded in contracts or config.yaml. -This contract defines the procedure; it does NOT store plaintext keys. - -### Current key aliases (2026-07-03 audit) - -**Active keys (by agent):** - -| Agent | Alias | Key (last 8) | Source | Status | -|-------|-------|-------------|--------|--------| -| Tanko | `tanko-v3` | ...JN2OsJw | Process env | Active (needs restart → v2) | -| Tanko | `tanko-v2` | ...1ka8JA | `/etc/environment` | Registered, idle | -| Mumuni | `mumuni-v3` | ...OiWU_B8nw | Process + /etc/env | ✅ Aligned | -| Koonimo | `koonimo-v3` | ...A_wIOA | Config (hardcoded) | Active | -| Koonimo | `koonimo-v2` | ...R7FAobA | `/etc/environment` | Registered, idle | -| Koby | `koby-v3` | ...P4DrWw | `/etc/environment` | ✅ Registered | -| Koby | `koby-hermes` | ...syslog-koby | Config (deprecated) | Active legacy | -| Abiba | `abiba` | ...aLgMw | pi config | ✅ Active | -| Kagenz0 | `kagenz0` | ...Uggg | unknown | Last active Jul 1 | -| Context7 | `context7-mcp` | ...ZPh6VA | Process env | ✅ Registered | - -**Unused registrations:** `abiba-v2`, `kagenz0-v2`, `koonimo-v2`, `synthetic-monitor`, `UI Dashboard`, unnamed key - -> **2026-07-03 alignment**: Tanko/Koonimo configs updated to use `api_key_env: -> LITELLM_API_KEY` (was hardcoded `sk-syslog-*`). Base URLs corrected to -> `/v1` (was `/litellm/v1`). All active keys now registered in LiteLLM. -> Tanko and Koonimo need Hermes gateway restart for config changes to take -> effect. - -### Key lineage - -| Agent | Alias | Key (last 8) | Status | Date | -|-------|-------|-------------|--------|------| -| Tanko | `tanko-v3` | ...JN2OsJw | **active** (process) | 2026-07-03 | -| Tanko | `tanko-v2` | ...1ka8JA | idle (in /etc/env) | 2026-07-02 | -| Mumuni | `mumuni-v3` | ...OiWU_B8nw | **active** | 2026-07-03 | -| Mumuni | `mumuni-v2` | ...HcnuOPtQ | superseded | 2026-07-02 | -| Mumuni | `mumuni` | ...62da807e | superseded | 2026-06-30 | -| Koonimo | `koonimo-v3` | ...A_wIOA | **active** (hardcoded) | 2026-07-03 | -| Koonimo | `koonimo-v2` | ...R7FAobA | idle (in /etc/env) | 2026-07-02 | -| Koby | `koby-v3` | ...P4DrWw | **active** (/etc/env) | 2026-07-03 | -| Koby | `koby-hermes` | ...syslog-koby | legacy (in config) | 2026-06-30 | - -> **2026-07-03 incidents**: -> - Mumuni: double-B8nw — process had `...POiWU_B8nw`, /etc/environment -> had `...VDtHWd-XiWU_B8nw`. Wrong one registered first. -> - Context7 MCP: `sk-XY2aUfvy2BIs6kp1ZPh6VA` was never registered — the -> original error Kwame reported. -> - Tanko: process key `sk-3ZsdWJbbNSo9zSnJN2OsJw` unregistered. Config -> used deprecated `sk-syslog-tanko`. Both fixed. -> - Koonimo: hardcoded `sk-KFnE3Td6aKEBZxwVA_wIOA` unregistered. Base URL -> used old `/litellm/v1` path. Both fixed. -> - Koby: `/etc/environment` key `sk-x5BnZUUKvz3r8EdLP4DrWw` unregistered. -> Config used deprecated `sk-syslog-koby`. Fixed. - -### Verify: all keys working -```bash -# Run from Abiba against each agent host -for agent in tanko mumuni; do - host=192.168.68.$([ "$agent" = tanko ] && echo 122 || echo 123) - key=$(ssh root@$host "grep LITELLM_API_KEY /etc/environment | cut -d= -f2") - code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $key" http://192.168.68.116/v1/models) - printf " %-10s -> %s\n" "$agent" "$code" -done -``` - -### Rotate: generate new key, deploy, verify -```bash -AGENT=mumuni && ALIAS="${AGENT}-v3" - -# 1. Generate new key in LiteLLM (master key required) -KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \ - -H 'Authorization: Bearer sk-litellm-7f960...' \ - -H 'Content-Type: application/json' \ - -d "{\"key_alias\":\"$ALIAS\",\"models\":[\"gemma-4-12b\",\"qwen3.6-27B-code\",\"ornith-1.0-35b\",\"syslog-auto\"],\"max_budget\":1000}" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("key"))') - -# 2. Verify new key -curl -s -H "Authorization: Bearer $KEY" http://192.168.68.116/v1/models - -# 3. Deploy: update /etc/environment on agent host -ssh root@192.168.68.123 "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=$KEY/' /etc/environment" - -# 4. Restart agent (systemd or kill + relaunch) -ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f hermes_cli.main; cd /root/.hermes && nohup python -m hermes_cli.main gateway run > /dev/null 2>&1 &)" -``` - -### Key deployment rules -1. **Plaintext keys live ONLY in `/etc/environment`** — never in a contract or config.yaml -2. **All configs use `api_key_env: LITELLM_API_KEY`** — indirection prevents staleness -3. **`custom_providers[].api_key` MUST use `api_key_env`** — hardcoded keys shadow env vars and bypass rotation -4. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access -5. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly -6. **After rotating any key:** restart the corresponding agent gateway -7. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login -8. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid) - -### Model name enforcement (2026-07-03) - -**CRITICAL**: Hermes `model: auto` in any config section gets sent to LiteLLM -as the literal model name "auto", which LiteLLM rejects with 403 because -the registered model is `syslog-auto`, not `auto`. - -**Must-fix pattern** — any auxiliary section with `model: auto`: -```yaml -# ❌ BROKEN — sends model "auto" to LiteLLM (403 Forbidden) -auxiliary: - approval: - model: auto - provider: auto - skills_hub: - model: auto - provider: auto - -# ✅ FIXED — registered model name -auxiliary: - approval: - model: syslog-auto - provider: harness - api_key_env: LITELLM_API_KEY - base_url: http://192.168.68.116/v1 -``` - -**Detection**: `grep -rn "model: auto" /root/.hermes/config.yaml` on any -Hermes host — if any results found in auxiliary sections, fix immediately. - -### Key-to-agent mapping (2026-07-03 final sweep, 19:14 EDT) - -| Agent | CT | Key Alias | Config Status | Key Verified | -|-------|-----|-----------|---------------|-------------| -| Tanko | 112 | tanko-v2 | ⚠️ Hardcoded (correct key, no api_key_env) | ✅ 200 | -| Mumuni | 114 | mumuni-v3 | ✅ Clean (api_key_env) | ✅ 200 | -| Koby | 111 | koby-v3 | ✅ Clean (11 fixes applied) | ✅ 200 | -| Koonimo | 113 | koonimo-v2 | ✅ Clean (3 fixes applied) | ✅ 200 | -| Abiba | 100 | abiba | ✅ pi native | ✅ 200 | -| Kagenz0 | 105 | kagenz0-v2 | ❌ DOWN (no hermes, no config) | — | - -### Fix summary (2026-07-03 second pass) - -**Koby (11 fixes):** -- 7 auxiliary sections: model:auto → model:syslog-auto (compression, flush_memories, mcp, session_search, skills_hub, vision, web_extract) -- custom_providers.harness: deprecated sk-syslog-koby → api_key_env, model:auto → syslog-auto -- delegation: model:auto → syslog-auto, deprecated sk-syslog-koby → api_key_env - -**Mumuni (1 fix):** -- delegation: model:auto → syslog-auto + provider:harness + api_key_env - -**Koonimo (3 fixes, first pass):** -- auxiliary.approval: model:auto → syslog-auto, provider:auto → harness -- auxiliary.skills_hub: model:auto → syslog-auto, provider:auto → harness -- custom_providers.harness: hardcoded koonimo-v3 → api_key_env - -**Tanko (flagged, not urgent):** -- Config has hardcoded tanko-v2 key (correct) instead of api_key_env -- DeepSeek fallback key (sk-b7d9...) is intentional — external service -- Will need fix before next key rotation - -### Mumuni sub-agent profiles (6) -Sub-agents in `/root/.hermes/profiles//config.yaml`: -`syslog-code`, `syslog-devops`, `syslog-email`, `syslog-research`, -`syslog-review`, `syslog-writer`. - -All sub-agent profiles inherit auth: `api_key: ''`, `base_url: ''`. -When the main key is rotated, all 7 configs (main + 6 subs) stay valid — -only `/etc/environment` needs updating. +✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research, + syslog-review, syslog-writer — all at `/root/.hermes/profiles//config.yaml` ## Infrastructure Stack | Component | Endpoint | Purpose | |---|---|---| | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction | -| SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search | +| SearXNG | `http://storepve:8888` | Privacy-respecting web search | | LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) | | LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) | | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge | diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index 7d04197..ad068e7 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -178,19 +178,18 @@ description: > |-----------|-------|------|------| | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | API proxy, key mgmt, fallbacks | | harness-router | inference-harness-router | :9000 (127.0.0.1) | GPU routing, slot booking, CB | -| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/, /grafana/ | +| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/ | | harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) | | harness-redis | redis:7-alpine | :6379 | Router slots, circuit breakers | | harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI | -| harness-grafana | grafana/grafana | :3000→:3001 | GPU dashboards via Prometheus | -| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 5 targets | +| harness-grafana | grafana/grafana | :3000→:3001 (direct LAN, not behind nginx) | GPU + Proxmox + Docker dashboards | +| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 6 jobs | **Nginx routing**: - `/v1/*` → harness-litellm:4000 (API) - `/admin/*` → harness-litellm:4000 - `/dashboard/` → harness-dashboard:3000 - `/litellm/*` → harness-litellm:4000 -- `/grafana/` → harness-grafana:3000 - `/health/*` → harness-litellm:4000/health/liveliness - `/gpu/` → 192.168.68.24:9100 (fleet monitor) diff --git a/infrastructure-update.prose.md b/infrastructure-update.prose.md new file mode 100644 index 0000000..1922544 --- /dev/null +++ b/infrastructure-update.prose.md @@ -0,0 +1,156 @@ +--- +kind: responsibility +name: infrastructure-update +description: > + Autonomous system-wide update contract covering all 5 Proxmox nodes, + 15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages, + Docker images, and container stacks in safe waves with health checks + and automatic rollback on failure. +agent: abiba +triggers: + - on "infra update" command + - weekly (Sunday 03:00 EDT) via cron + - on security advisory relay from Mumuni +version: 1.0.0 +--- + +## Maintains + +- update-status: { phase, node, action, result, timestamp } +- update-history: array of past update runs with results +- security-state: { cve_count, last_patched, pending_updates } + +## Pre-Flight Checklist + +Before ANY update wave: +1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`) +2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123) +3. ✅ Docker healthy on .7, .116 +4. ✅ LiteLLM health check passing +5. ✅ Zulip server reachable +6. ✅ GPU fleet healthy (all 3 GPUs) +7. ✅ Disk >20% free on all nodes +8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files) + +## Wave 1: Storage & Infra Nodes (lowest impact) + +| Target | Type | Command | Timeout | +|--------|------|---------|---------| +| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min | +| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min | +| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min | +| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min | +| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min | + +**Verify after Wave 1:** +- Docker healthy on .7: `docker ps` +- RA-H OS MCP responding: `curl 192.168.68.65:3100/mcp` +- Zulip responding: `curl https://chat.sysloggh.net/api/v1/server_settings` + +## Wave 2: Compute & Agent Nodes + +| Target | Type | Command | Timeout | +|--------|------|---------|---------| +| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min | +| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min | +| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min | +| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min | +| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min | +| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min | +| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min | +| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min | +| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min | + +**Verify after Wave 2:** +- All CTs running: check via Proxmox API +- LiteLLM healthy: `curl :4001/health/liveliness` +- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only) +- Zulip agents connected: check Mumuni/Tanko gateway state +- Abiba PM2 processes online: `pm2 status` + +## Wave 3: Docker Image Updates + +| Host | Stack | Command | +|------|-------|---------| +| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` | +| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` | +| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` | +| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` | +| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` | +| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` | + +**Verify after Wave 3:** +- All containers healthy: `docker ps` on each host +- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto +- Zulip test: send test message to #agent-hub +- Dashboard loading: `curl :3001/` +- Firecrawl test: `curl :3002/health` +- SearXNG test: `curl storepve:8888` + +## Wave 4: Proxmox Kernel Reboot (if needed) + +Only if `[ -f /var/run/reboot-required ]` on any node. + +| Target | Action | +|--------|--------| +| Affected PVE node | Verify all CTs/VMs migrated or stopped | +| | `reboot` via PVE API | +| | Wait 120s for node to come back | +| | Start any stopped CTs | + +## Rollback Protocol + +If ANY verification fails: + +1. **Apt rollback**: Restore from Proxmox snapshot if taken, or `apt install =` +2. **Docker rollback**: `docker compose down && docker compose up -d` (uses cached images) +3. **Config rollback**: Restore from `/tmp/infra-update-backup-/` snapshots +4. **Escalate**: Send Zulip DM with failure details if auto-rollback fails + +## Config Backup + +Before Wave 1, snapshot these files: +``` +/opt/inference-harness/litellm_config.yaml (.116) +/opt/inference-harness/docker-compose.yml (.116) +/etc/nginx/nginx.conf (.116 container) +/opt/search-stack/*/docker-compose.yaml (.7) +/root/.pi/agent/extensions/config.yaml (.24) +/etc/systemd/system/ornith-server.service (.15) +/etc/systemd/system/llama-server.service (.8, .110) +``` + +Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...` + +## Security-Specific Updates + +| Check | Command | Action | +|-------|---------|--------| +| CVE count | `apt list --upgradable 2>/dev/null \| grep -i security \| wc -l` | Report in update summary | +| Kernel vulns | `uname -r` vs latest available | Flag if >2 versions behind | +| Docker CVEs | `docker scout quickview` or `trivy image` | Flag critical CVEs | +| SSL certs | `openssl s_client -connect chat.sysloggh.net:443 /dev/null \| openssl x509 -noout -dates` | Alert if <30 days | + +## Success Criteria + +- [ ] All 5 PVE nodes updated, no reboot-loop +- [ ] All CTs running post-update +- [ ] All 22 Docker containers healthy +- [ ] LiteLLM inference passing (syslog-auto test) +- [ ] Zulip server + all 3 agents connected +- [ ] GPU fleet at full capacity (3/3) +- [ ] Zero security CVEs remaining +- [ ] <10 min total downtime per service + +## Report + +After completion, send Zulip DM: +``` +📋 Infrastructure Update — YYYY-MM-DD + +Updated: 5 PVE nodes, 12 CTs, 22 containers +Security fixes: N CVEs patched +Downtime: +Failures: none /
+Configs backed up: /tmp/infra-update-backup-YYYYMMDD/ +``` diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index bffd6be..8a696f9 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -1,6 +1,11 @@ --- kind: responsibility name: litellm-self-heal +status: manual-only +note: > + Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04). + This contract is now manual-only — triggers require explicit user request. + Consider reimplementing as a standalone cron job or prose contract. description: > Standing responsibility that monitors LiteLLM health and proactively fixes common issues. Reports every action via Zulip DM and RA-H OS diff --git a/pi-approval-architecture.prose.md b/pi-approval-architecture.prose.md index 2097ea2..c157845 100644 --- a/pi-approval-architecture.prose.md +++ b/pi-approval-architecture.prose.md @@ -1,5 +1,5 @@ --- -kind: pattern +kind: architecture name: pi-approval-architecture description: > Documents pi's approval model vs Hermes, what commands are available, diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md index f18f9e1..23026b6 100644 --- a/pm2-self-heal.prose.md +++ b/pm2-self-heal.prose.md @@ -10,10 +10,13 @@ description: > ## Maintains -- abiba-zulip: { status: "online", uptime: string, restarts: number } - abiba-telegram: { status: "online", uptime: string, restarts: number } +- gpu-watchdog: { status: "online", uptime: string, restarts: number } +- gitea-runner: { status: "online", uptime: string, restarts: number } - last_check: timestamp +> **Note (2026-07-04):** `abiba-zulip` removed — Zulip extension decommissioned. + ## Continuity - Self-driven: check every 300 seconds (5 min) diff --git a/proxmox-monitor.prose.md b/proxmox-monitor.prose.md index ceee5c1..74a7b91 100644 --- a/proxmox-monitor.prose.md +++ b/proxmox-monitor.prose.md @@ -7,7 +7,7 @@ description: > exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve - the layerdb). Dashboards exposed at http://192.168.68.116/grafana/. + the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx). agent: abiba --- @@ -92,7 +92,7 @@ agent: abiba ## Operations ### view-dashboards -Open `http://192.168.68.116/grafana/` → "Syslog Fleet" folder +Open `http://192.168.68.116:3001/` → "Syslog Fleet" folder ### add-dashboard Edit `build-dashboards.py`, run it, `docker restart harness-grafana` diff --git a/runs/20260704-005804-51f9a9/receipts/disk-gc-threat-response.jsonl b/runs/20260704-005804-51f9a9/receipts/disk-gc-threat-response.jsonl new file mode 100644 index 0000000..03362b2 --- /dev/null +++ b/runs/20260704-005804-51f9a9/receipts/disk-gc-threat-response.jsonl @@ -0,0 +1 @@ +{"node":"disk-gc-threat-response","status":"rendered","fingerprints":{"@atomic":"fleet-18-green-0-amber-0-red"},"semantic_diff":"Initial fleet baseline scan: 18 hosts scanned, all GREEN. Previously resolved CT 105 kagentz from 87%→27%. CT 118 jitsi offline.","cost":{},"prev":null,"timestamp":"2026-07-04T00:58:30Z"} diff --git a/runs/20260704-005804-51f9a9/vm.log.md b/runs/20260704-005804-51f9a9/vm.log.md new file mode 100644 index 0000000..6b83e7f --- /dev/null +++ b/runs/20260704-005804-51f9a9/vm.log.md @@ -0,0 +1,26 @@ +# run:20260704-005804-51f9a9 disk-gc-threat-response +root: disk-gc-threat-response + +1→ [scan] fleet-wide disk scan started +2→ CT 100 abiba (amdpve): 12G/59G 21% GREEN +2→ CT 101 llm-gpu (acerpve): 144G/255G 60% GREEN +2→ CT 102 adguard (acerpve): 5.9G/32G 20% GREEN +2→ CT 103 ocu-llm (ocupve): 48G/186G 27% GREEN +2→ CT 104 authentik (minipve): 4.6G/9.8G 50% GREEN +2→ CT 105 kagentz (amdpve): 16G/59G 27% GREEN (resolved: was 87%, pruned 35.67G) +2→ CT 106 ra-h-os (storepve): 2.1G/20G 11% GREEN +2→ CT 107 pbs (storepve): 1.1G/40G 3% GREEN +2→ CT 108 media (storepve): 19G/196G 10% GREEN +2→ CT 109 docker-vm (storepve): 25G/158G 17% GREEN +2→ CT 110 gitea (minipve): 5.6G/40G 16% GREEN +2→ CT 111 tdunna (amdpve): 16G/63G 26% GREEN +2→ CT 112 tanko (amdpve): 17G/49G 36% GREEN +2→ CT 113 baggy (amdpve): 7.0G/52G 15% GREEN +2→ CT 114 mumuni (minipve): 29G/59G 51% GREEN +2→ CT 115 scottdenya (amdpve): 14G/59G 25% GREEN +2→ CT 116 syslog-api (minipve): 16G/40G 42% GREEN +2→ CT 117 zulip (storepve): 6.1G/60G 11% GREEN +2→ CT 118 jitsi (minipve): OFFLINE — no disk scan +3→ [summary] Fleet: 19 hosts, 18 scanned, 18 GREEN, 0 AMBER, 0 RED, 0 CRITICAL, 1 OFFLINE +3→ [summary] Previously resolved: CT 105 kagentz 87%→27% (35.67GB reclaimed) +---end 2026-07-04T00:58:04Z diff --git a/runs/20260704-005804-51f9a9/world-model/disk-gc-threat-response/disk-health.md b/runs/20260704-005804-51f9a9/world-model/disk-gc-threat-response/disk-health.md new file mode 100644 index 0000000..4d5db85 --- /dev/null +++ b/runs/20260704-005804-51f9a9/world-model/disk-gc-threat-response/disk-health.md @@ -0,0 +1,52 @@ +# Fleet Disk Health — 2026-07-04 00:58 UTC + +## Summary + +| Metric | Count | +|--------|-------| +| Total hosts | 19 | +| Scanned | 18 | +| GREEN (<75%) | 18 | +| AMBER (75-84%) | 0 | +| RED (85-94%) | 0 | +| CRITICAL (≥95%) | 0 | +| OFFLINE | 1 (jitsi CT 118) | + +## Per-Host Status + +| CT | Name | Node | Used | Total | % | Status | +|----|------|------|------|-------|---|--------| +| 100 | abiba | amdpve | 12G | 59G | 21% | 🟢 GREEN | +| 101 | llm-gpu | acerpve | 144G | 255G | 60% | 🟢 GREEN | +| 102 | adguard | acerpve | 5.9G | 32G | 20% | 🟢 GREEN | +| 103 | ocu-llm | ocupve | 48G | 186G | 27% | 🟢 GREEN | +| 104 | authentik | minipve | 4.6G | 9.8G | 50% | 🟢 GREEN | +| 105 | kagentz | amdpve | 16G | 59G | 27% | 🟢 GREEN | +| 106 | ra-h-os | storepve | 2.1G | 20G | 11% | 🟢 GREEN | +| 107 | pbs | storepve | 1.1G | 40G | 3% | 🟢 GREEN | +| 108 | media | storepve | 19G | 196G | 10% | 🟢 GREEN | +| 109 | docker-vm | storepve | 25G | 158G | 17% | 🟢 GREEN | +| 110 | gitea | minipve | 5.6G | 40G | 16% | 🟢 GREEN | +| 111 | tdunna | amdpve | 16G | 63G | 26% | 🟢 GREEN | +| 112 | tanko | amdpve | 17G | 49G | 36% | 🟢 GREEN | +| 113 | baggy | amdpve | 7.0G | 52G | 15% | 🟢 GREEN | +| 114 | mumuni | minipve | 29G | 59G | 51% | 🟢 GREEN | +| 115 | scottdenya | amdpve | 14G | 59G | 25% | 🟢 GREEN | +| 116 | syslog-api | minipve | 16G | 40G | 42% | 🟢 GREEN | +| 117 | zulip | storepve | 6.1G | 60G | 11% | 🟢 GREEN | +| 118 | jitsi | minipve | — | — | — | ⚫ OFFLINE | + +## Docker Hosts - GC Status + +| Host | Docker present | Last GC | Status | +|------|---------------|---------|--------| +| kagentz (CT 105) | Yes | 2026-07-04 (just now) | 35.67GB reclaimed, 27% | +| syslog-api (CT 116) | Yes | Never | 42%, no action needed | +| docker-vm (CT 109) | Yes | Unknown | 17%, no action needed | +| abiba (CT 100) | Yes | Never | 21%, /var/lib/docker=2.8G | + +## Notes +- CT 118 (jitsi) is not running — no disk scan possible. May need `qm start 118` if service is needed. +- CT 114 (mumuni) at 51% — closest to AMBER threshold (75%). Agent memory accumulation likely. +- CT 105 (kagentz) resolved from 87% RED → 27% GREEN. Root cause: docker image bloat. Prevented: GC contract in place. +- All Docker hosts are healthy. Next automated scan in 6 hours. diff --git a/zulip-adapter-lessons.prose.md b/zulip-adapter-lessons.prose.md index f237d40..1e0a874 100644 --- a/zulip-adapter-lessons.prose.md +++ b/zulip-adapter-lessons.prose.md @@ -1,6 +1,10 @@ --- kind: pattern name: zulip-adapter-lessons +status: historical +note: > + pi Zulip extension retired 2026-07-04. These lessons remain relevant for + Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge. description: > Lessons learned from building and debugging the pi Zulip extension and Hermes Zulip plugin. Documents failure modes, fixes, and patterns that diff --git a/zulip-mention-reliability.prose.md b/zulip-mention-reliability.prose.md index 9b28ef7..8e20aea 100644 --- a/zulip-mention-reliability.prose.md +++ b/zulip-mention-reliability.prose.md @@ -1,13 +1,18 @@ --- kind: responsibility name: zulip-mention-reliability +status: retired description: > - Monitors whether Abiba Bot is correctly detecting and responding to - @mentions in Zulip stream topics. Tests periodically, diagnoses - failures, and attempts fixes. Same self-improving pattern as - litellm-self-heal. + RETIRED 2026-07-04 — Zulip extension decommissioned. Abiba now operates + exclusively through Telegram (@AbibaBot). This contract is kept for + historical reference only. --- +> **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and +> PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability +> monitoring now happens through Telegram. Hermes agents (Tanko, Mumuni) and +> Agent Zero (kagentz) continue to use Zulip. + ## Maintains - mention_response_rate: number — % of @mentions that get a response (target: >95%) diff --git a/zulip-self-heal.prose.md b/zulip-self-heal.prose.md new file mode 100644 index 0000000..9ee024c --- /dev/null +++ b/zulip-self-heal.prose.md @@ -0,0 +1,92 @@ +--- +kind: responsibility +name: zulip-self-heal +status: retired +description: > + RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned). + Self-healing for Zulip infrastructure continues through Hermes agents. + Abiba no longer monitors or manages Zulip. +agent: abiba +triggers: + - none (retired) +--- + +> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code +> (`performHealthCheck()`). That code has been removed. Zulip self-healing for +> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring. + +## Maintains + +- zulip-server: { status: "healthy" | "degraded" | "down" } +- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state } +- zulip-config: { syntax_valid: bool, proxy_configured: bool } +- last_action: { timestamp, action, result } + +## Detection Rules + +### Rule 1: Zulip Server Down (502/000) +- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000 +- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status +- **Fix**: `docker restart zulip-zulip-1` if container crashed +- **Verify**: Re-test server_settings, wait for "healthy" + +### Rule 2: Zulip Config Corruption (400/500 non-proxy) +- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs +- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log` +- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed +- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s + +### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError) +- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs +- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10") +- **Fix**: Add `[loadbalancer] ips = ` to `/etc/zulip/zulip.conf` + restart +- **Verify**: Re-test via NetBird URL + +### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID) +- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID +- **Fix (pi)**: `pm2 restart abiba-zulip` +- **Fix (Hermes)**: `hermes gateway restart` on agent host +- **Verify**: Check health endpoint for `queue_healthy: true` + +### Rule 5: Agent Adapter Crashed (no heartbeat >10min) +- **Detect**: No heartbeat in agent log for >600s +- **Fix**: Restart gateway on affected host +- **Verify**: Check for new heartbeat in logs within 60s + +### Rule 6: NetBird Tunnel Down +- **Detect**: Server returns 502 with NetBird HTML in response +- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up +- **Fix**: Alert user — NetBird outage requires manual intervention +- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net" + +## Hosts & Credentials + +| Host | IP | User | Service | +|------|----|------|---------| +| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` | +| Abiba (pi) | localhost | root | PM2: `abiba-zulip` | +| Mumuni | 192.168.68.123 | root | `hermes gateway restart` | +| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` | + +## Debounce + +All restart actions must debounce: minimum 300s between restarts per agent. +Track via `/tmp/zulip-heal-debounce-` (unix timestamp of last restart). + +## Reporting + +Every cycle produces a knowledge graph node: +- Title: `[LEARN] zulip-self-heal: ` +- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" } +- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed " +- Issues escalated → DM: "⚠️ Zulip Self-Heal: needs attention" + +## Configuration + +Zulip server config at 192.168.68.19: +- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust +- `/home/zulip/deployments//zproject/prod_settings.py` — Django settings +- Fixed values: + - `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]` + - `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")` + - `LOAD_BALANCER_IPS = ["192.168.68.10"]`