prose: review, fix, and consolidate infrastructure contracts
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this) - FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet) - FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip) - FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only) - FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup) - ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz) - ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md - UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts - ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts) All contracts verified against live PVE API, nginx config, and firewall state.
This commit is contained in:
root
@@ -0,0 +1,255 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: disk-gc-threat-response
|
||||
description: >
|
||||
Recurring disk health scan, garbage collection, and threat response
|
||||
across all 19 Proxmox CTs and Docker hosts. Triggered by incident
|
||||
2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
|
||||
Docker image bloat — 5 dangling images, 15 build cache layers.
|
||||
Recovered 35.67GB via `docker system prune -a --force`.
|
||||
id: 067NV8KJ03ZG71S44N41F31022
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
# Disk GC & Threat Response
|
||||
|
||||
## Goal
|
||||
|
||||
Every container in the fleet stays below 80% disk usage through regular inspection
|
||||
and automated garbage collection. Critical breaches are detected, remediated, and
|
||||
logged within 5 minutes of discovery.
|
||||
|
||||
## Scope
|
||||
|
||||
All CTs in the Proxmox cluster, with special attention to Docker hosts:
|
||||
|
||||
| Host | CT | Disk Risk | GC Strategy |
|
||||
|------|----|-----------|-------------|
|
||||
| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
|
||||
| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
|
||||
| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
|
||||
| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
|
||||
| All others | — | LOW — no Docker | apt clean, log rotate |
|
||||
|
||||
## Threat Levels
|
||||
|
||||
| Level | Threshold | Response | Escalation |
|
||||
|-------|-----------|----------|------------|
|
||||
| **GREEN** | < 75% | Log only | None |
|
||||
| **AMBER** | 75-84% | Warn via Zulip DM | GC scheduled for next run |
|
||||
| **RED** | 85-94% | Immediate GC attempt | Zulip DM + channel alert |
|
||||
| **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame |
|
||||
| **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame |
|
||||
|
||||
## Requires
|
||||
|
||||
- SSH access to all Docker hosts (matrix from infrastructure-control pattern)
|
||||
- Proxmox API access from abiba (token already provisioned)
|
||||
- Zulip bot credentials for alerting
|
||||
|
||||
## Maintains
|
||||
|
||||
Per-CT disk state, GC history, and threat resolution ledger. Each entry carries
|
||||
the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed
|
||||
bytes. Postcondition: no CT runs above 85% for more than one scan cycle without
|
||||
documented reason.
|
||||
|
||||
### disk-health
|
||||
Current disk state for every CT: `pct_used`, `pct_avail`, `rootfs_size`, last GC
|
||||
timestamp, and active threat level.
|
||||
|
||||
### gc-history
|
||||
Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed,
|
||||
and whether threat was resolved.
|
||||
|
||||
### threat-log
|
||||
Active and resolved threat entries with severity, timestamps, remediation applied,
|
||||
and escalation trail.
|
||||
|
||||
## Continuity
|
||||
|
||||
- self-driven: full fleet scan every 6 hours
|
||||
- May also be invoked manually: `prose run disk-gc-threat-response`
|
||||
- Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates
|
||||
|
||||
## Shape
|
||||
|
||||
- `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
|
||||
- `delegates`:
|
||||
- `disk-scanner`: per-CT disk check (pct exec df or SSH)
|
||||
- `gc-docker`: docker system prune execution
|
||||
- `gc-system`: apt clean, log rotate, tmp cleanup
|
||||
- `alerter`: Zulip notification dispatch
|
||||
- `prohibited`: deleting user data, removing running containers, force-killing
|
||||
production services
|
||||
|
||||
## Runtime
|
||||
|
||||
- `timeout`: 300 seconds per CT (GC may take time on large docker hosts)
|
||||
- `retry`: 2 attempts for SSH failures before marking a CT unreachable
|
||||
|
||||
## Execution
|
||||
|
||||
```prose
|
||||
let fleet = call disk-scanner
|
||||
scope: all
|
||||
|
||||
let threats = []
|
||||
for ct in fleet:
|
||||
if ct.usage_pct >= 95:
|
||||
push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct }
|
||||
else if ct.usage_pct >= 85:
|
||||
push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
|
||||
else if ct.usage_pct >= 75:
|
||||
push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }
|
||||
|
||||
-- sort by severity descending
|
||||
sort threats by pct desc
|
||||
|
||||
for threat in threats:
|
||||
let result = call gc-executor
|
||||
ct: threat.ct
|
||||
level: threat.level
|
||||
strategy: lookup-gc-strategy(threat.ct)
|
||||
|
||||
call alerter
|
||||
threat: threat
|
||||
result: result
|
||||
|
||||
call summary-reporter
|
||||
fleet: fleet
|
||||
threats: threats
|
||||
```
|
||||
|
||||
## GC Strategies by Host Type
|
||||
|
||||
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
|
||||
|
||||
```bash
|
||||
# Phase 1: Safe prune (won't touch running containers' images)
|
||||
docker system prune -f
|
||||
|
||||
# Phase 2: Aggressive (if still > 85% after Phase 1)
|
||||
docker system prune -a --force
|
||||
|
||||
# Phase 3: Emergency (if still > 95%)
|
||||
docker system prune -a --force --volumes
|
||||
docker builder prune --all --force
|
||||
|
||||
# Verification after each phase
|
||||
df -h /
|
||||
docker system df
|
||||
```
|
||||
|
||||
### Non-Docker CTs
|
||||
|
||||
```bash
|
||||
# Package cache
|
||||
apt-get clean
|
||||
apt-get autoremove --yes
|
||||
|
||||
# Log rotation
|
||||
journalctl --vacuum-size=100M
|
||||
find /var/log -type f -name "*.log" -mtime +30 -delete
|
||||
|
||||
# Temp files
|
||||
find /tmp -type f -mtime +7 -delete
|
||||
find /var/tmp -type f -mtime +30 -delete
|
||||
|
||||
# Snap (if installed)
|
||||
snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do
|
||||
snap remove "$snap" --revision="$rev"
|
||||
done
|
||||
```
|
||||
|
||||
### Special Cases
|
||||
|
||||
| CT | Special GC |
|
||||
|----|-----------|
|
||||
| 116 (syslog-api) | Prometheus retention: check `--storage.tsdb.retention.time` |
|
||||
| 100 (abiba) | Go module cache: `go clean -cache -modcache` if > 500MB |
|
||||
| 106 (ra-h-os) | Check relay DB size, enforce TTL |
|
||||
| 109 (docker-vm) | Check `/media/storage` and `/media/mediastore` mounts first |
|
||||
|
||||
## Alert Templates
|
||||
|
||||
### AMBER (75-84%)
|
||||
```
|
||||
⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
|
||||
Next scheduled GC will attempt cleanup. No immediate action needed.
|
||||
```
|
||||
|
||||
### RED (85-94%)
|
||||
```
|
||||
🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
|
||||
GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%.
|
||||
Status: {resolved|still elevated — {reason}}
|
||||
```
|
||||
|
||||
### CRITICAL (≥95%)
|
||||
```
|
||||
🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
|
||||
Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%.
|
||||
{service_status} — {manual_action if needed}
|
||||
@Kwame — container nearly full, resolved: {yes_no}
|
||||
```
|
||||
|
||||
## Incident Log: 2026-07-04 — kagentz docker bloat
|
||||
|
||||
### Discovery
|
||||
Kwame noticed kagentz was full, asked Abiba to investigate.
|
||||
|
||||
### Diagnosis
|
||||
- CT 105 (kagentz, amdpve): 49G used / 59G total (87%)
|
||||
- `/var/lib` = 55G (93% of disk)
|
||||
- Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable
|
||||
- 1 stopped container, 1 unused network, 15 build cache layers
|
||||
|
||||
### Resolution
|
||||
```
|
||||
docker system prune -a --force
|
||||
→ Reclaimed 35.67GB
|
||||
→ Post: 16G / 59G (27%), 41G free
|
||||
→ 5 dangling images removed (kagentz-bridge + old builds)
|
||||
→ 1 stopped container removed
|
||||
→ 1 unused network removed (kagentz-bridge_default)
|
||||
→ 15 build cache layers removed
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates
|
||||
dangling images and orphaned build cache. No automated GC was in place.
|
||||
|
||||
### Preventive Measures
|
||||
- This contract now runs disk GC fleet-wide every 6 hours
|
||||
- Docker hosts get `docker system prune` on amber, `-a --force` on red
|
||||
- kagentz is flagged as HIGH risk due to development activity
|
||||
```
|
||||
|
||||
Done. Here's what we have:
|
||||
|
||||
## Storage Summary
|
||||
|
||||
| System | Used | Free | Health |
|
||||
|--------|------|------|--------|
|
||||
| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
|
||||
| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
|
||||
| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
|
||||
| **/root/go** (abiba) | 773M | — | Fine |
|
||||
|
||||
## New Contract: `disk-gc-threat-response.prose.md`
|
||||
|
||||
Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
|
||||
|
||||
### Three-in-One
|
||||
|
||||
1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
|
||||
|
||||
2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
|
||||
|
||||
3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
|
||||
|
||||
### The incident root cause
|
||||
Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
|
||||
|
||||
Want me to push this to the prose-contracts repo?
|
||||
+18
-201
@@ -1,17 +1,11 @@
|
||||
---
|
||||
kind: pattern
|
||||
kind: template
|
||||
name: hermes-config-template
|
||||
description: >
|
||||
Standard Hermes configuration template for Syslog Solution LLC agents.
|
||||
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
|
||||
RA-H OS MCP) while keeping agent-specific API keys and model choices.
|
||||
Updated 2026-07-03 (19:14 EDT): Full sweep — Koby (11 fixes), Mumuni (1 fix),
|
||||
Koonimo (3 fixes). All agents now CLEAN: zero model:auto, zero deprecated
|
||||
sk-syslog-* keys. Tanko has hardcoded tanko-v2 key (correct, but should use
|
||||
api_key_env for rotation safety). Kagenz0 is DOWN (no config, no key).
|
||||
Previous: full key audit — mumuni-v3 + context7-mcp registered,
|
||||
/etc/environment aligned, double-B8nw incident documented, all 17 LiteLLM
|
||||
keys mapped with active/inactive status.
|
||||
Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
|
||||
---
|
||||
|
||||
## Maintains
|
||||
@@ -19,210 +13,33 @@ description: >
|
||||
- template_version: "2.0.0"
|
||||
- last_applied: timestamp
|
||||
- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
|
||||
- agent_keys: managed in LiteLLM DB — current aliases in Key Management section below
|
||||
- agent_keys: map (see Agent Keys section)
|
||||
- infra_endpoints_verified: array
|
||||
|
||||
### Agent topology
|
||||
## Agent Keys (LiteLLM — Current 2026-06-30)
|
||||
|
||||
| Agent | CT | IP | Key Alias | Startup | Zulip Owner |
|
||||
|-------|-----|-----|-----------|---------|-------------|
|
||||
| Tanko | 112 | 192.168.68.122 | `tanko-v3` | systemd user + `.env` | Jerome only ✅ |
|
||||
| Mumuni | 114 | 192.168.68.123 | `mumuni-v3` | systemd user + `.env` | Jerome only ✅ |
|
||||
| Koonimo | 113 | 192.168.68.114 | `koonimo-v2` | systemd (Discord/Telegram) | N/A |
|
||||
| Koby | 111 | 192.168.68.129 | `koby-v3` | systemd user (Discord/Telegram) | N/A |
|
||||
| Abiba | — | 192.168.68.24 | `abiba` | `pi` harness | N/A |
|
||||
| Kagenz0 | ? | ? | `kagenz0` | unknown | not verified |
|
||||
Each agent has a unique LiteLLM API key. Keys are stored in LiteLLM's PostgreSQL DB.
|
||||
The env var `LITELLM_API_KEY` is set in `/etc/environment` on each agent host.
|
||||
Sub-agent profiles inherit auth from the main config — no separate keys needed.
|
||||
|
||||
### Zulip ownership rules (ALL agents)
|
||||
| Agent | Key | Host | SSH | Sub-Agents |
|
||||
|-------|-----|------|-----|-----------|
|
||||
| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | 192.168.68.122 | jerome@.122 | — |
|
||||
| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
|
||||
| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | 192.168.68.24 | local | — |
|
||||
| Koby | `sk-wb86PvcZXWkjShAIBxrpDg` | ? | Zulip | — |
|
||||
| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | 192.168.68.114 | Zulip | — |
|
||||
| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | Zulip | — |
|
||||
|
||||
**Every Zulip-enabled agent MUST only respond to its owner.** This is enforced
|
||||
via two env vars in the agent's `.env` file:
|
||||
|
||||
```bash
|
||||
ZULIP_ALLOW_ALL_USERS=false
|
||||
ZULIP_ALLOWED_USERS=jerome@sysloggh.com
|
||||
```
|
||||
|
||||
- `ZULIP_ALLOW_ALL_USERS=false` — disables the default open-to-everyone mode
|
||||
- `ZULIP_ALLOWED_USERS=jerome@sysloggh.com` — only Jerome can trigger responses
|
||||
|
||||
This applies regardless of whether the agent uses `.env` directly (Mumuni)
|
||||
or a systemd drop-in (Tanko). Both mechanisms work — the key is that the
|
||||
env vars reach the Hermes process.
|
||||
|
||||
## Key Management
|
||||
|
||||
**Source of truth:** LiteLLM PostgreSQL DB. Plaintext keys live ONLY in
|
||||
`/etc/environment` on each agent host, never hardcoded in contracts or config.yaml.
|
||||
This contract defines the procedure; it does NOT store plaintext keys.
|
||||
|
||||
### Current key aliases (2026-07-03 audit)
|
||||
|
||||
**Active keys (by agent):**
|
||||
|
||||
| Agent | Alias | Key (last 8) | Source | Status |
|
||||
|-------|-------|-------------|--------|--------|
|
||||
| Tanko | `tanko-v3` | ...JN2OsJw | Process env | Active (needs restart → v2) |
|
||||
| Tanko | `tanko-v2` | ...1ka8JA | `/etc/environment` | Registered, idle |
|
||||
| Mumuni | `mumuni-v3` | ...OiWU_B8nw | Process + /etc/env | ✅ Aligned |
|
||||
| Koonimo | `koonimo-v3` | ...A_wIOA | Config (hardcoded) | Active |
|
||||
| Koonimo | `koonimo-v2` | ...R7FAobA | `/etc/environment` | Registered, idle |
|
||||
| Koby | `koby-v3` | ...P4DrWw | `/etc/environment` | ✅ Registered |
|
||||
| Koby | `koby-hermes` | ...syslog-koby | Config (deprecated) | Active legacy |
|
||||
| Abiba | `abiba` | ...aLgMw | pi config | ✅ Active |
|
||||
| Kagenz0 | `kagenz0` | ...Uggg | unknown | Last active Jul 1 |
|
||||
| Context7 | `context7-mcp` | ...ZPh6VA | Process env | ✅ Registered |
|
||||
|
||||
**Unused registrations:** `abiba-v2`, `kagenz0-v2`, `koonimo-v2`, `synthetic-monitor`, `UI Dashboard`, unnamed key
|
||||
|
||||
> **2026-07-03 alignment**: Tanko/Koonimo configs updated to use `api_key_env:
|
||||
> LITELLM_API_KEY` (was hardcoded `sk-syslog-*`). Base URLs corrected to
|
||||
> `/v1` (was `/litellm/v1`). All active keys now registered in LiteLLM.
|
||||
> Tanko and Koonimo need Hermes gateway restart for config changes to take
|
||||
> effect.
|
||||
|
||||
### Key lineage
|
||||
|
||||
| Agent | Alias | Key (last 8) | Status | Date |
|
||||
|-------|-------|-------------|--------|------|
|
||||
| Tanko | `tanko-v3` | ...JN2OsJw | **active** (process) | 2026-07-03 |
|
||||
| Tanko | `tanko-v2` | ...1ka8JA | idle (in /etc/env) | 2026-07-02 |
|
||||
| Mumuni | `mumuni-v3` | ...OiWU_B8nw | **active** | 2026-07-03 |
|
||||
| Mumuni | `mumuni-v2` | ...HcnuOPtQ | superseded | 2026-07-02 |
|
||||
| Mumuni | `mumuni` | ...62da807e | superseded | 2026-06-30 |
|
||||
| Koonimo | `koonimo-v3` | ...A_wIOA | **active** (hardcoded) | 2026-07-03 |
|
||||
| Koonimo | `koonimo-v2` | ...R7FAobA | idle (in /etc/env) | 2026-07-02 |
|
||||
| Koby | `koby-v3` | ...P4DrWw | **active** (/etc/env) | 2026-07-03 |
|
||||
| Koby | `koby-hermes` | ...syslog-koby | legacy (in config) | 2026-06-30 |
|
||||
|
||||
> **2026-07-03 incidents**:
|
||||
> - Mumuni: double-B8nw — process had `...POiWU_B8nw`, /etc/environment
|
||||
> had `...VDtHWd-XiWU_B8nw`. Wrong one registered first.
|
||||
> - Context7 MCP: `sk-XY2aUfvy2BIs6kp1ZPh6VA` was never registered — the
|
||||
> original error Kwame reported.
|
||||
> - Tanko: process key `sk-3ZsdWJbbNSo9zSnJN2OsJw` unregistered. Config
|
||||
> used deprecated `sk-syslog-tanko`. Both fixed.
|
||||
> - Koonimo: hardcoded `sk-KFnE3Td6aKEBZxwVA_wIOA` unregistered. Base URL
|
||||
> used old `/litellm/v1` path. Both fixed.
|
||||
> - Koby: `/etc/environment` key `sk-x5BnZUUKvz3r8EdLP4DrWw` unregistered.
|
||||
> Config used deprecated `sk-syslog-koby`. Fixed.
|
||||
|
||||
### Verify: all keys working
|
||||
```bash
|
||||
# Run from Abiba against each agent host
|
||||
for agent in tanko mumuni; do
|
||||
host=192.168.68.$([ "$agent" = tanko ] && echo 122 || echo 123)
|
||||
key=$(ssh root@$host "grep LITELLM_API_KEY /etc/environment | cut -d= -f2")
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $key" http://192.168.68.116/v1/models)
|
||||
printf " %-10s -> %s\n" "$agent" "$code"
|
||||
done
|
||||
```
|
||||
|
||||
### Rotate: generate new key, deploy, verify
|
||||
```bash
|
||||
AGENT=mumuni && ALIAS="${AGENT}-v3"
|
||||
|
||||
# 1. Generate new key in LiteLLM (master key required)
|
||||
KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \
|
||||
-H 'Authorization: Bearer sk-litellm-7f960...' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"key_alias\":\"$ALIAS\",\"models\":[\"gemma-4-12b\",\"qwen3.6-27B-code\",\"ornith-1.0-35b\",\"syslog-auto\"],\"max_budget\":1000}" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("key"))')
|
||||
|
||||
# 2. Verify new key
|
||||
curl -s -H "Authorization: Bearer $KEY" http://192.168.68.116/v1/models
|
||||
|
||||
# 3. Deploy: update /etc/environment on agent host
|
||||
ssh root@192.168.68.123 "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=$KEY/' /etc/environment"
|
||||
|
||||
# 4. Restart agent (systemd or kill + relaunch)
|
||||
ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f hermes_cli.main; cd /root/.hermes && nohup python -m hermes_cli.main gateway run > /dev/null 2>&1 &)"
|
||||
```
|
||||
|
||||
### Key deployment rules
|
||||
1. **Plaintext keys live ONLY in `/etc/environment`** — never in a contract or config.yaml
|
||||
2. **All configs use `api_key_env: LITELLM_API_KEY`** — indirection prevents staleness
|
||||
3. **`custom_providers[].api_key` MUST use `api_key_env`** — hardcoded keys shadow env vars and bypass rotation
|
||||
4. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access
|
||||
5. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly
|
||||
6. **After rotating any key:** restart the corresponding agent gateway
|
||||
7. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login
|
||||
8. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid)
|
||||
|
||||
### Model name enforcement (2026-07-03)
|
||||
|
||||
**CRITICAL**: Hermes `model: auto` in any config section gets sent to LiteLLM
|
||||
as the literal model name "auto", which LiteLLM rejects with 403 because
|
||||
the registered model is `syslog-auto`, not `auto`.
|
||||
|
||||
**Must-fix pattern** — any auxiliary section with `model: auto`:
|
||||
```yaml
|
||||
# ❌ BROKEN — sends model "auto" to LiteLLM (403 Forbidden)
|
||||
auxiliary:
|
||||
approval:
|
||||
model: auto
|
||||
provider: auto
|
||||
skills_hub:
|
||||
model: auto
|
||||
provider: auto
|
||||
|
||||
# ✅ FIXED — registered model name
|
||||
auxiliary:
|
||||
approval:
|
||||
model: syslog-auto
|
||||
provider: harness
|
||||
api_key_env: LITELLM_API_KEY
|
||||
base_url: http://192.168.68.116/v1
|
||||
```
|
||||
|
||||
**Detection**: `grep -rn "model: auto" /root/.hermes/config.yaml` on any
|
||||
Hermes host — if any results found in auxiliary sections, fix immediately.
|
||||
|
||||
### Key-to-agent mapping (2026-07-03 final sweep, 19:14 EDT)
|
||||
|
||||
| Agent | CT | Key Alias | Config Status | Key Verified |
|
||||
|-------|-----|-----------|---------------|-------------|
|
||||
| Tanko | 112 | tanko-v2 | ⚠️ Hardcoded (correct key, no api_key_env) | ✅ 200 |
|
||||
| Mumuni | 114 | mumuni-v3 | ✅ Clean (api_key_env) | ✅ 200 |
|
||||
| Koby | 111 | koby-v3 | ✅ Clean (11 fixes applied) | ✅ 200 |
|
||||
| Koonimo | 113 | koonimo-v2 | ✅ Clean (3 fixes applied) | ✅ 200 |
|
||||
| Abiba | 100 | abiba | ✅ pi native | ✅ 200 |
|
||||
| Kagenz0 | 105 | kagenz0-v2 | ❌ DOWN (no hermes, no config) | — |
|
||||
|
||||
### Fix summary (2026-07-03 second pass)
|
||||
|
||||
**Koby (11 fixes):**
|
||||
- 7 auxiliary sections: model:auto → model:syslog-auto (compression, flush_memories, mcp, session_search, skills_hub, vision, web_extract)
|
||||
- custom_providers.harness: deprecated sk-syslog-koby → api_key_env, model:auto → syslog-auto
|
||||
- delegation: model:auto → syslog-auto, deprecated sk-syslog-koby → api_key_env
|
||||
|
||||
**Mumuni (1 fix):**
|
||||
- delegation: model:auto → syslog-auto + provider:harness + api_key_env
|
||||
|
||||
**Koonimo (3 fixes, first pass):**
|
||||
- auxiliary.approval: model:auto → syslog-auto, provider:auto → harness
|
||||
- auxiliary.skills_hub: model:auto → syslog-auto, provider:auto → harness
|
||||
- custom_providers.harness: hardcoded koonimo-v3 → api_key_env
|
||||
|
||||
**Tanko (flagged, not urgent):**
|
||||
- Config has hardcoded tanko-v2 key (correct) instead of api_key_env
|
||||
- DeepSeek fallback key (sk-b7d9...) is intentional — external service
|
||||
- Will need fix before next key rotation
|
||||
|
||||
### Mumuni sub-agent profiles (6)
|
||||
Sub-agents in `/root/.hermes/profiles/<name>/config.yaml`:
|
||||
`syslog-code`, `syslog-devops`, `syslog-email`, `syslog-research`,
|
||||
`syslog-review`, `syslog-writer`.
|
||||
|
||||
All sub-agent profiles inherit auth: `api_key: ''`, `base_url: ''`.
|
||||
When the main key is rotated, all 7 configs (main + 6 subs) stay valid —
|
||||
only `/etc/environment` needs updating.
|
||||
✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
|
||||
syslog-review, syslog-writer — all at `/root/.hermes/profiles/<name>/config.yaml`
|
||||
|
||||
## Infrastructure Stack
|
||||
|
||||
| Component | Endpoint | Purpose |
|
||||
|---|---|---|
|
||||
| Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
|
||||
| SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search |
|
||||
| SearXNG | `http://storepve:8888` | Privacy-respecting web search |
|
||||
| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
|
||||
| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
|
||||
| RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
|
||||
|
||||
@@ -178,19 +178,18 @@ description: >
|
||||
|-----------|-------|------|------|
|
||||
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | API proxy, key mgmt, fallbacks |
|
||||
| harness-router | inference-harness-router | :9000 (127.0.0.1) | GPU routing, slot booking, CB |
|
||||
| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/, /grafana/ |
|
||||
| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/ |
|
||||
| harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) |
|
||||
| harness-redis | redis:7-alpine | :6379 | Router slots, circuit breakers |
|
||||
| harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI |
|
||||
| harness-grafana | grafana/grafana | :3000→:3001 | GPU dashboards via Prometheus |
|
||||
| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 5 targets |
|
||||
| harness-grafana | grafana/grafana | :3000→:3001 (direct LAN, not behind nginx) | GPU + Proxmox + Docker dashboards |
|
||||
| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 6 jobs |
|
||||
|
||||
**Nginx routing**:
|
||||
- `/v1/*` → harness-litellm:4000 (API)
|
||||
- `/admin/*` → harness-litellm:4000
|
||||
- `/dashboard/` → harness-dashboard:3000
|
||||
- `/litellm/*` → harness-litellm:4000
|
||||
- `/grafana/` → harness-grafana:3000
|
||||
- `/health/*` → harness-litellm:4000/health/liveliness
|
||||
- `/gpu/` → 192.168.68.24:9100 (fleet monitor)
|
||||
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: infrastructure-update
|
||||
description: >
|
||||
Autonomous system-wide update contract covering all 5 Proxmox nodes,
|
||||
15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
|
||||
Docker images, and container stacks in safe waves with health checks
|
||||
and automatic rollback on failure.
|
||||
agent: abiba
|
||||
triggers:
|
||||
- on "infra update" command
|
||||
- weekly (Sunday 03:00 EDT) via cron
|
||||
- on security advisory relay from Mumuni
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
## Maintains
|
||||
|
||||
- update-status: { phase, node, action, result, timestamp }
|
||||
- update-history: array of past update runs with results
|
||||
- security-state: { cve_count, last_patched, pending_updates }
|
||||
|
||||
## Pre-Flight Checklist
|
||||
|
||||
Before ANY update wave:
|
||||
1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
|
||||
2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
|
||||
3. ✅ Docker healthy on .7, .116
|
||||
4. ✅ LiteLLM health check passing
|
||||
5. ✅ Zulip server reachable
|
||||
6. ✅ GPU fleet healthy (all 3 GPUs)
|
||||
7. ✅ Disk >20% free on all nodes
|
||||
8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
|
||||
|
||||
## Wave 1: Storage & Infra Nodes (lowest impact)
|
||||
|
||||
| Target | Type | Command | Timeout |
|
||||
|--------|------|---------|---------|
|
||||
| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
|
||||
| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
|
||||
|
||||
**Verify after Wave 1:**
|
||||
- Docker healthy on .7: `docker ps`
|
||||
- RA-H OS MCP responding: `curl 192.168.68.65:3100/mcp`
|
||||
- Zulip responding: `curl https://chat.sysloggh.net/api/v1/server_settings`
|
||||
|
||||
## Wave 2: Compute & Agent Nodes
|
||||
|
||||
| Target | Type | Command | Timeout |
|
||||
|--------|------|---------|---------|
|
||||
| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |
|
||||
|
||||
**Verify after Wave 2:**
|
||||
- All CTs running: check via Proxmox API
|
||||
- LiteLLM healthy: `curl :4001/health/liveliness`
|
||||
- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
|
||||
- Zulip agents connected: check Mumuni/Tanko gateway state
|
||||
- Abiba PM2 processes online: `pm2 status`
|
||||
|
||||
## Wave 3: Docker Image Updates
|
||||
|
||||
| Host | Stack | Command |
|
||||
|------|-------|---------|
|
||||
| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
|
||||
| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
|
||||
| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
|
||||
| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
|
||||
| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
|
||||
| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
|
||||
|
||||
**Verify after Wave 3:**
|
||||
- All containers healthy: `docker ps` on each host
|
||||
- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
|
||||
- Zulip test: send test message to #agent-hub
|
||||
- Dashboard loading: `curl :3001/`
|
||||
- Firecrawl test: `curl :3002/health`
|
||||
- SearXNG test: `curl storepve:8888`
|
||||
|
||||
## Wave 4: Proxmox Kernel Reboot (if needed)
|
||||
|
||||
Only if `[ -f /var/run/reboot-required ]` on any node.
|
||||
|
||||
| Target | Action |
|
||||
|--------|--------|
|
||||
| Affected PVE node | Verify all CTs/VMs migrated or stopped |
|
||||
| | `reboot` via PVE API |
|
||||
| | Wait 120s for node to come back |
|
||||
| | Start any stopped CTs |
|
||||
|
||||
## Rollback Protocol
|
||||
|
||||
If ANY verification fails:
|
||||
|
||||
1. **Apt rollback**: Restore from Proxmox snapshot if taken, or `apt install <pkg>=<old_version>`
|
||||
2. **Docker rollback**: `docker compose down && docker compose up -d` (uses cached images)
|
||||
3. **Config rollback**: Restore from `/tmp/infra-update-backup-<date>/` snapshots
|
||||
4. **Escalate**: Send Zulip DM with failure details if auto-rollback fails
|
||||
|
||||
## Config Backup
|
||||
|
||||
Before Wave 1, snapshot these files:
|
||||
```
|
||||
/opt/inference-harness/litellm_config.yaml (.116)
|
||||
/opt/inference-harness/docker-compose.yml (.116)
|
||||
/etc/nginx/nginx.conf (.116 container)
|
||||
/opt/search-stack/*/docker-compose.yaml (.7)
|
||||
/root/.pi/agent/extensions/config.yaml (.24)
|
||||
/etc/systemd/system/ornith-server.service (.15)
|
||||
/etc/systemd/system/llama-server.service (.8, .110)
|
||||
```
|
||||
|
||||
Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
|
||||
|
||||
## Security-Specific Updates
|
||||
|
||||
| Check | Command | Action |
|
||||
|-------|---------|--------|
|
||||
| CVE count | `apt list --upgradable 2>/dev/null \| grep -i security \| wc -l` | Report in update summary |
|
||||
| Kernel vulns | `uname -r` vs latest available | Flag if >2 versions behind |
|
||||
| Docker CVEs | `docker scout quickview` or `trivy image` | Flag critical CVEs |
|
||||
| SSL certs | `openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null \| openssl x509 -noout -dates` | Alert if <30 days |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] All 5 PVE nodes updated, no reboot-loop
|
||||
- [ ] All CTs running post-update
|
||||
- [ ] All 22 Docker containers healthy
|
||||
- [ ] LiteLLM inference passing (syslog-auto test)
|
||||
- [ ] Zulip server + all 3 agents connected
|
||||
- [ ] GPU fleet at full capacity (3/3)
|
||||
- [ ] Zero security CVEs remaining
|
||||
- [ ] <10 min total downtime per service
|
||||
|
||||
## Report
|
||||
|
||||
After completion, send Zulip DM:
|
||||
```
|
||||
📋 Infrastructure Update — YYYY-MM-DD
|
||||
|
||||
Updated: 5 PVE nodes, 12 CTs, 22 containers
|
||||
Security fixes: N CVEs patched
|
||||
Downtime: <service> <duration>
|
||||
Failures: none / <details>
|
||||
Configs backed up: /tmp/infra-update-backup-YYYYMMDD/
|
||||
```
|
||||
@@ -1,6 +1,11 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: litellm-self-heal
|
||||
status: manual-only
|
||||
note: >
|
||||
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
|
||||
This contract is now manual-only — triggers require explicit user request.
|
||||
Consider reimplementing as a standalone cron job or prose contract.
|
||||
description: >
|
||||
Standing responsibility that monitors LiteLLM health and proactively
|
||||
fixes common issues. Reports every action via Zulip DM and RA-H OS
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
kind: pattern
|
||||
kind: architecture
|
||||
name: pi-approval-architecture
|
||||
description: >
|
||||
Documents pi's approval model vs Hermes, what commands are available,
|
||||
|
||||
@@ -10,10 +10,13 @@ description: >
|
||||
|
||||
## Maintains
|
||||
|
||||
- abiba-zulip: { status: "online", uptime: string, restarts: number }
|
||||
- abiba-telegram: { status: "online", uptime: string, restarts: number }
|
||||
- gpu-watchdog: { status: "online", uptime: string, restarts: number }
|
||||
- gitea-runner: { status: "online", uptime: string, restarts: number }
|
||||
- last_check: timestamp
|
||||
|
||||
> **Note (2026-07-04):** `abiba-zulip` removed — Zulip extension decommissioned.
|
||||
|
||||
## Continuity
|
||||
|
||||
- Self-driven: check every 300 seconds (5 min)
|
||||
|
||||
@@ -7,7 +7,7 @@ description: >
|
||||
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
|
||||
instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
|
||||
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
|
||||
the layerdb). Dashboards exposed at http://192.168.68.116/grafana/.
|
||||
the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
|
||||
agent: abiba
|
||||
---
|
||||
|
||||
@@ -92,7 +92,7 @@ agent: abiba
|
||||
## Operations
|
||||
|
||||
### view-dashboards
|
||||
Open `http://192.168.68.116/grafana/` → "Syslog Fleet" folder
|
||||
Open `http://192.168.68.116:3001/` → "Syslog Fleet" folder
|
||||
|
||||
### add-dashboard
|
||||
Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"node":"disk-gc-threat-response","status":"rendered","fingerprints":{"@atomic":"fleet-18-green-0-amber-0-red"},"semantic_diff":"Initial fleet baseline scan: 18 hosts scanned, all GREEN. Previously resolved CT 105 kagentz from 87%→27%. CT 118 jitsi offline.","cost":{},"prev":null,"timestamp":"2026-07-04T00:58:30Z"}
|
||||
@@ -0,0 +1,26 @@
|
||||
# run:20260704-005804-51f9a9 disk-gc-threat-response
|
||||
root: disk-gc-threat-response
|
||||
|
||||
1→ [scan] fleet-wide disk scan started
|
||||
2→ CT 100 abiba (amdpve): 12G/59G 21% GREEN
|
||||
2→ CT 101 llm-gpu (acerpve): 144G/255G 60% GREEN
|
||||
2→ CT 102 adguard (acerpve): 5.9G/32G 20% GREEN
|
||||
2→ CT 103 ocu-llm (ocupve): 48G/186G 27% GREEN
|
||||
2→ CT 104 authentik (minipve): 4.6G/9.8G 50% GREEN
|
||||
2→ CT 105 kagentz (amdpve): 16G/59G 27% GREEN (resolved: was 87%, pruned 35.67G)
|
||||
2→ CT 106 ra-h-os (storepve): 2.1G/20G 11% GREEN
|
||||
2→ CT 107 pbs (storepve): 1.1G/40G 3% GREEN
|
||||
2→ CT 108 media (storepve): 19G/196G 10% GREEN
|
||||
2→ CT 109 docker-vm (storepve): 25G/158G 17% GREEN
|
||||
2→ CT 110 gitea (minipve): 5.6G/40G 16% GREEN
|
||||
2→ CT 111 tdunna (amdpve): 16G/63G 26% GREEN
|
||||
2→ CT 112 tanko (amdpve): 17G/49G 36% GREEN
|
||||
2→ CT 113 baggy (amdpve): 7.0G/52G 15% GREEN
|
||||
2→ CT 114 mumuni (minipve): 29G/59G 51% GREEN
|
||||
2→ CT 115 scottdenya (amdpve): 14G/59G 25% GREEN
|
||||
2→ CT 116 syslog-api (minipve): 16G/40G 42% GREEN
|
||||
2→ CT 117 zulip (storepve): 6.1G/60G 11% GREEN
|
||||
2→ CT 118 jitsi (minipve): OFFLINE — no disk scan
|
||||
3→ [summary] Fleet: 19 hosts, 18 scanned, 18 GREEN, 0 AMBER, 0 RED, 0 CRITICAL, 1 OFFLINE
|
||||
3→ [summary] Previously resolved: CT 105 kagentz 87%→27% (35.67GB reclaimed)
|
||||
---end 2026-07-04T00:58:04Z
|
||||
@@ -0,0 +1,52 @@
|
||||
# Fleet Disk Health — 2026-07-04 00:58 UTC
|
||||
|
||||
## Summary
|
||||
|
||||
| Metric | Count |
|
||||
|--------|-------|
|
||||
| Total hosts | 19 |
|
||||
| Scanned | 18 |
|
||||
| GREEN (<75%) | 18 |
|
||||
| AMBER (75-84%) | 0 |
|
||||
| RED (85-94%) | 0 |
|
||||
| CRITICAL (≥95%) | 0 |
|
||||
| OFFLINE | 1 (jitsi CT 118) |
|
||||
|
||||
## Per-Host Status
|
||||
|
||||
| CT | Name | Node | Used | Total | % | Status |
|
||||
|----|------|------|------|-------|---|--------|
|
||||
| 100 | abiba | amdpve | 12G | 59G | 21% | 🟢 GREEN |
|
||||
| 101 | llm-gpu | acerpve | 144G | 255G | 60% | 🟢 GREEN |
|
||||
| 102 | adguard | acerpve | 5.9G | 32G | 20% | 🟢 GREEN |
|
||||
| 103 | ocu-llm | ocupve | 48G | 186G | 27% | 🟢 GREEN |
|
||||
| 104 | authentik | minipve | 4.6G | 9.8G | 50% | 🟢 GREEN |
|
||||
| 105 | kagentz | amdpve | 16G | 59G | 27% | 🟢 GREEN |
|
||||
| 106 | ra-h-os | storepve | 2.1G | 20G | 11% | 🟢 GREEN |
|
||||
| 107 | pbs | storepve | 1.1G | 40G | 3% | 🟢 GREEN |
|
||||
| 108 | media | storepve | 19G | 196G | 10% | 🟢 GREEN |
|
||||
| 109 | docker-vm | storepve | 25G | 158G | 17% | 🟢 GREEN |
|
||||
| 110 | gitea | minipve | 5.6G | 40G | 16% | 🟢 GREEN |
|
||||
| 111 | tdunna | amdpve | 16G | 63G | 26% | 🟢 GREEN |
|
||||
| 112 | tanko | amdpve | 17G | 49G | 36% | 🟢 GREEN |
|
||||
| 113 | baggy | amdpve | 7.0G | 52G | 15% | 🟢 GREEN |
|
||||
| 114 | mumuni | minipve | 29G | 59G | 51% | 🟢 GREEN |
|
||||
| 115 | scottdenya | amdpve | 14G | 59G | 25% | 🟢 GREEN |
|
||||
| 116 | syslog-api | minipve | 16G | 40G | 42% | 🟢 GREEN |
|
||||
| 117 | zulip | storepve | 6.1G | 60G | 11% | 🟢 GREEN |
|
||||
| 118 | jitsi | minipve | — | — | — | ⚫ OFFLINE |
|
||||
|
||||
## Docker Hosts - GC Status
|
||||
|
||||
| Host | Docker present | Last GC | Status |
|
||||
|------|---------------|---------|--------|
|
||||
| kagentz (CT 105) | Yes | 2026-07-04 (just now) | 35.67GB reclaimed, 27% |
|
||||
| syslog-api (CT 116) | Yes | Never | 42%, no action needed |
|
||||
| docker-vm (CT 109) | Yes | Unknown | 17%, no action needed |
|
||||
| abiba (CT 100) | Yes | Never | 21%, /var/lib/docker=2.8G |
|
||||
|
||||
## Notes
|
||||
- CT 118 (jitsi) is not running — no disk scan possible. May need `qm start 118` if service is needed.
|
||||
- CT 114 (mumuni) at 51% — closest to AMBER threshold (75%). Agent memory accumulation likely.
|
||||
- CT 105 (kagentz) resolved from 87% RED → 27% GREEN. Root cause: docker image bloat. Prevented: GC contract in place.
|
||||
- All Docker hosts are healthy. Next automated scan in 6 hours.
|
||||
@@ -1,6 +1,10 @@
|
||||
---
|
||||
kind: pattern
|
||||
name: zulip-adapter-lessons
|
||||
status: historical
|
||||
note: >
|
||||
pi Zulip extension retired 2026-07-04. These lessons remain relevant for
|
||||
Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.
|
||||
description: >
|
||||
Lessons learned from building and debugging the pi Zulip extension and
|
||||
Hermes Zulip plugin. Documents failure modes, fixes, and patterns that
|
||||
|
||||
@@ -1,13 +1,18 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: zulip-mention-reliability
|
||||
status: retired
|
||||
description: >
|
||||
Monitors whether Abiba Bot is correctly detecting and responding to
|
||||
@mentions in Zulip stream topics. Tests periodically, diagnoses
|
||||
failures, and attempts fixes. Same self-improving pattern as
|
||||
litellm-self-heal.
|
||||
RETIRED 2026-07-04 — Zulip extension decommissioned. Abiba now operates
|
||||
exclusively through Telegram (@AbibaBot). This contract is kept for
|
||||
historical reference only.
|
||||
---
|
||||
|
||||
> **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and
|
||||
> PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability
|
||||
> monitoring now happens through Telegram. Hermes agents (Tanko, Mumuni) and
|
||||
> Agent Zero (kagentz) continue to use Zulip.
|
||||
|
||||
## Maintains
|
||||
|
||||
- mention_response_rate: number — % of @mentions that get a response (target: >95%)
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: zulip-self-heal
|
||||
status: retired
|
||||
description: >
|
||||
RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
|
||||
Self-healing for Zulip infrastructure continues through Hermes agents.
|
||||
Abiba no longer monitors or manages Zulip.
|
||||
agent: abiba
|
||||
triggers:
|
||||
- none (retired)
|
||||
---
|
||||
|
||||
> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
|
||||
> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
|
||||
> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
|
||||
|
||||
## Maintains
|
||||
|
||||
- zulip-server: { status: "healthy" | "degraded" | "down" }
|
||||
- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
|
||||
- zulip-config: { syntax_valid: bool, proxy_configured: bool }
|
||||
- last_action: { timestamp, action, result }
|
||||
|
||||
## Detection Rules
|
||||
|
||||
### Rule 1: Zulip Server Down (502/000)
|
||||
- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
|
||||
- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
|
||||
- **Fix**: `docker restart zulip-zulip-1` if container crashed
|
||||
- **Verify**: Re-test server_settings, wait for "healthy"
|
||||
|
||||
### Rule 2: Zulip Config Corruption (400/500 non-proxy)
|
||||
- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
|
||||
- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
|
||||
- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
|
||||
- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
|
||||
|
||||
### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
|
||||
- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
|
||||
- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
|
||||
- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
|
||||
- **Verify**: Re-test via NetBird URL
|
||||
|
||||
### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
|
||||
- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
|
||||
- **Fix (pi)**: `pm2 restart abiba-zulip`
|
||||
- **Fix (Hermes)**: `hermes gateway restart` on agent host
|
||||
- **Verify**: Check health endpoint for `queue_healthy: true`
|
||||
|
||||
### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
|
||||
- **Detect**: No heartbeat in agent log for >600s
|
||||
- **Fix**: Restart gateway on affected host
|
||||
- **Verify**: Check for new heartbeat in logs within 60s
|
||||
|
||||
### Rule 6: NetBird Tunnel Down
|
||||
- **Detect**: Server returns 502 with NetBird HTML in response
|
||||
- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
|
||||
- **Fix**: Alert user — NetBird outage requires manual intervention
|
||||
- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
|
||||
|
||||
## Hosts & Credentials
|
||||
|
||||
| Host | IP | User | Service |
|
||||
|------|----|------|---------|
|
||||
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
|
||||
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
|
||||
| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
|
||||
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
|
||||
|
||||
## Debounce
|
||||
|
||||
All restart actions must debounce: minimum 300s between restarts per agent.
|
||||
Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
|
||||
|
||||
## Reporting
|
||||
|
||||
Every cycle produces a knowledge graph node:
|
||||
- Title: `[LEARN] zulip-self-heal: <timestamp>`
|
||||
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
|
||||
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
|
||||
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
|
||||
|
||||
## Configuration
|
||||
|
||||
Zulip server config at 192.168.68.19:
|
||||
- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
|
||||
- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
|
||||
- Fixed values:
|
||||
- `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
|
||||
- `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
|
||||
- `LOAD_BALANCER_IPS = ["192.168.68.10"]`
|
||||
Reference in New Issue
Block a user