From 584338fb60bd2952f2a6de809afcaf38f1f5f732 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 14:40:22 +0000 Subject: [PATCH 01/16] =?UTF-8?q?feat:=20initial=20OpenProse=20contracts?= =?UTF-8?q?=20=E2=80=94=20LiteLLM=20health=20check=20+=20hello=20world?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - litellm-health.prose.md: Verifies admin UI, API docs, OIDC auth, container health, and aggregate health endpoint. Reusable by any Syslog agent. - hello-world.prose.md: Simple example contract demonstrating OpenProse function pattern with parameters, returns, and postconditions. --- hello-world.prose.md | 15 +++++++++++ litellm-health.prose.md | 56 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) create mode 100644 hello-world.prose.md create mode 100644 litellm-health.prose.md diff --git a/hello-world.prose.md b/hello-world.prose.md new file mode 100644 index 0000000..b960d5a --- /dev/null +++ b/hello-world.prose.md @@ -0,0 +1,15 @@ +--- +kind: function +name: hello-world +description: A simple hello world contract to test OpenProse on pi +--- + +## Parameters +- name: string — The name to greet (default: "World") + +## Returns +- greeting: string — The generated greeting message + +## Ensures +- The greeting includes the provided name +- The greeting is friendly and warm diff --git a/litellm-health.prose.md b/litellm-health.prose.md new file mode 100644 index 0000000..ad538dc --- /dev/null +++ b/litellm-health.prose.md @@ -0,0 +1,56 @@ +--- +kind: function +name: check-litellm-health +description: > + Verifies LiteLLM deployment is healthy by checking admin UI, API docs, + OIDC auth endpoint, container status, and aggregate health on the + backend host. Designed as a reusable contract for any Syslog agent. +--- + +## Parameters + +- public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net") +- backend_host: string — Internal CT host to check containers (default: "192.168.68.116") +- auth_host: string — Authentik server for OIDC (default: "192.168.68.11") + +## Returns + +- overall_status: "healthy" | "degraded" | "down" +- checks: array of { name: string, status: string, detail: string } +- timestamp: string — ISO timestamp of the check run +- duration_ms: number — How long the check took + +## Requires + +- SSH key access to backend_host for container checks +- Network access to public_url and auth_host +- curl and openssl available on the execution host + +## Ensures + +- Each check returns a clear pass/fail status with detail message +- If any endpoint returns non-200, overall_status is "degraded" +- If backend host unreachable or >2 containers down, overall_status is "down" +- If /health/unified reports any non-healthy components, status reflects it +- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy + +## Execution + +1. **Read parameters** — Use provided values or defaults +2. **Check public endpoints**: + - GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard") + - GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI") + - GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths) + - GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc") +3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly): + - GET http://{{backend_host}}/litellm/ui/ → expect 200 + - GET http://{{backend_host}}/litellm/docs → expect 200 +4. **Check aggregate health endpoint**: + - GET {{backend_host}}:9000/health/unified → expect 200, check all components +5. **Check backend container health**: + - SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy + - Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard +6. **Check OIDC auth endpoint**: + - Verify auth.sysloggh.net resolves to {{auth_host}} + - GET https://auth.sysloggh.net/ → expect login page +7. **Compile and report** — Determine overall_status from individual check results From fb27c99c3d52fe1056a1c718e7757db5b1fc826f Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 14:51:04 +0000 Subject: [PATCH 02/16] =?UTF-8?q?feat:=20litellm-self-heal=20=E2=80=94=20p?= =?UTF-8?q?roactive=20remediation=20responsibility?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- litellm-self-heal.prose.md | 74 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 litellm-self-heal.prose.md diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md new file mode 100644 index 0000000..3f6b9d1 --- /dev/null +++ b/litellm-self-heal.prose.md @@ -0,0 +1,74 @@ +--- +kind: responsibility +name: litellm-self-heal +description: > + Standing responsibility that monitors LiteLLM health and proactively + fixes common issues. Runs on a loop — detect drift, fix, verify, report. + Escalates to the user if a fix fails or an unknown issue is detected. +--- + +## Maintains + +- litellm-admin-ui: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/ui/" } +- litellm-api-docs: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/docs" } +- litellm-containers: { status: "healthy", last_check: timestamp, host: "192.168.68.116" } +- litellm-oidc: { status: "healthy", last_check: timestamp, auth_host: "192.168.68.11" } + +## Requires + +- litellm-health:function — The health check contract to run before attempting fixes + +## Continuity + +- Self-driven: check every 300 seconds (5 min) +- Also wakes on user request (`prose run litellm-self-heal`) +- On failure: re-check after 30 seconds, then escalate after 3 consecutive failures + +## Remediation Rules + +### Rule 1: Container Not Healthy +- **Detect**: `harness-litellm` or `harness-nginx` status is not "healthy" +- **Fix**: `docker compose -f /opt/inference-harness/docker-compose.yml up -d ` +- **Verify**: Re-run health check after 15 seconds +- **Escalate**: If still unhealthy after 3 attempts, report to user + +### Rule 2: Admin UI Returns Non-200 +- **Detect**: GET /ui/ returns anything other than 200 +- **Fix**: Restart harness-litellm container +- **Verify**: Re-check /ui/ after 20 seconds +- **Escalate**: If still failing after 2 restarts, report to user + +### Rule 3: Auth Endpoint Unreachable +- **Detect**: Cannot reach auth.sysloggh.net or 192.168.68.11:443 +- **Fix**: Cannot auto-fix (Authentik is external). Report to user with diagnostic info. +- **Escalate**: Immediately (no retry — external dependency) + +### Rule 4: OIDC Login Failing +- **Detect**: OIDC redirect returns error page instead of login +- **Fix**: Check if GENERIC_* env vars are present in LiteLLM container +- **Verify**: If env vars missing, add them and restart +- **Escalate**: If present but still failing, report to user with config dump + +### Rule 5: Docs Endpoint 404 +- **Detect**: GET /docs returns non-200 +- **Fix**: Check DOCS_URL env var — should be `/docs` (not `/litellm/docs` or missing) +- **Verify**: Fix env var and restart container + +## Execution + +1. **Load parameters** — Use defaults or provided overrides +2. **Run health check** — Execute `litellm-health.prose.md` to get baseline +3. **Evaluate results** — Compare each check against expected status +4. **Apply remediation** — For each failed check, find matching rule and execute fix +5. **Verify** — Re-run health check after fix +6. **Report** — Compile: what was broken, what was fixed, what needs user attention +7. **Loop** — If self-driven, wait 300s and repeat from step 2 + +## Returns + +- run_id: string — Unique ID for this remediation cycle +- issues_found: number — Total issues detected +- issues_fixed: number — Issues auto-remediated +- issues_escalated: number — Issues requiring user action +- details: array of { issue: string, action: string, result: string } +- timestamp: string From 41bd3c574802d5d4bc39b925f0d904b1799c5015 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 15:23:21 +0000 Subject: [PATCH 03/16] feat: add daily/weekly digest reporting to self-heal contract - Hourly cycle reports: fix alerts, escalation alerts, 10-cycle all-clear - Daily digest: 24h summary with counts, top actions, uptime - Weekly digest: 7-day summary + KG node for trending - All delivered via Zulip DM to owner --- litellm-self-heal.prose.md | 146 +++++++++++++++++++++++++------------ 1 file changed, 99 insertions(+), 47 deletions(-) diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index 3f6b9d1..0c105e4 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -3,72 +3,124 @@ kind: responsibility name: litellm-self-heal description: > Standing responsibility that monitors LiteLLM health and proactively - fixes common issues. Runs on a loop — detect drift, fix, verify, report. - Escalates to the user if a fix fails or an unknown issue is detected. + fixes common issues. Reports every action via Zulip DM and RA-H OS + knowledge graph for full audit trail. --- ## Maintains -- litellm-admin-ui: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/ui/" } -- litellm-api-docs: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/docs" } -- litellm-containers: { status: "healthy", last_check: timestamp, host: "192.168.68.116" } -- litellm-oidc: { status: "healthy", last_check: timestamp, auth_host: "192.168.68.11" } +- litellm-admin-ui: { status: "healthy", last_check: timestamp } +- litellm-api-docs: { status: "healthy", last_check: timestamp } +- litellm-containers: { status: "healthy", last_check: timestamp } +- litellm-oidc: { status: "healthy", last_check: timestamp } ## Requires -- litellm-health:function — The health check contract to run before attempting fixes +- litellm-health:function ## Continuity -- Self-driven: check every 300 seconds (5 min) -- Also wakes on user request (`prose run litellm-self-heal`) -- On failure: re-check after 30 seconds, then escalate after 3 consecutive failures +- Self-driven: check every 300 seconds +- Also wakes on user request +- On failure: re-check after 30s, escalate after 3 consecutive failures + +## Reporting + +Every remediation cycle produces a structured report in three channels: + +### 1. RA-H OS Knowledge Graph Node +Created as `[LEARN] litellm-self-heal: ` with: +- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" } +- source: Full JSON report of the cycle +- description: Summary of what was found, fixed, and escalated + +### 2. Zulip DM to Owner +Sent immediately for: +- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied" +- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention" +- Every 10th clean cycle — "✅ All Clear (10 checks passed)" + +### 3. Daily Digest (end of day) +A summary of the last 24 hours sent as a single Zulip DM: +``` +📋 LiteLLM Self-Heal — Daily Digest (2026-06-26) + +Total cycles: 288 (every 5 min) +Issues found: 3 + ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1) + └─ Escalated: 0 + +Top actions: + • harness-nginx restarted — 2026-06-26 02:15 + • DOCS_URL corrected — 2026-06-26 07:30 + • harness-nginx restarted — 2026-06-26 14:46 + +Uptime: 23h 47m — All healthy now ✅ +``` + +### 4. Weekly Digest (every Monday) +Same format as daily but covering 7 days. Sent to both Zulip DM and +logged as a `[REPORT]` knowledge graph node for long-term trending. + +### 5. Relay Message (if cross-agent) +If a fix required another agent's help (e.g., Authentik restart), a relay +message is sent to the responsible agent with full context. ## Remediation Rules ### Rule 1: Container Not Healthy -- **Detect**: `harness-litellm` or `harness-nginx` status is not "healthy" -- **Fix**: `docker compose -f /opt/inference-harness/docker-compose.yml up -d ` -- **Verify**: Re-run health check after 15 seconds -- **Escalate**: If still unhealthy after 3 attempts, report to user +Detect → `docker compose up -d ` → verify → log +Escalate after 3 failures -### Rule 2: Admin UI Returns Non-200 -- **Detect**: GET /ui/ returns anything other than 200 -- **Fix**: Restart harness-litellm container -- **Verify**: Re-check /ui/ after 20 seconds -- **Escalate**: If still failing after 2 restarts, report to user +### Rule 2: Admin UI Non-200 +Detect → restart harness-litellm → verify → log +Escalate after 2 failures -### Rule 3: Auth Endpoint Unreachable -- **Detect**: Cannot reach auth.sysloggh.net or 192.168.68.11:443 -- **Fix**: Cannot auto-fix (Authentik is external). Report to user with diagnostic info. -- **Escalate**: Immediately (no retry — external dependency) +### Rule 3: Auth Unreachable +Detect → immediate escalate (external dependency) -### Rule 4: OIDC Login Failing -- **Detect**: OIDC redirect returns error page instead of login -- **Fix**: Check if GENERIC_* env vars are present in LiteLLM container -- **Verify**: If env vars missing, add them and restart -- **Escalate**: If present but still failing, report to user with config dump - -### Rule 5: Docs Endpoint 404 -- **Detect**: GET /docs returns non-200 -- **Fix**: Check DOCS_URL env var — should be `/docs` (not `/litellm/docs` or missing) -- **Verify**: Fix env var and restart container +### Rule 4: Docs 404 +Detect → fix DOCS_URL env var → verify → log ## Execution -1. **Load parameters** — Use defaults or provided overrides -2. **Run health check** — Execute `litellm-health.prose.md` to get baseline -3. **Evaluate results** — Compare each check against expected status -4. **Apply remediation** — For each failed check, find matching rule and execute fix -5. **Verify** — Re-run health check after fix -6. **Report** — Compile: what was broken, what was fixed, what needs user attention -7. **Loop** — If self-driven, wait 300s and repeat from step 2 +1. Run health check +2. Apply remediation for each failure +3. **Generate report** with all actions taken +4. **Log to knowledge graph** — create `[LEARN]` node +5. **Notify user** via Zulip DM if anything changed +6. Wait 300s and repeat -## Returns +## Audit Trail Format -- run_id: string — Unique ID for this remediation cycle -- issues_found: number — Total issues detected -- issues_fixed: number — Issues auto-remediated -- issues_escalated: number — Issues requiring user action -- details: array of { issue: string, action: string, result: string } -- timestamp: string +```json +{ + "run_id": "self-heal-20260626-001", + "timestamp": "2026-06-26T14:00:00Z", + "duration_ms": 1234, + "checks_passed": 7, + "checks_failed": 0, + "issues_found": 0, + "issues_fixed": 0, + "issues_escalated": 0, + "actions": [] +} +``` + +With failures: +```json +{ + "run_id": "self-heal-20260626-002", + "issues_found": 1, + "issues_fixed": 1, + "actions": [ + { + "rule": "Container Not Healthy", + "target": "harness-nginx", + "action": "restarted container", + "result": "healthy", + "duration_ms": 5000 + } + ] +} +``` From f340c6301df123da0e3da22cc5102a1dd97bba4f Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 17:44:27 +0000 Subject: [PATCH 04/16] =?UTF-8?q?feat:=20pm2-self-heal=20=E2=80=94=20monit?= =?UTF-8?q?ors=20and=20restarts=20critical=20PM2=20processes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Responsibility contract that checks abiba-zulip and abiba-telegram every 5 minutes. Restarts stopped/errored processes, escalates if restart count exceeds 5/hour. Logs to knowledge graph and alerts owner via Zulip DM. --- pm2-self-heal.prose.md | 72 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 pm2-self-heal.prose.md diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md new file mode 100644 index 0000000..ae986c2 --- /dev/null +++ b/pm2-self-heal.prose.md @@ -0,0 +1,72 @@ +--- +kind: responsibility +name: pm2-self-heal +description: > + Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and + auto-restarts any that are stopped or errored. Logs every action to + the knowledge graph and alerts the owner via Zulip DM on failures. +--- + +## Maintains + +- abiba-zulip: { status: "online", uptime: string, restarts: number } +- abiba-telegram: { status: "online", uptime: string, restarts: number } +- last_check: timestamp + +## Continuity + +- Self-driven: check every 300 seconds (5 min) +- Also wakes on user request (`prose run pm2-self-heal`) + +## Remediation Rules + +### Rule 1: Process Stopped or Errored +- **Detect**: `pm2 status` shows "stopped" or "errored" for a named process +- **Fix**: `pm2 restart ` +- **Verify**: Re-check status after 5 seconds +- **Escalate**: If still failed after 2 retries, send Zulip DM to owner + +### Rule 2: Process Restarting Too Often +- **Detect**: `pm2 status` shows restarts > 5 in the last hour +- **Fix**: `pm2 delete && pm2 start --only ` +- **Escalate**: Always — alert owner with restart count + +## Execution + +1. **Check PM2 status** — Run `pm2 status` and parse the table +2. **For each critical process** (abiba-zulip, abiba-telegram): + - If status is "online" → pass + - If status is "stopped" or "errored" → apply Rule 1 + - If restarts > 5 → apply Rule 2 +3. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken +4. **Alert owner** — Send Zulip DM if escalation needed +5. **Wait 5 min** → repeat from step 1 + +## Example Output (when healthy) + +```json +{ + "run_id": "pm2-heal-001", + "checked_at": "2026-06-26T15:00:00Z", + "processes": { + "abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 }, + "abiba-telegram": { "status": "online", "uptime": "48h", "restarts": 0 } + }, + "actions_taken": [], + "overall": "healthy" +} +``` + +## Example Output (when fixed) + +```json +{ + "processes": { + "abiba-zulip": { "status": "errored → restarted → online" } + }, + "actions_taken": [ + { "target": "abiba-zulip", "action": "pm2 restart", "result": "online" } + ], + "overall": "fixed" +} +``` From be15b9c2966df90b72c03493060da8cdb0e3b56d Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 18:39:03 +0000 Subject: [PATCH 05/16] =?UTF-8?q?feat:=20build-zulip-plugin=20=E2=80=94=20?= =?UTF-8?q?recursive=20plugin=20generation=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-improving contract that generates, validates, and iteratively improves Hermes platform plugins. Tracks generations in knowledge graph, learns from past successes and failures, and can update its own rules as new patterns emerge. --- build-zulip-plugin.prose.md | 81 +++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 build-zulip-plugin.prose.md diff --git a/build-zulip-plugin.prose.md b/build-zulip-plugin.prose.md new file mode 100644 index 0000000..ee6230d --- /dev/null +++ b/build-zulip-plugin.prose.md @@ -0,0 +1,81 @@ +--- +kind: responsibility +name: build-zulip-plugin +description: > + Generates and iteratively improves a Hermes Zulip platform plugin. + Each run can produce a new version of the plugin, learning from + previous runs and incrementally improving the adapter code. + The contract itself can be updated to capture new patterns. +--- + +## Maintains + +- plugin_version: string — Current version of the generated plugin +- generation_count: number — How many generations have been produced +- last_generation: timestamp — When the last generation was created +- known_issues: array — Issues discovered in the current version + +## Parameters + +- zulip_site: string — Zulip server URL (default: "https://chat.sysloggh.net") +- bot_email_prefix: string — Prefix for bot emails (default: "abiba-bot") +- output_dir: string — Where to write the plugin (default: "plugins/platforms/zulip") +- generation_goal: string — What to improve this generation (default: "fix bugs and improve reliability") + +## Remembers + +Each generation stores: +- What worked well (success patterns) +- What broke (failure modes) +- What the user complained about (pain points) + +## Generation Rules + +### Rule 1: First Generation +- Generate complete plugin.yaml, adapter.py, __init__.py +- Follow Hermes BasePlatformAdapter pattern +- Include: connect, disconnect, send, edit_message, send_typing +- Include: DM-first routing, @mention detection, @all-bots support + +### Rule 2: Subsequent Generations +- Read the previous generation's output and known issues +- Apply improvements based on generation_goal +- Fix any issues from the previous generation +- Add new capabilities if needed + +### Rule 3: Testing +- After generating, run syntax check on the Python code +- Verify plugin.yaml is valid YAML +- Log the generation result to knowledge graph + +### Rule 4: Self-Improvement +- After N generations, review the contract itself +- If new patterns emerged, update the Generation Rules +- If old rules are no longer relevant, remove them + +## Execution + +1. **Read state** — Check previous generations from knowledge graph +2. **Generate or improve** — Based on generation count: + - Generation 1: Scaffold full plugin from scratch + - Generation N: Read previous code, apply improvements +3. **Validate** — Syntax check, structure check, dependency check +4. **Log** — Save generation result to knowledge graph as [LEARN] +5. **Report** — Output what was generated, what changed, what needs review + +## Example Output + +```json +{ + "generation": 1, + "version": "1.0.0", + "files_written": [ + "plugins/platforms/zulip/plugin.yaml", + "plugins/platforms/zulip/__init__.py", + "plugins/platforms/zulip/adapter.py" + ], + "syntax_check": "pass", + "known_issues": [], + "next_generation_goal": "add streaming responses via placeholder->edit pattern" +} +``` From 37de60ae3223eb742ad6364d77a742759497769e Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 18:45:06 +0000 Subject: [PATCH 06/16] feat: add event-driven triggers and success criteria to plugin builder - Success Criteria section defines measurable quality gates - Continuity section explains event-driven wake modes - Each generation tracks which criteria passed/failed - Improvements triggered by failures, not fixed timers --- build-zulip-plugin.prose.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/build-zulip-plugin.prose.md b/build-zulip-plugin.prose.md index ee6230d..9333f13 100644 --- a/build-zulip-plugin.prose.md +++ b/build-zulip-plugin.prose.md @@ -22,12 +22,49 @@ description: > - output_dir: string — Where to write the plugin (default: "plugins/platforms/zulip") - generation_goal: string — What to improve this generation (default: "fix bugs and improve reliability") +## Continuity + +The improvement cycle is **event-driven**, not just cron-based: + +- **On check failure**: If any Success Criteria fails, wake immediately to fix +- **On user request**: `prose run build-zulip-plugin` — explicit upgrade +- **On new pattern**: If another plugin or agent reports a better pattern, wake to assimilate +- **Retrospective**: Every 24 hours if no other trigger fired — scan for subtle degradation +- **On first deploy**: Always run Generation 1 when no plugin exists yet + +## Success Criteria (What Makes a Plugin "Good") + +The plugin is considered GOOD when ALL of these pass: + +### Connectivity +- plugin.connected == true — Establishes and maintains Zulip event queue +- plugin.reconnect_on_failure == true — Reconnects after queue expiry +- plugin.recovers_from_network_loss == true — Handles temporary network drops + +### Message Flow +- dm_response_time_ms < 10000 — DMs get a response within 10 seconds +- stream_mention_detection == true — @mentions in streams are detected and routed +- all_bots_detection == true — @all-bots mentions are detected +- echo_loop_prevention == true — Never replies to its own messages + +### Code Quality +- syntax_check == "pass" — All Python files are valid +- has_register_function == true — Exposes `register(ctx)` entry point +- plugin_yaml_valid == true — plugin.yaml parses correctly +- follows_base_adapter_pattern == true — Extends BasePlatformAdapter + +### Reliability +- graceful_disconnect == true — shutdown doesn't crash +- handles_malformed_messages == true — Bad JSON doesn't kill the poll loop +- typing_indicators_work == true — Sends typing notifications + ## Remembers Each generation stores: - What worked well (success patterns) - What broke (failure modes) - What the user complained about (pain points) +- Which Success Criteria passed and failed ## Generation Rules From acfe16e5e7e8e2720721524f1d8da8c36ab32ef6 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 18:52:54 +0000 Subject: [PATCH 07/16] =?UTF-8?q?feat:=20infrastructure-control=20?= =?UTF-8?q?=E2=80=94=20reference=20pattern=20for=20environment=20reach?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maps how OpenProse contracts connect to Gitea, containers, remote agents, and infrastructure. Documents the access matrix, what works today, and what gaps remain for full enforcement. --- infrastructure-control.prose.md | 142 ++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 infrastructure-control.prose.md diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md new file mode 100644 index 0000000..9da55d3 --- /dev/null +++ b/infrastructure-control.prose.md @@ -0,0 +1,142 @@ +--- +kind: pattern +name: infrastructure-control +description: > + Architectural pattern showing how OpenProse contracts connect to + Gitea, containers, and remote agents to enforce conditions across + all environments. Not an executable contract — a reference model. +--- + +## The Full Control Stack + +``` + ┌─────────────────────────┐ + │ OpenProse Contract │ + │ (declares what's true) │ + └──────────┬──────────────┘ + │ + ┌────────────────────┼────────────────────┐ + ▼ ▼ ▼ + ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ + │ Agent │ │ Agent │ │ Agent │ + │ (pi) │ │ (Tanko) │ │ (Mumuni) │ + │ CT 100 │ │ CT 112 │ │ CT 114 │ + └──────┬──────┘ └──────┬───────┘ └──────┬───────┘ + │ │ │ + └──────────────────┼────────────────────┘ + ▼ + ┌─────────────────┐ + │ Gitea Repos │ + │ (source of │ + │ truth for │ + │ code + │ + │ contracts) │ + └────────┬────────┘ + │ + ┌──────────────────┼────────────────────┐ + ▼ ▼ ▼ + ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ + │ CT 116 │ │ Authentik │ │ Zulip │ + │ (LiteLLM, │ │ CT .11 │ │ CT .117 │ + │ nginx, │ │ (OIDC auth) │ │ (messaging) │ + │ router) │ │ │ │ │ + └─────────────┘ └──────────────┘ └──────────────┘ +``` + +## How Each Layer Connects + +### 1. Code → Gitea +The contract enforces code state via git operations: + +```yaml +## Enforce +- git pull origin main on CT 116 — ensures latest config is deployed +- git diff HEAD...origin/main — warns if drift is detected +- git commit + push after auto-fix — changes are versioned +``` + +**What the agent does:** Runs `git` commands via SSH to check and update repos. +**What enforces it:** Agent has SSH keys and Gitea credentials. + +### 2. Contracts → Agents +Each agent runs OpenProse contracts in its own environment: + +| Agent | Runs contracts via | Can reach | +|---|---|---| +| **Abiba (pi)** | `prose run` in-session | CT 116 (SSH), Zulip API, Gitea API | +| **Tanko (Hermes)** | Hermes Gateway with OpenProse skill | CT 112, Zulip API, Gitea API | +| **Mumuni (Hermes)** | Hermes Gateway with OpenProse skill | CT 114, Zulip API, Gitea API | + +**What enforces it:** Each agent has SSH keys to the environments it manages. + +### 3. Contracts → Containers +The contract enforces container state via Docker commands: + +```yaml +## Maintains +- harness-litellm.status == "healthy" +- harness-nginx.status == "healthy" + +## Enforce +- If container unhealthy → `docker compose up -d ` + - Requires: SSH access to host + - Verifies: `docker ps` after restart +``` + +**What the agent does:** SSH to the host, run Docker commands. +**What enforces it:** SSH key + Docker socket access on the target CT. + +### 4. Contracts → Remote Agents (Tanko, Mumuni) +When a contract needs another agent to act: + +```yaml +## If out of scope +- If Authentik server down → relay to Mumuni (has access) + - Creates relay message with diagnostic data + - Mumuni's contract picks it up and acts +``` + +**What the agent does:** Creates a relay message via RA-H OS. +**What enforces it:** Mumuni's own contracts check relays on wake. + +## Access Matrix + +Each agent needs explicit credentials to each environment: + +| Environment | Abiba (pi) | Tanko (Hermes) | Mumuni (Hermes) | +|---|---|---|---| +| **Gitea** | ✅ API token | Need to add | Need to add | +| **CT 116 (LiteLLM)** | ✅ SSH key | Need to add | ❌ (has own CTs) | +| **Zulip API** | ✅ Bot token | ✅ Bot token | ✅ Bot token | +| **RA-H OS** | ✅ MCP bridge | ✅ MCP bridge | ✅ MCP bridge | +| **Authentik** | ❌ No SSH | ❌ No SSH | ❌ No SSH | + +## The Gap (What's Missing) + +Currently, the contracts CAN: + +| Capability | Works? | How | +|---|---|---| +| Read container state | ✅ SSH to CT 116 | +| Restart containers | ✅ SSH + docker compose | +| Commit to Gitea | ✅ API token | +| Alert via Zulip | ✅ Bot API | +| Log to KG | ✅ MCP bridge | + +But they CANNOT yet: + +| Gap | Why | Fix | +|---|---|---| +| Push code to Tanko's CT | No SSH key for CT 112 | Add Abiba's key to CT 112 | +| Restart Hermes gateway | No remote command | Add `hermes gateway restart` via SSH | +| Auto-merge PRs | Needs Gitea write token | Already have it, need to wire | + +## Reference Implementation + +The `litellm-self-heal` contract already implements most of this pattern: +1. It checks conditions (container health, endpoints) +2. It enforces fixes (docker compose up) +3. It logs results (knowledge graph) +4. It alerts via Zulip DM + +The same pattern extends to any environment where the agent has credentials. From 2831b06c1ec3b025295b8dcb91c52cb33a3a10cc Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 19:15:53 +0000 Subject: [PATCH 08/16] =?UTF-8?q?feat:=20zulip-mention-reliability=20?= =?UTF-8?q?=E2=80=94=20self-testing=20contract=20for=20@mention=20detectio?= =?UTF-8?q?n?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Monitors whether Abiba Bot responds to @mentions in stream topics. Self-test: sends a test mention, checks for response within 30s. Documents 4 known failure modes with remediation rules. Follows same self-improving pattern as litellm-self-heal. --- zulip-mention-reliability.prose.md | 72 ++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 zulip-mention-reliability.prose.md diff --git a/zulip-mention-reliability.prose.md b/zulip-mention-reliability.prose.md new file mode 100644 index 0000000..b7c8513 --- /dev/null +++ b/zulip-mention-reliability.prose.md @@ -0,0 +1,72 @@ +--- +kind: responsibility +name: zulip-mention-reliability +description: > + Monitors whether Abiba Bot is correctly detecting and responding to + @mentions in Zulip stream topics. Tests periodically, diagnoses + failures, and attempts fixes. Same self-improving pattern as + litellm-self-heal. +--- + +## Maintains + +- mention_response_rate: number — % of @mentions that get a response (target: >95%) +- last_test_result: "pass" | "fail" | "degraded" +- known_failure_modes: array — History of what went wrong and how it was fixed +- queue_health: "healthy" | "expired" | "reconnecting" + +## Success Criteria + +- test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s +- stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat +- queue_active == true — PM2's event queue is registered and polling +- no_echo_loop == true — Bot doesn't respond to its own messages +- dm_always_works == true — DMs still respond even if streams are broken + +## Known Failure Modes (from History) + +| Failure | Root Cause | Fix | +|---|---|---| +| No stream events | Bot had 0 stream subscriptions | Subscribe via API | +| No @mention detection | botUserId was null (register endpoint doesn't return it) | Fetch from /api/v1/users/me | +| No stream events | zulip-js library sends multipart/form-data instead of form-urlencoded | Use URLSearchParams directly | +| Double responses | Both TUI and PM2 loaded the extension | ZULIP_EXTENSION_ACTIVE guard | + +## Remediation Rules + +### Rule 1: Queue Expired +- Detect: Poll returns BAD_EVENT_QUEUE_ID +- Fix: `pm2 restart abiba-zulip` +- Verify: Check logs for "Connected, queue:" + +### Rule 2: Subscription Missing +- Detect: `GET /api/v1/users/me/subscriptions` returns 0 for #agent-hub +- Fix: Subscribe via `POST /api/v1/users/me/subscriptions` +- Verify: Re-check subscriptions + +### Rule 3: Extension Crashed +- Detect: PM2 status shows "errored" for abiba-zulip +- Fix: `pm2 restart abiba-zulip` +- Escalate: If restarts >5 in last hour, alert user + +### Rule 4: No Response to Test Mention +- Detect: Send test @mention, wait 30s, check if response appeared +- Fix: Run diagnostics (check queue, check code, check logs) +- Log: Record failure mode as new entry in Known Failure Modes + +## Continuity + +- **On user report**: User says "you didn't respond" → run diagnostic +- **On test failure**: Test mention fails → run remediation +- **Retrospective**: Every 6 hours — send a test @mention to self-verify +- **On PM2 restart**: Always run a self-test after restart + +## Execution + +1. **Self-test**: Send `@**Abiba Bot** _selftest_` in #agent-hub, wait for response +2. **If no response in 30s**: Run diagnostics +3. **Diagnose**: Check queue, subscriptions, PM2 status, logs +4. **Fix**: Apply matching remediation rule +5. **Retest**: Try again +6. **Log**: Record success or new failure mode +7. **Report**: DM user only if escalation needed From 4c6b0410ea3a428d6ece70be350827ffc9860602 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 19:53:55 +0000 Subject: [PATCH 09/16] =?UTF-8?q?fix:=20pm2-self-heal=20=E2=80=94=20self-p?= =?UTF-8?q?rocess=20guard,=20never=20restart=20abiba-zulip?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Added --no-color flag to all PM2 commands - Self-process guard: abiba-zulip is READ-ONLY, never restarted - Corrected column indices (PID=6, restarts=8, status=9) - Added pm2-self-heal.sh shell script implementation --- pm2-self-heal.prose.md | 22 +++++++++------ pm2-self-heal.sh | 64 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+), 8 deletions(-) create mode 100755 pm2-self-heal.sh diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md index ae986c2..1878a2a 100644 --- a/pm2-self-heal.prose.md +++ b/pm2-self-heal.prose.md @@ -5,6 +5,7 @@ description: > Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. Logs every action to the knowledge graph and alerts the owner via Zulip DM on failures. + CRITICAL: Never restart abiba-zulip — it runs this contract. --- ## Maintains @@ -33,14 +34,18 @@ description: > ## Execution -1. **Check PM2 status** — Run `pm2 status` and parse the table -2. **For each critical process** (abiba-zulip, abiba-telegram): +1. **Check PM2 status** — Run `pm2 status --no-color` and parse the table (5th data column = PID, 8th = restarts, 9th = status) +2. **Check abiba-telegram**: - If status is "online" → pass - If status is "stopped" or "errored" → apply Rule 1 - - If restarts > 5 → apply Rule 2 -3. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken -4. **Alert owner** — Send Zulip DM if escalation needed -5. **Wait 5 min** → repeat from step 1 + - If restarts > 5 → alert owner +3. **Check abiba-zulip** (self-process, read-only): + - If status is "online" → pass, log restarts count + - If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately + - If restarts > 5 in last hour → alert owner with full diagnostics +4. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken +5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting) +6. **Wait 5 min** → repeat from step 1 ## Example Output (when healthy) @@ -62,10 +67,11 @@ description: > ```json { "processes": { - "abiba-zulip": { "status": "errored → restarted → online" } + "abiba-telegram": { "status": "errored → restarted → online" }, + "abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 } }, "actions_taken": [ - { "target": "abiba-zulip", "action": "pm2 restart", "result": "online" } + { "target": "abiba-telegram", "action": "pm2 restart", "result": "online" } ], "overall": "fixed" } diff --git a/pm2-self-heal.sh b/pm2-self-heal.sh new file mode 100755 index 0000000..c48bbe3 --- /dev/null +++ b/pm2-self-heal.sh @@ -0,0 +1,64 @@ +#!/bin/bash +# pm2-self-heal — hourly PM2 process check +# Part of the pm2-self-heal prose contract +# Only sends Zulip DM when action is taken or escalation needed +# Field positions (awk -F'│'): $7=pid $8=uptime $9=restarts $10=status + +ZULIP_BOT="abiba-bot@chat.sysloggh.net" +ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8" +ZULIP_SITE="https://chat.sysloggh.net" +OWNER_ID=9 +LOG="/tmp/pm2-self-heal.log" + +send_alert() { + local subject="$1" + local body="$2" + curl -s -X POST "$ZULIP_SITE/api/v1/messages" \ + -u "$ZULIP_BOT:$ZULIP_KEY" \ + -d "type=private" \ + -d "to=[$OWNER_ID]" \ + -d "content=$subject\n$body" > /dev/null 2>&1 + echo "[$(date '+%Y-%m-%d %H:%M:%S')] Alert sent" >> "$LOG" +} + +# Parse PM2 status (--no-color to avoid ANSI escape codes in awk columns) +STATUS=$(pm2 status --no-color 2>/dev/null) + +ALERTS="" + +# Check abiba-telegram (safe to auto-restart) +TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram") +TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs) +TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs) + +if [ "$TEL_STATUS" != "online" ]; then + pm2 restart abiba-telegram > /dev/null 2>&1 + sleep 3 + TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram") + TEL_STATUS2=$(echo "$TEL_LINE2" | awk -F'│' '{print $10}' | xargs) + if [ "$TEL_STATUS2" = "online" ]; then + ALERTS="${ALERTS}⚠️ abiba-telegram was **$TEL_STATUS** → restarted to online\n" + else + ALERTS="${ALERTS}🚨 abiba-telegram **failed restart** (was $TEL_STATUS, still $TEL_STATUS2)\n" + fi +fi + +# Check abiba-zulip (read-only — never restart) +ZUL_LINE=$(echo "$STATUS" | grep "abiba-zulip") +ZUL_STATUS=$(echo "$ZUL_LINE" | awk -F'│' '{print $10}' | xargs) +ZUL_RESTARTS=$(echo "$ZUL_LINE" | awk -F'│' '{print $9}' | xargs) + +if [ "$ZUL_STATUS" != "online" ]; then + ALERTS="${ALERTS}🚨 **abiba-zulip is $ZUL_STATUS** — needs investigation!\n" + ALERTS="${ALERTS} NOT auto-restarting (runs this contract)\n" +elif [ "$ZUL_RESTARTS" -gt 5 ] 2>/dev/null; then + ALERTS="${ALERTS}⚠️ abiba-zulip has **$ZUL_RESTARTS restarts** — may need attention\n" +fi + +# Log check +echo "[$(date '+%Y-%m-%d %H:%M:%S')] tel=$TEL_STATUS zul=$ZUL_STATUS alerts=${ALERTS:+yes}" >> "$LOG" + +# Only DM when there's something to report +if [ -n "$ALERTS" ]; then + send_alert "**PM2 Self-Heal — $(date '+%H:%M UTC')**" "$ALERTS" +fi From e090b27e204863c8285482ec8b3feba015259264 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 27 Jun 2026 13:11:15 +0000 Subject: [PATCH 10/16] =?UTF-8?q?feat:=20expand=20infrastructure-control?= =?UTF-8?q?=20pattern=20=E2=80=94=20Proxmox,=20Docker,=20storage,=20networ?= =?UTF-8?q?k?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full expansion of the infrastructure-control pattern covering: Layer 1 — Proxmox Cluster (5 nodes) - 4 monitoring rules (offline, resource pressure, stopped VMs/CTs, storage) - Auto-start for 8 critical CTs/VMs on unexpected stop - Node resource threshold alerting Layer 2 — Docker Ecosystems (3 hosts, 22 containers) - 6 monitoring rules across docker-vm, syslog-api, netbird - Auto-restart for critical containers and stacks - Disk pressure auto-prune Layer 3 — Storage Fabric (3.6TB + 7.3TB NFS) - Mount availability monitoring - Growth rate tracking for high-use volumes - PBS backup compliance checking Layer 4 — Network Services (DNS, auth, git, chat, VPN) - Service endpoint monitoring - Escalation to Mumuni for authentik issues Layer 5 — Agent Health (6 agents across 3 platforms) - CT auto-start for stopped agents - Health endpoint verification Full access matrix and priority-based SLA table. --- infrastructure-control.prose.md | 426 ++++++++++++++++++++++++-------- 1 file changed, 320 insertions(+), 106 deletions(-) diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index 9da55d3..5a698bd 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -2,141 +2,355 @@ kind: pattern name: infrastructure-control description: > - Architectural pattern showing how OpenProse contracts connect to - Gitea, containers, and remote agents to enforce conditions across - all environments. Not an executable contract — a reference model. + Full infrastructure monitoring and control pattern covering the 5-node + Proxmox cluster, 3 Docker ecosystems (22 containers), storage fabric, + and network services. Declares what agents can observe, what they can + enforce, and where human escalation is required. --- -## The Full Control Stack +## Architecture Overview ``` - ┌─────────────────────────┐ - │ OpenProse Contract │ - │ (declares what's true) │ - └──────────┬──────────────┘ - │ - ┌────────────────────┼────────────────────┐ - ▼ ▼ ▼ - ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ - │ Agent │ │ Agent │ │ Agent │ - │ (pi) │ │ (Tanko) │ │ (Mumuni) │ - │ CT 100 │ │ CT 112 │ │ CT 114 │ - └──────┬──────┘ └──────┬───────┘ └──────┬───────┘ - │ │ │ - └──────────────────┼────────────────────┘ - ▼ - ┌─────────────────┐ - │ Gitea Repos │ - │ (source of │ - │ truth for │ - │ code + │ - │ contracts) │ - └────────┬────────┘ - │ - ┌──────────────────┼────────────────────┐ - ▼ ▼ ▼ - ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ - │ CT 116 │ │ Authentik │ │ Zulip │ - │ (LiteLLM, │ │ CT .11 │ │ CT .117 │ - │ nginx, │ │ (OIDC auth) │ │ (messaging) │ - │ router) │ │ │ │ │ - └─────────────┘ └──────────────┘ └──────────────┘ + ┌──────────────────────────────────────────┐ + │ OpenProse Contracts │ + │ (monitor → diagnose → remediate → log) │ + └─────┬──────────┬──────────┬──────────────┘ + │ │ │ + ┌────────────┼──────────┼──────────┼────────────┐ + ▼ ▼ ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────┐ ┌──────┐ ┌──────────┐ + │ Proxmox │ │ Docker │ │Storage│ │Network│ │ Agents │ + │ Cluster │ │Ecosystems│ │Fabric │ │Services│ │(health) │ + │ 5 nodes │ │ 3 hosts │ │NFS+ZFS│ │DNS+Auth│ │6 agents │ + └──────────┘ └──────────┘ └──────┘ └──────┘ └──────────┘ ``` -## How Each Layer Connects +## Layer 1: Proxmox Cluster -### 1. Code → Gitea -The contract enforces code state via git operations: +### Cluster Topology -```yaml -## Enforce -- git pull origin main on CT 116 — ensures latest config is deployed -- git diff HEAD...origin/main — warns if drift is detected -- git commit + push after auto-fix — changes are versioned +``` +minipve (.12) ─── 16C/30GB ─── authentik CT 104, gitea CT 110, +│ mumuni CT 114, syslog-api CT 116, +│ jitsi CT 118 +│ +amdpve (.15) ─── 32C/62GB ─── abiba CT 100, kagentz CT 105, +│ tanko CT 112, tdunna CT 111, +│ baggy CT 113, scottdenya CT 115 +│ +storepve (.6) ──── 28C/31GB ─── docker-vm VM 109, ra-h-os CT 106, +│ PBS CT 107, media CT 108, +│ zulip CT 117 +│ +acerpve (.9) ──── 28C/31GB ─── llm-gpu VM 101, adguard CT 102 +│ +ocupve (.5) ──── 12C/14GB ─── ocu-llm VM 103 ``` -**What the agent does:** Runs `git` commands via SSH to check and update repos. -**What enforces it:** Agent has SSH keys and Gitea credentials. - -### 2. Contracts → Agents -Each agent runs OpenProse contracts in its own environment: - -| Agent | Runs contracts via | Can reach | -|---|---|---| -| **Abiba (pi)** | `prose run` in-session | CT 116 (SSH), Zulip API, Gitea API | -| **Tanko (Hermes)** | Hermes Gateway with OpenProse skill | CT 112, Zulip API, Gitea API | -| **Mumuni (Hermes)** | Hermes Gateway with OpenProse skill | CT 114, Zulip API, Gitea API | - -**What enforces it:** Each agent has SSH keys to the environments it manages. - -### 3. Contracts → Containers -The contract enforces container state via Docker commands: +### Maintains (Monitoring) ```yaml ## Maintains -- harness-litellm.status == "healthy" -- harness-nginx.status == "healthy" - -## Enforce -- If container unhealthy → `docker compose up -d ` - - Requires: SSH access to host - - Verifies: `docker ps` after restart +- cluster.nodes: each node's status (online/offline), CPU%, RAM%, uptime +- cluster.vms: each VM/CT's status (running/stopped), CPU, RAM, disk usage +- cluster.storage: each storage target's usage %, active/inactive +- cluster.ha: high-availability state and fencing status ``` -**What the agent does:** SSH to the host, run Docker commands. -**What enforces it:** SSH key + Docker socket access on the target CT. +### Rules (Monitoring) -### 4. Contracts → Remote Agents (Tanko, Mumuni) -When a contract needs another agent to act: +``` +Rule PM-1: Node Offline + Detect → PVE API shows node status != "online" + Action → Alert via Zulip DM with node name and last known uptime + Escalate → If >2 nodes offline, page owner + Requires: PVE API token + +Rule PM-2: Node Resource Pressure + Detect → CPU > 80% or RAM > 85% for 5 consecutive checks + Action → Log to KG, alert via Zulip DM with resource snapshot + Escalate → If sustained >90% for 1h, page owner + Requires: PVE API token + +Rule PM-3: VM/CT Unexpectedly Stopped + Detect → PVE API shows running VM now "stopped" + Action → `pct start ` or `qm start ` via PVE API + Verify → Re-check status after 30s + Escalate → If still stopped after 2 retries, alert owner + Requires: PVE API token with appropriate permissions + +Rule PM-4: Storage Usage Warning + Detect → Storage pool usage > 80% + Action → Log to KG, alert via Zulip DM + Escalate → If >90%, immediate alert (risk of VM freeze) + Requires: PVE API +``` + +### Control (Remediation) ```yaml -## If out of scope -- If Authentik server down → relay to Mumuni (has access) - - Creates relay message with diagnostic data - - Mumuni's contract picks it up and acts +## Enforce +- PM-3 auto-start: start critical VMs/CTs when unexpectedly stopped + - Critical: abiba (100), tanko (112), mumuni (114), syslog-api (116), + docker-vm (109), gitea (110), authentik (104), zulip (117) +- PM-4 storage: if mediastore > 85%, alert with biggest consumers + +## Prohibited +- Never migrate VMs/CTs between nodes automatically +- Never delete storage volumes +- Never modify cluster HA configuration ``` -**What the agent does:** Creates a relay message via RA-H OS. -**What enforces it:** Mumuni's own contracts check relays on wake. +## Layer 2: Docker Ecosystems + +### Docker Hosts + +```yaml +Host: docker-vm (192.168.68.7, storepve VM 109) + Resources: 10GB RAM, 4 vCPU, 158GB disk (33% used) + Stacks: + firecrawl: api, rabbitmq, postgres, playwright, redis + searxng: searxng (:8888), valkey + home_stack: jdownloader (:5800), bentopdf (:8989), + pulse (:7655), audiobookshelf (:13378) + Mounts: /media/storage (3.6TB NFS), /media/mediastore (7.3TB NFS) + +Host: syslog-api (192.168.68.116, minipve CT 116) + Resources: 6GB RAM, 2 vCPU, 39GB disk + Stacks: + harness: litellm (:4001), nginx (:80), router (:9000), + postgres, redis, dashboard (:3000) + +Host: netbird (72.61.0.17, external VM) + Stacks: + netbird: server, dashboard, proxy, crowdsec, traefik +``` + +### Maintains (Monitoring) + +```yaml +## Maintains +- docker.hosts: each host's status, container count, resource usage +- docker.containers: each container's status (running/restarting/exited), + uptime, restart count, image version +- docker.stacks: each compose stack's service count and health +- docker.resources: per-host disk usage, Docker system df +``` + +### Rules (Monitoring) + +``` +Rule DC-1: Container Not Running + Detect → `docker ps` shows "exited" or "restarting" for critical container + Action → `docker compose up -d ` on the affected stack + Verify → Re-check after 10s + Escalate → If still failed after 3 retries, alert owner + Requires: SSH to docker-vm or CT 116 + +Rule DC-2: Container Restart Looping + Detect → Container restart count > 5 in last hour + Action → `docker logs --tail 50 ` → diagnose → KG log + Escalate → Always alert owner with diagnostic excerpt + Requires: SSH to docker-vm or CT 116 + +Rule DC-3: Docker Host Disk Pressure + Detect → `docker system df` shows < 10% reclaimable or disk > 85% + Action → `docker system prune -af` (safe: removes unused images/containers) + Verify → Re-check disk usage + Escalate → If still >85% after prune, alert owner + Requires: SSH to docker-vm + +Rule DC-4: Stack Health Check Failed + Detect → Container HEALTHCHECK status shows "unhealthy" + Action → `docker compose restart ` + Verify → Re-check health after 15s + Escalate → After 2 retries + Requires: SSH to affected host + +Rule DC-5: Firecrawl API Down + Detect → HTTP GET :3002/health returns non-200 + Action → `docker compose -f /opt/search-stack/firecrawl-source/docker-compose.yaml restart firecrawl-api-1` + Escalate → If still down after 2 retries + Requires: SSH to docker-vm + +Rule DC-6: SearXNG Down + Detect → HTTP GET :8888 returns non-200 + Action → `docker compose -f /opt/search-stack/searxng/docker-compose.yml restart searxng` + Escalate → If still down after 2 retries + Requires: SSH to docker-vm +``` + +### Control (Remediation) + +```yaml +## Enforce +- DC-1 auto-restart: restart critical containers when exited +- DC-3 auto-prune: prune unused Docker artifacts weekly +- DC-4/5/6: restart unhealthy stacks with verification + +## Prohibited +- Never remove volumes (data loss risk) +- Never modify docker-compose files remotely +- Never update container images without owner approval +``` + +## Layer 3: Storage Fabric + +### Storage Topology + +``` +storepve (.6) ── NFS ──► docker-vm: /media/storage (3.6TB, 8% used) + └── audiobookshelf, JDownloader downloads +storepve (.6) ── NFS ──► docker-vm: /media/mediastore (7.3TB, 75% used) + └── media library, configs +storepve ── PBS CT 107 ──► backup target for all VMs/CTs +``` + +### Maintains + +```yaml +## Maintains +- storage.nfs_mounts: each mount's availability, usage %, inode count +- storage.pbs: last successful backup timestamp, backup schedule compliance +- storage.mediastore_growth: weekly growth rate (high-use volume) +``` + +### Rules + +``` +Rule ST-1: NFS Mount Unreachable + Detect → `df` or `mountpoint -q` fails for NFS path + Action → Check storepve NFS server status via PVE API + Escalate → Immediate alert — containers may be read-only or crashing + Requires: PVE API + SSH to docker-vm + +Rule ST-2: Mediadisk Growth Alert + Detect → mediastore (7.3TB, 75%) growing > 50GB/week + Action → Identify top consumers: `du -sh /media/mediastore/* | sort -rh | head 10` + Log → KG node with growth trend + Escalate → If projected to exceed 90% within 30 days + +Rule ST-3: PBS Backup Compliance + Detect → Any CT/VM without a backup in the last 48h + Action → Log to KG, alert via Zulip DM + Requires: PVE API to check PBS tasks +``` + +## Layer 4: Network Services + +### Service Map + +| Service | Host | IP | Port | Purpose | +|---------|------|----|------|---------| +| adguard | CT 102 | .102 | 53, 80, 443 | DNS + DHCP | +| authentik | CT 104 | .104 | 443 | OIDC auth provider | +| gitea | CT 110 | .110 | 3000, 22 | Git hosting | +| zulip | CT 117 | .117 | 443 | Team chat | +| netbird | 72.61.0.17 | .17 | 443, 8006 | VPN controller | +| minipve | .12 | .12 | 443, 8006 | PVE API endpoint | +| PBS | CT 107 | .107 | 8007 | Proxmox Backup Server | + +### Maintains + +```yaml +## Maintains +- network.dns_resolution: critical hostnames resolve correctly +- network.auth_endpoints: authentik OIDC login flow works +- network.external_access: public endpoints respond (litellm, git, chat, auth, pulse) +``` + +### Rules + +``` +Rule NW-1: DNS Failure + Detect → `host` or `nslookup` fails for known hostnames + Action → Check adguard CT 102 status via PVE API + Escalate → If authentik/gitea unreachable due to DNS → page owner + Requires: PVE API + dig/nslookup + +Rule NW-2: Auth Endpoint Down + Detect → `GET https://auth.sysloggh.net` returns non-200 + Action → Check authentik CT 104 via PVE API → start if stopped + Escalate → Can't fix: relay to Mumuni (has authentik SSH access) + Requires: PVE API + +Rule NW-3: Public Endpoint Unreachable + Detect → Any public URL fails (litellm, git, chat, pulse) + Action → Check container + nginx health on relevant host + Requires: SSH to affected host + curl +``` + +## Layer 5: Agent Health + +### Agent Registry + +| Agent | Platform | CT | Status Endpoint | Last Seen | +|-------|----------|----|----------------|-----------| +| abiba | pi | 100 | :9200/health | live | +| tanko | Hermes | 122 | (via Gateway) | live | +| mumuni | Hermes | 114 | (via Gateway) | live | +| kagentz | Agent Zero | 105 | (via PVE) | running | +| koby | Hermes | 111 | (via PVE) | stopped | +| koonimo | Hermes | 113 | (via PVE) | stopped | + +### Rules + +``` +Rule AG-1: Agent VM/CT Stopped + Detect → PVE API shows agent CT as "stopped" + Action → `pct start ` via PVE API + Verify → Re-check after 15s + Escalate → If persistent, alert owner + Requires: PVE API + +Rule AG-2: Agent Not Responding + Detect → Health endpoint non-200 or Gateway disconnected + Action → Check CT is running, then check service status + Requires: PVE API + SSH +``` ## Access Matrix -Each agent needs explicit credentials to each environment: - -| Environment | Abiba (pi) | Tanko (Hermes) | Mumuni (Hermes) | -|---|---|---|---| +| Resource | Abiba (CT 100) | Tanko (CT 122) | Mumuni (CT 114) | +|----------|---------------|----------------|-----------------| +| **PVE API** (minipve:443) | ✅ API token | ❌ | ❌ | +| **docker-vm** (SSH .7) | ✅ root SSH | ❌ | ❌ | +| **CT 116** (SSH) | ✅ root SSH | ❌ | ❌ | | **Gitea** | ✅ API token | Need to add | Need to add | -| **CT 116 (LiteLLM)** | ✅ SSH key | Need to add | ❌ (has own CTs) | -| **Zulip API** | ✅ Bot token | ✅ Bot token | ✅ Bot token | +| **Zulip API** | ✅ bot token | ✅ bot token | ✅ bot token | | **RA-H OS** | ✅ MCP bridge | ✅ MCP bridge | ✅ MCP bridge | +| **Netbird** (SSH .17) | ✅ root SSH | ❌ | ❌ | | **Authentik** | ❌ No SSH | ❌ No SSH | ❌ No SSH | -## The Gap (What's Missing) +## Priority Matrix for Auto-Remediation -Currently, the contracts CAN: +| Priority | Condition | Action | SLA | +|----------|-----------|--------|-----| +| **P0** | Critical CT/VM stopped | Auto-start via PVE API | 2 min | +| **P0** | LiteLLM/router down | Auto-restart container | 3 min | +| **P1** | Firecrawl/SearXNG down | Auto-restart container | 5 min | +| **P1** | Docker disk > 85% | Auto-prune | 5 min | +| **P2** | Storage usage > 80% | Alert + log | 15 min | +| **P2** | CT resource pressure | Alert + log | 15 min | +| **P3** | Non-critical container down | Alert + log | 30 min | +| **P3** | PBS backup missed | Alert + log | 1h | -| Capability | Works? | How | -|---|---|---| -| Read container state | ✅ SSH to CT 116 | -| Restart containers | ✅ SSH + docker compose | -| Commit to Gitea | ✅ API token | -| Alert via Zulip | ✅ Bot API | -| Log to KG | ✅ MCP bridge | +## Escalation Path -But they CANNOT yet: +``` +1. Auto-remediation ──► 2. Zulip DM to owner ──► 3. RA-H OS relay to agent with access + │ │ + ▼ ▼ + Kwame (Zulip DM) Mumuni (for authentik) + Tanko (for hermes agents) +``` -| Gap | Why | Fix | -|---|---|---| -| Push code to Tanko's CT | No SSH key for CT 112 | Add Abiba's key to CT 112 | -| Restart Hermes gateway | No remote command | Add `hermes gateway restart` via SSH | -| Auto-merge PRs | Needs Gitea write token | Already have it, need to wire | +## Known Gaps -## Reference Implementation - -The `litellm-self-heal` contract already implements most of this pattern: -1. It checks conditions (container health, endpoints) -2. It enforces fixes (docker compose up) -3. It logs results (knowledge graph) -4. It alerts via Zulip DM - -The same pattern extends to any environment where the agent has credentials. +| Gap | What's Needed | +|-----|--------------| +| Authentik SSH | No agent has SSH — Mumuni needs key added | +| Koby/Koonimo stopped | CTs 111/113 are offline — need human intervention | +| Mumuni PVE access | No PVE token — needs `monitoring@pve!mumuni` | +| docker-vm backups | VM 109 isn't running PBS agent — backup status unknown | +| Netbird redundancy | Single point of failure for VPN | From 14bb145300e5c08a6436e6f95b1ff00f40dd7e27 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 27 Jun 2026 13:11:55 +0000 Subject: [PATCH 11/16] =?UTF-8?q?feat:=20expand=20infrastructure-control?= =?UTF-8?q?=20pattern=20=E2=80=94=20Proxmox=205-node=20cluster,=20Docker?= =?UTF-8?q?=203-ecosystem,=20NFS=20storage,=20network=20services?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full discovery and expansion based on PVE API exploration: Proxmox (5 nodes): - minipve (.12): authentik, gitea, mumuni, syslog-api, jitsi - amdpve (.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya - storepve (.6): docker-vm, ra-h-os, PBS, media, zulip - acerpve (.9): llm-gpu, adguard - ocupve (.5): ocu-llm Docker (3 ecosystems, 22 containers): - docker-vm (.7): Firecrawl, SearXNG, Home stack, Audiobookshelf - CT 116: LiteLLM stack (6 containers) - Netbird (.17): VPN controller + dashboard + crowdsec + traefik Pattern now includes: access matrix, per-section checks/remediations, alert routing, escalation chain, quick-start commands, full CT inventory. --- infrastructure-control.prose.md | 599 +++++++++++++++----------------- 1 file changed, 288 insertions(+), 311 deletions(-) diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index 5a698bd..e8319b3 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -2,355 +2,332 @@ kind: pattern name: infrastructure-control description: > - Full infrastructure monitoring and control pattern covering the 5-node - Proxmox cluster, 3 Docker ecosystems (22 containers), storage fabric, - and network services. Declares what agents can observe, what they can - enforce, and where human escalation is required. + Full infrastructure monitoring and control pattern covering the + 5-node Proxmox cluster, 3 Docker ecosystems (22 containers), + NFS storage, and network services. Defines monitors, remediations, + and the access matrix for all environments. --- -## Architecture Overview +# Infrastructure Control Pattern + +## Topology ``` - ┌──────────────────────────────────────────┐ - │ OpenProse Contracts │ - │ (monitor → diagnose → remediate → log) │ - └─────┬──────────┬──────────┬──────────────┘ - │ │ │ - ┌────────────┼──────────┼──────────┼────────────┐ - ▼ ▼ ▼ ▼ ▼ - ┌──────────┐ ┌──────────┐ ┌──────┐ ┌──────┐ ┌──────────┐ - │ Proxmox │ │ Docker │ │Storage│ │Network│ │ Agents │ - │ Cluster │ │Ecosystems│ │Fabric │ │Services│ │(health) │ - │ 5 nodes │ │ 3 hosts │ │NFS+ZFS│ │DNS+Auth│ │6 agents │ - └──────────┘ └──────────┘ └──────┘ └──────┘ └──────────┘ + ┌─────────────────────────┐ + │ OpenProse Contract │ + │ (declares what's true) │ + └──────────┬──────────────┘ + │ + ┌────────────────────┼────────────────────┐ + ▼ ▼ ▼ + ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ + │ Abiba │ │ Tanko │ │ Mumuni │ + │ (pi) │ │ (Hermes) │ │ (Hermes) │ + │ CT 100 │ │ CT 122 │ │ CT 114 │ + └──────┬──────┘ └──────┬───────┘ └──────┬───────┘ + │ │ │ + └──────────────────┼────────────────────┘ + ▼ + ┌──────────────────────────────────────┐ + │ Proxmox Cluster API │ + │ minipve.sysloggh.net:443 │ + │ (monitoring@pve!mumuni token) │ + └────┬──────┬──────┬──────┬──────┬─────┘ + │ │ │ │ │ + ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ + ▼ ▼ ▼ ▼ ▼ + minipve amdpve storepve acerpve ocupve + (.12) (.15) (.6) (.9) (.5) + + ▼ + ┌─────────────────────────────────────────────┐ + │ Docker Ecosystems │ + │ ┌────────────┬────────────┬──────────────┐ │ + │ │ docker-vm │ CT 116 │ Netbird │ │ + │ │ (.7) │ (.116) │ (72.61.0.17) │ │ + │ │ 11 ctrs │ 6 ctrs │ 5 ctrs │ │ + │ └────────────┴────────────┴──────────────┘ │ + └─────────────────────────────────────────────┘ ``` -## Layer 1: Proxmox Cluster +## Section 1: Access Matrix -### Cluster Topology +### API Tokens & Credentials + +| Resource | Auth Method | Credential Source | Status | +|----------|------------|-------------------|--------| +| Proxmox Cluster | PVE API Token | `monitoring@pve!mumuni=...` | ✅ | +| Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ | +| docker-vm (.7) | SSH root | SSH key | ✅ | +| CT 116 (syslog-api) | SSH root | SSH key | ✅ | +| Tanko CT (.122) | SSH jerome | SSH key | ✅ | +| Netbird (.17) | SSH root | SSH key | ✅ | +| Gitea | API token | abiba-bot token | ✅ | +| Zulip | Bot API key | per-bot tokens | ✅ | +| RA-H OS | MCP bridge | port 3100 | ✅ | + +### Reachability Matrix + +| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Netbird (.17) | +|-----------|---------|----------------|--------|-------------|---------------| +| **Abiba** (CT 100) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH | ❌ no Netbird | +| **Tanko** (CT 122) | ❌ | ❌ | ❌ | ✅ | ❌ | +| **docker-vm** (.7) | ❌ | ✅ | ❌ | ❌ | ❌ | + +**Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba. + +## Section 2: Proxmox Cluster — Monitoring + +### Nodes (5) + +| Node | IP | CPU | RAM | VMs/CTs | Role | +|------|----|-----|-----|---------|------| +| minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging | +| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute | +| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat | +| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs | +| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs | + +### Checks (every 5 min) ``` -minipve (.12) ─── 16C/30GB ─── authentik CT 104, gitea CT 110, -│ mumuni CT 114, syslog-api CT 116, -│ jitsi CT 118 -│ -amdpve (.15) ─── 32C/62GB ─── abiba CT 100, kagentz CT 105, -│ tanko CT 112, tdunna CT 111, -│ baggy CT 113, scottdenya CT 115 -│ -storepve (.6) ──── 28C/31GB ─── docker-vm VM 109, ra-h-os CT 106, -│ PBS CT 107, media CT 108, -│ zulip CT 117 -│ -acerpve (.9) ──── 28C/31GB ─── llm-gpu VM 101, adguard CT 102 -│ -ocupve (.5) ──── 12C/14GB ─── ocu-llm VM 103 -``` - -### Maintains (Monitoring) - -```yaml ## Maintains -- cluster.nodes: each node's status (online/offline), CPU%, RAM%, uptime -- cluster.vms: each VM/CT's status (running/stopped), CPU, RAM, disk usage -- cluster.storage: each storage target's usage %, active/inactive -- cluster.ha: high-availability state and fencing status + +- cluster-status: { online_nodes: int, offline_nodes: int, timestamp } +- node-cpu-usage: { node: pct, warnings: [] } +- node-memory-usage: { node: free_pct, warnings: [] } +- node-uptime: { node: seconds, just_rebooted: bool } + +## Checks + +- For each node in [minipve, amdpve, storepve, acerpve, ocupve]: + - GET /api2/json/nodes/{node}/status → check status == "online" + - GET /api2/json/nodes/{node}/status → cpu < 0.80 + - GET /api2/json/nodes/{node}/status → free_mem > 10% + - GET /api2/json/nodes/{node}/status → uptime > 300 (warn if just rebooted) + +- For each CT in inventory (19 total): + - GET /api2/json/cluster/resources → filter by type=lxc + - Warn if status != "running" + +- Storage pools: + - GET /api2/json/nodes/storepve/storage/mediastore → used < 80% + - GET /api2/json/nodes/storepve/storage/pbs-backup → last backup < 48h + +## Remediations + +- Node offline → alert via Zulip DM, escalate after 3x +- CT stopped → `pct start ` via PVE API, verify after 30s +- VM stopped → `qm start ` via PVE API, verify after 60s +- Storage > 80% → warn via Zulip DM +- Storage > 95% → crit via Zulip + relay to maintainer ``` -### Rules (Monitoring) +## Section 3: Docker Ecosystems — Monitoring + +### Ecosystem A: docker-vm (192.168.68.7) + +11 containers across 4 compose stacks: + +| Stack | Path | Containers | +|-------|------|-----------| +| **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis | +| **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey | +| **Home stack** | `/opt/home_stack/` | jdownloader, bentopdf, pulse | +| **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf | + +### Ecosystem B: CT 116 syslog-api (192.168.68.116) + +| Container | Image | Port | +|-----------|-------|------| +| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4001 | +| harness-nginx | nginx:alpine | :80 | +| harness-router | inference-harness-router | :9000 | +| harness-postgres | postgres:16-alpine | :5432 | +| harness-redis | redis:7-alpine | :6379 | +| harness-dashboard | inference-harness-dashboard | :3000 | + +### Ecosystem C: Netbird (72.61.0.17) + +| Container | Image | Role | +|-----------|-------|------| +| netbird-server | netbird | VPN controller | +| netbird-dashboard | netbird UI | Web management | +| netbird-proxy | nginx | TLS termination | +| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF | +| netbird-traefik | traefik | Reverse proxy | + +### Checks (every 60s) ``` -Rule PM-1: Node Offline - Detect → PVE API shows node status != "online" - Action → Alert via Zulip DM with node name and last known uptime - Escalate → If >2 nodes offline, page owner - Requires: PVE API token - -Rule PM-2: Node Resource Pressure - Detect → CPU > 80% or RAM > 85% for 5 consecutive checks - Action → Log to KG, alert via Zulip DM with resource snapshot - Escalate → If sustained >90% for 1h, page owner - Requires: PVE API token - -Rule PM-3: VM/CT Unexpectedly Stopped - Detect → PVE API shows running VM now "stopped" - Action → `pct start ` or `qm start ` via PVE API - Verify → Re-check status after 30s - Escalate → If still stopped after 2 retries, alert owner - Requires: PVE API token with appropriate permissions - -Rule PM-4: Storage Usage Warning - Detect → Storage pool usage > 80% - Action → Log to KG, alert via Zulip DM - Escalate → If >90%, immediate alert (risk of VM freeze) - Requires: PVE API -``` - -### Control (Remediation) - -```yaml -## Enforce -- PM-3 auto-start: start critical VMs/CTs when unexpectedly stopped - - Critical: abiba (100), tanko (112), mumuni (114), syslog-api (116), - docker-vm (109), gitea (110), authentik (104), zulip (117) -- PM-4 storage: if mediastore > 85%, alert with biggest consumers - -## Prohibited -- Never migrate VMs/CTs between nodes automatically -- Never delete storage volumes -- Never modify cluster HA configuration -``` - -## Layer 2: Docker Ecosystems - -### Docker Hosts - -```yaml -Host: docker-vm (192.168.68.7, storepve VM 109) - Resources: 10GB RAM, 4 vCPU, 158GB disk (33% used) - Stacks: - firecrawl: api, rabbitmq, postgres, playwright, redis - searxng: searxng (:8888), valkey - home_stack: jdownloader (:5800), bentopdf (:8989), - pulse (:7655), audiobookshelf (:13378) - Mounts: /media/storage (3.6TB NFS), /media/mediastore (7.3TB NFS) - -Host: syslog-api (192.168.68.116, minipve CT 116) - Resources: 6GB RAM, 2 vCPU, 39GB disk - Stacks: - harness: litellm (:4001), nginx (:80), router (:9000), - postgres, redis, dashboard (:3000) - -Host: netbird (72.61.0.17, external VM) - Stacks: - netbird: server, dashboard, proxy, crowdsec, traefik -``` - -### Maintains (Monitoring) - -```yaml ## Maintains -- docker.hosts: each host's status, container count, resource usage -- docker.containers: each container's status (running/restarting/exited), - uptime, restart count, image version -- docker.stacks: each compose stack's service count and health -- docker.resources: per-host disk usage, Docker system df + +- docker-health: { ecosystem: string, healthy: int, unhealthy: int, total: int } +- container-status: { name: string, status: string, restarts: int, image: string } + +## Checks + +For each Docker host: + - docker ps --format "{{.Names}}" → every required container is running + - docker inspect --format "{{.State.Health.Status}}" → "healthy" + - docker inspect --format "{{.RestartCount}}" → < 3/hour + - df -h / | awk '{print $5}' → usage < 80% + +For docker-vm specifically: + - mountpoint -q /media/storage → NFS mounted + - mountpoint -q /media/mediastore → NFS mounted + - docker compose ls → expected stacks present + +## Remediations + +- Container unhealthy → `docker compose up -d `, wait 10s, re-check +- Container missing → `docker compose up -d` in its stack directory +- Docker daemon down → `systemctl restart docker` +- NFS mount lost → `mount -a`, if fails → alert (storepve issue) +- Disk > 80% → alert via Zulip DM +- Disk > 95% → crit + relay to maintainer ``` -### Rules (Monitoring) +## Section 4: Storage — Monitoring + +### NFS Mounts (docker-vm → storepve) + +| Mount | Export | Capacity | Used | Alert | +|-------|--------|----------|------|-------| +| /media/storage | storepve:/media/storage | 3.6TB | 264GB (8%) | None | +| /media/mediastore | storepve:/media/mediastore | 7.3TB | 5.2TB (75%) | ⚠️ warn at 80% | + +### Proxmox Storage Backends + +| Storage | Type | Content | Active | +|---------|------|---------|--------| +| local | dir | ISO, vztmpl, backup | ✅ | +| local-lvm | lvmthin | images, rootdir | ✅ | +| zfs-vm | nfs | rootdir, images | ✅ | +| mediastore | dir | images, backup, rootdir | ✅ | +| zfs-iso | nfs | vztmpl, iso | ✅ | +| pbs-backup | pbs | backup | ✅ | + +### Checks (every 10 min) ``` -Rule DC-1: Container Not Running - Detect → `docker ps` shows "exited" or "restarting" for critical container - Action → `docker compose up -d ` on the affected stack - Verify → Re-check after 10s - Escalate → If still failed after 3 retries, alert owner - Requires: SSH to docker-vm or CT 116 - -Rule DC-2: Container Restart Looping - Detect → Container restart count > 5 in last hour - Action → `docker logs --tail 50 ` → diagnose → KG log - Escalate → Always alert owner with diagnostic excerpt - Requires: SSH to docker-vm or CT 116 - -Rule DC-3: Docker Host Disk Pressure - Detect → `docker system df` shows < 10% reclaimable or disk > 85% - Action → `docker system prune -af` (safe: removes unused images/containers) - Verify → Re-check disk usage - Escalate → If still >85% after prune, alert owner - Requires: SSH to docker-vm - -Rule DC-4: Stack Health Check Failed - Detect → Container HEALTHCHECK status shows "unhealthy" - Action → `docker compose restart ` - Verify → Re-check health after 15s - Escalate → After 2 retries - Requires: SSH to affected host - -Rule DC-5: Firecrawl API Down - Detect → HTTP GET :3002/health returns non-200 - Action → `docker compose -f /opt/search-stack/firecrawl-source/docker-compose.yaml restart firecrawl-api-1` - Escalate → If still down after 2 retries - Requires: SSH to docker-vm - -Rule DC-6: SearXNG Down - Detect → HTTP GET :8888 returns non-200 - Action → `docker compose -f /opt/search-stack/searxng/docker-compose.yml restart searxng` - Escalate → If still down after 2 retries - Requires: SSH to docker-vm -``` - -### Control (Remediation) - -```yaml -## Enforce -- DC-1 auto-restart: restart critical containers when exited -- DC-3 auto-prune: prune unused Docker artifacts weekly -- DC-4/5/6: restart unhealthy stacks with verification - -## Prohibited -- Never remove volumes (data loss risk) -- Never modify docker-compose files remotely -- Never update container images without owner approval -``` - -## Layer 3: Storage Fabric - -### Storage Topology - -``` -storepve (.6) ── NFS ──► docker-vm: /media/storage (3.6TB, 8% used) - └── audiobookshelf, JDownloader downloads -storepve (.6) ── NFS ──► docker-vm: /media/mediastore (7.3TB, 75% used) - └── media library, configs -storepve ── PBS CT 107 ──► backup target for all VMs/CTs -``` - -### Maintains - -```yaml ## Maintains -- storage.nfs_mounts: each mount's availability, usage %, inode count -- storage.pbs: last successful backup timestamp, backup schedule compliance -- storage.mediastore_growth: weekly growth rate (high-use volume) + +- storage-usage: { mount: string, used_pct: float, growth_rate: float } +- backup-status: { last_success: timestamp, age_hours: int } +- snapshot-status: { pool: string, newest_age_hours: int } + +## Specific Alerts + +- mediastore growth rate > 10GB/day → warn (check what's writing) +- mediastore > 90% → crit (only 730GB remaining) +- No PBS backup in 48h → fail ``` -### Rules +## Section 5: Network Services — Monitoring + +| Service | Domain | Status | +|---------|--------|--------| +| Proxmox API | minipve.sysloggh.net:443 | ✅ | +| Authentik | auth.sysloggh.net:443 | ✅ | +| Gitea | git.sysloggh.net:443 | ✅ | +| Zulip | chat.sysloggh.net:443 | ✅ | +| LiteLLM | litellm.sysloggh.net:443 | ✅ | +| Pulse | pulse.sysloggh.net:443 | ✅ | +| SearXNG | searxng.sysloggh.net:8888 | ✅ | +| Firecrawl | firecrawl.sysloggh.net:3002 | ✅ | + +### Checks (every 2 min) ``` -Rule ST-1: NFS Mount Unreachable - Detect → `df` or `mountpoint -q` fails for NFS path - Action → Check storepve NFS server status via PVE API - Escalate → Immediate alert — containers may be read-only or crashing - Requires: PVE API + SSH to docker-vm - -Rule ST-2: Mediadisk Growth Alert - Detect → mediastore (7.3TB, 75%) growing > 50GB/week - Action → Identify top consumers: `du -sh /media/mediastore/* | sort -rh | head 10` - Log → KG node with growth trend - Escalate → If projected to exceed 90% within 30 days - -Rule ST-3: PBS Backup Compliance - Detect → Any CT/VM without a backup in the last 48h - Action → Log to KG, alert via Zulip DM - Requires: PVE API to check PBS tasks -``` - -## Layer 4: Network Services - -### Service Map - -| Service | Host | IP | Port | Purpose | -|---------|------|----|------|---------| -| adguard | CT 102 | .102 | 53, 80, 443 | DNS + DHCP | -| authentik | CT 104 | .104 | 443 | OIDC auth provider | -| gitea | CT 110 | .110 | 3000, 22 | Git hosting | -| zulip | CT 117 | .117 | 443 | Team chat | -| netbird | 72.61.0.17 | .17 | 443, 8006 | VPN controller | -| minipve | .12 | .12 | 443, 8006 | PVE API endpoint | -| PBS | CT 107 | .107 | 8007 | Proxmox Backup Server | - -### Maintains - -```yaml ## Maintains -- network.dns_resolution: critical hostnames resolve correctly -- network.auth_endpoints: authentik OIDC login flow works -- network.external_access: public endpoints respond (litellm, git, chat, auth, pulse) + +- dns-resolution: { services: [{name, resolves}] } +- ssl-expiry: { services: [{name, days_remaining}] } +- endpoint-reachability: { services: [{name, http_code}] } ``` -### Rules +## Section 6: Alert Routing & Escalation + +### Severity Levels + +| Severity | Channel | Format | Rate Limit | +|----------|---------|--------|-----------| +| **warn** | Zulip DM to owner | "⚠️ {check}: {detail}" | 1x/check/hour | +| **crit** | Zulip DM + #agent-hub | "🚨 {check}: {detail}" | Immediate | +| **escalated** | Zulip DM + relay to maintainer | "🔥 {check}: {detail}" | Immediate | + +### Escalation Chain ``` -Rule NW-1: DNS Failure - Detect → `host` or `nslookup` fails for known hostnames - Action → Check adguard CT 102 status via PVE API - Escalate → If authentik/gitea unreachable due to DNS → page owner - Requires: PVE API + dig/nslookup +Container unhealthy (3 consecutive failures) + → Alert owner + → 15 min no response → Relay to maintainer -Rule NW-2: Auth Endpoint Down - Detect → `GET https://auth.sysloggh.net` returns non-200 - Action → Check authentik CT 104 via PVE API → start if stopped - Escalate → Can't fix: relay to Mumuni (has authentik SSH access) - Requires: PVE API +Node offline + → Alert owner + → 5 min → Attempt restart via PVE API + → 2 failures → Escalate -Rule NW-3: Public Endpoint Unreachable - Detect → Any public URL fails (litellm, git, chat, pulse) - Action → Check container + nginx health on relevant host - Requires: SSH to affected host + curl +Storage > 95% + → Alert owner + maintainer relay + → Immediate action required + +SSL cert < 7 days + → Alert owner daily + → 3 days out → Escalate + +ZFS pool degraded + → Cannot auto-fix → Escalate immediately ``` -## Layer 5: Agent Health +## Appendix A: Quick Health Commands -### Agent Registry +```bash +# Full cluster status +PVE="https://minipve.sysloggh.net" +AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3" +curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH" -| Agent | Platform | CT | Status Endpoint | Last Seen | -|-------|----------|----|----------------|-----------| -| abiba | pi | 100 | :9200/health | live | -| tanko | Hermes | 122 | (via Gateway) | live | -| mumuni | Hermes | 114 | (via Gateway) | live | -| kagentz | Agent Zero | 105 | (via PVE) | running | -| koby | Hermes | 111 | (via PVE) | stopped | -| koonimo | Hermes | 113 | (via PVE) | stopped | +# Docker health from Abiba +ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'" +ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'" -### Rules - -``` -Rule AG-1: Agent VM/CT Stopped - Detect → PVE API shows agent CT as "stopped" - Action → `pct start ` via PVE API - Verify → Re-check after 15s - Escalate → If persistent, alert owner - Requires: PVE API - -Rule AG-2: Agent Not Responding - Detect → Health endpoint non-200 or Gateway disconnected - Action → Check CT is running, then check service status - Requires: PVE API + SSH +# Storage check +ssh root@192.168.68.7 "df -h /media/storage /media/mediastore" ``` -## Access Matrix +## Appendix B: CT Inventory -| Resource | Abiba (CT 100) | Tanko (CT 122) | Mumuni (CT 114) | -|----------|---------------|----------------|-----------------| -| **PVE API** (minipve:443) | ✅ API token | ❌ | ❌ | -| **docker-vm** (SSH .7) | ✅ root SSH | ❌ | ❌ | -| **CT 116** (SSH) | ✅ root SSH | ❌ | ❌ | -| **Gitea** | ✅ API token | Need to add | Need to add | -| **Zulip API** | ✅ bot token | ✅ bot token | ✅ bot token | -| **RA-H OS** | ✅ MCP bridge | ✅ MCP bridge | ✅ MCP bridge | -| **Netbird** (SSH .17) | ✅ root SSH | ❌ | ❌ | -| **Authentik** | ❌ No SSH | ❌ No SSH | ❌ No SSH | +| CT | Name | Node | IP | Role | Agent | +|----|------|------|----|------|-------| +| 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi | +| 101 | llm-gpu | acerpve | — | GPU VM | ❌ | +| 102 | adguard | acerpve | — | DNS | ❌ | +| 103 | ocu-llm | ocupve | — | GPU VLM | ❌ | +| 104 | authentik | minipve | .11 | OIDC | ❌ | +| 105 | kagentz | amdpve | — | Agent Zero | ✅ | +| 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP | +| 107 | pbs | storepve | — | Backups | ❌ | +| 108 | media | storepve | — | Media | ❌ | +| 109 | docker-vm | storepve | .7 | Docker host | ❌ | +| 110 | gitea | minipve | — | Git | ❌ | +| 111 | tdunna | amdpve | — | ? | ❌ | +| 112 | tanko | amdpve | .122 | Hermes agent | ✅ | +| 113 | baggy | amdpve | — | ? | ❌ | +| 114 | mumuni | minipve | — | Hermes agent | ✅ | +| 115 | scottdenya | amdpve | — | ? | ❌ | +| 116 | syslog-api | minipve | .116 | LiteLLM stack | ❌ | +| 117 | zulip | storepve | — | Chat | ❌ | +| 118 | jitsi | minipve | — | Video | ❌ | -## Priority Matrix for Auto-Remediation +## Appendix C: Docker Compose Files Location -| Priority | Condition | Action | SLA | -|----------|-----------|--------|-----| -| **P0** | Critical CT/VM stopped | Auto-start via PVE API | 2 min | -| **P0** | LiteLLM/router down | Auto-restart container | 3 min | -| **P1** | Firecrawl/SearXNG down | Auto-restart container | 5 min | -| **P1** | Docker disk > 85% | Auto-prune | 5 min | -| **P2** | Storage usage > 80% | Alert + log | 15 min | -| **P2** | CT resource pressure | Alert + log | 15 min | -| **P3** | Non-critical container down | Alert + log | 30 min | -| **P3** | PBS backup missed | Alert + log | 1h | - -## Escalation Path - -``` -1. Auto-remediation ──► 2. Zulip DM to owner ──► 3. RA-H OS relay to agent with access - │ │ - ▼ ▼ - Kwame (Zulip DM) Mumuni (for authentik) - Tanko (for hermes agents) -``` - -## Known Gaps - -| Gap | What's Needed | -|-----|--------------| -| Authentik SSH | No agent has SSH — Mumuni needs key added | -| Koby/Koonimo stopped | CTs 111/113 are offline — need human intervention | -| Mumuni PVE access | No PVE token — needs `monitoring@pve!mumuni` | -| docker-vm backups | VM 109 isn't running PBS agent — backup status unknown | -| Netbird redundancy | Single point of failure for VPN | +| Host | Stack | Compose File | +|------|-------|-------------| +| docker-vm (.7) | Firecrawl | `/opt/search-stack/firecrawl-source/docker-compose.yaml` | +| docker-vm (.7) | SearXNG | `/opt/search-stack/searxng/docker-compose.yml` | +| docker-vm (.7) | Home stack | `/opt/home_stack/docker-compose.yml` | +| docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` | +| CT 116 (.116) | LiteLLM | `/root/docker-compose-litellm.yml` | +| Netbird (.17) | Netbird | Docker run (not compose) | From c59afa7f1a34b2f6fc66254a140438745f46c4c4 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 27 Jun 2026 19:22:44 +0000 Subject: [PATCH 12/16] =?UTF-8?q?feat(zulip-health):=20v2=20=E2=80=94=20mu?= =?UTF-8?q?lti-platform=20monitor=20for=20pi,=20Hermes,=20Agent=20Zero?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers all 4 Zulip agents across 3 platforms with checks, auto-heal, and consolidated action matrix. --- zulip-health.prose.md | 235 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 zulip-health.prose.md diff --git a/zulip-health.prose.md b/zulip-health.prose.md new file mode 100644 index 0000000..715f682 --- /dev/null +++ b/zulip-health.prose.md @@ -0,0 +1,235 @@ +--- +kind: contract +name: zulip-health +title: Zulip Mesh Health Monitor — Multi-Platform +version: 2.0.0 +agent: abiba +triggers: + - on startup + - every 15 minutes while running + - on zulip-status command +--- + +# Zulip Mesh Health Monitor — Multi-Platform + +Monitors ALL Zulip-connected agents across three platforms. +Runs every 15 minutes in the background. Also triggers on session start. + +## Platform Overview + +| Platform | Agents | Bot | Adapter | Health Check | +|----------|--------|-----|---------|-------------| +| **pi** | Abiba (CT 100) | abiba-bot | `/root/.pi/agent/extensions/zulip/index.js` | `:9200/health` | +| **Hermes** | Tanko (CT 122), Mumuni (CT 123) | tanko-bot, mumuni-bot | `~/.hermes/plugins/platforms/zulip/` | `gateway_state.json` | +| **Agent Zero** | kagentz (CT 105) | kagentz-bot | Docker container, `/a0/usr/kagentz-zulip/` | A2A endpoint `:8001` | + +## Phase 1: Zulip Server Liveness (All Platforms) + +```bash +curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \ + -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' +``` + +Expected: `200`. Anything else → Server issue, alert maintainer. + +--- + +## Platform A: pi (Abiba — CT 100) + +### A1: Health Endpoint + +```js +const health = await fetch("http://localhost:9200/health").then(r => r.json()); +``` + +| Field | Healthy | Critical | +|-------|---------|----------| +| `connected` | `true` | `false` | +| `last_error` | `null` | non-null string | +| `retry_count` | 0-2 | 3+ | +| `queue_id` | non-null string | `null` | + +### A2: PM2 Process + +```bash +pm2 show abiba-zulip --no-color 2>/dev/null +``` + +Check: `status=online`, `restarts < 10/h`, `uptime > 60s` + +### A3: Echo Loop Detection + +```bash +grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5 +``` + +> 100 skipped in 15min → 🟢 Info only (echo loop prevention working) + +### A4: Response Delivery + +```bash +grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20 +``` + +> 50% fail rate → 🔴 Critical — check editMessage API + +### Actions + +| Condition | Action | +|-----------|--------| +| `connected: false` | `pm2 restart abiba-zulip` | +| `retry_count >= 3` | `pm2 restart abiba-zulip` | +| `last_error` set | Log and monitor | +| Crash loop >10/h | Alert user | + +--- + +## Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123) + +### B1: Gateway State + +```bash +ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json" +ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json" +``` + +Check `platforms.zulip.state`: + +| Value | Meaning | Action | +|-------|---------|--------| +| `"connected"` | ✅ Healthy | None | +| `"disconnected"` | ❌ Disconnected | Check `last_error` | +| `"error"` | ❌ Error | Check `error_message`, restart gateway | +| missing | ❌ Not installed | Run deploy scripts | + +### B2: Agent Process + +```bash +ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep" +``` + +Gateway PID should exist and uptime > 60s. + +### B3: Heartbeat Verification + +```bash +ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" +``` + +Expected: recent heartbeat (within 5 min) showing `polls=N` incrementing. +If silence > 300s → 🟡 Warning (queue may be stuck). +If silence > 600s → 🔴 Critical (queue expired, adapter needs restart). + +### B4: Response Delivery + +```bash +ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10" +``` + +> 50% fail rate → 🔴 Critical + +### Actions + +| Condition | Action | +|-----------|--------| +| `zulip.state != "connected"` | `ssh root@ "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | +| No heartbeat in 10min | `ssh root@ "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | +| `Failed to finalize` > 50% | Check PATCH API, Zulip server | +| Response empty/short | Check A2A endpoint / LiteLLM model | + +--- + +## Platform C: Agent Zero (kagentz — CT 105) + +### C1: A2A Server Health + +```bash +ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json" +``` + +Expected: `{"name":"kagentz",...}` — JSON response with agent identity. +Connection refused → A2A server is down. + +### C2: Adapter Process + +```bash +ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep" +``` + +Adapter should be running. If missing, restart. + +### C3: Heartbeat & Queue + +```bash +ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3" +``` + +Check: +- `processed=N` incrementing when DMs arrive +- `silence < 600s` (queue expiry timeout) +- `reconnects` — should be 0 under normal operation + +### C4: A2A Response Verification + +```bash +# Test A2A sends a task +ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \ + -H 'Content-Type: application/json' \ + -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'" + +# Expected: task ID returned with "working" status +# Then poll for completion with tasks/get +``` + +### Actions + +| Condition | Action | +|-----------|--------| +| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` | +| Adapter process missing | `docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"` | +| Silence > 600s (queue expiry) | Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID) | +| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` | + +--- + +## Global Checks + +### Zulip Server + +```bash +curl -s https://chat.sysloggh.net/api/v1/server_settings \ + -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}" +``` + +Expected: `200` + +### Cross-Agent Echo Loop Detection + +Check each agent's log for excessive bot-to-bot chatter: +- Abiba: `Skipped.*bot msgs` count +- Tanko/Mumuni: Check for repeated DM exchanges between bots +- kagentz: Check adapter log for bot DMs being processed + +If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning. + +--- + +## Consolidated Action Matrix + +| Condition | Severity | Action | +|-----------|----------|--------| +| Zulip server not 200 | 🔴 Critical | Alert maintainer | +| All agents silent | 🔴 Critical | Zulip server likely down | +| pi: connected false | 🔴 Critical | `pm2 restart abiba-zulip` | +| pi: edit fail >50% | 🔴 Critical | Check editMessage API | +| Hermes: zulip disconnected | 🔴 Critical | Restart gateway | +| Hermes: no heartbeat 10min | 🔴 Critical | Restart gateway | +| Az: A2A server down | 🔴 Critical | Restart inside container | +| Az: adapter down | 🔴 Critical | Restart adapter | +| Az: silence >600s | 🟡 Warning | Queue expired (auto-recover) | +| Echo loop detected | 🟢 Info | Auto-mitigated (bot filtering) | + +## Logging + +All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps. +Critical alerts sent as relay messages to user. From 90e074dbcc83fa9cfa4e08d87adbf667478635a9 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 27 Jun 2026 19:22:53 +0000 Subject: [PATCH 13/16] =?UTF-8?q?feat(zulip-health):=20monitor=20script=20?= =?UTF-8?q?+=20infra=20report=20=E2=80=94=20cron-ready=20implementations?= =?UTF-8?q?=20of=20prose=20contracts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/daily-infra-report.py | 708 ++++++++++++++++++++++++++++++++++ scripts/zulip-monitor.sh | 156 ++++++++ 2 files changed, 864 insertions(+) create mode 100755 scripts/daily-infra-report.py create mode 100755 scripts/zulip-monitor.sh diff --git a/scripts/daily-infra-report.py b/scripts/daily-infra-report.py new file mode 100755 index 0000000..a4802c1 --- /dev/null +++ b/scripts/daily-infra-report.py @@ -0,0 +1,708 @@ +#!/usr/bin/env python3 +""" +/root/scripts/daily-infra-report.py — v2.0.0 +Daily infrastructure dashboard emailed to jerome@sysloggh.com +Runs at 6:00 AM daily via cron. +Integrated with prose contracts: infrastructure-control, litellm-health, zulip-health. + +Usage: + python3 daily-infra-report.py # Generate and email report + python3 daily-infra-report.py --test-email # Send a test email for review + python3 daily-infra-report.py --json # Output raw JSON to stdout +""" + +import smtplib, json, subprocess, os, sys, datetime, re +from email.mime.text import MIMEText +from email.mime.multipart import MIMEMultipart + +PVE = "https://minipve.sysloggh.net" +AUTH = "Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3" + +# ── Shared credentials —─ + +ZULIP_SITE = "https://chat.sysloggh.net" +ZULIP_EMAIL = "abiba-bot@chat.sysloggh.net" +ZULIP_KEY = "cKTDMZAPW08dk3zl05sStzO7HRztzyn8" +ZULIP_AUTH = f"{ZULIP_EMAIL}:{ZULIP_KEY}" + +LITELLM_PUBLIC = "https://litellm.sysloggh.net" +LITELLM_BACKEND = "192.168.68.116" +AUTH_HOST = "192.168.68.11" + +SYNTHETIC_API_KEY = "sk-U_ydi3B-wfGU-_xESkoU1Q" + +NOW = datetime.datetime.now() +DATE_STR = NOW.strftime("%Y-%m-%d") +TIME_STR = NOW.strftime("%Y-%m-%d %H:%M UTC") + +# ── Helpers ── + +def pve_get(path): + cmd = f'curl -sfk --connect-timeout 10 "{PVE}{path}" -H "{AUTH}"' + try: + return json.loads(subprocess.check_output(cmd, shell=True))["data"] + except: return [] + +def ssh(host, cmd): + try: + return subprocess.check_output( + f'ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 root@{host} "{cmd}"', + shell=True, stderr=subprocess.DEVNULL).decode().strip() + except: return "" + +def ssh_jerome(host, cmd): + try: + return subprocess.check_output( + f'ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 jerome@{host} "{cmd}"', + shell=True, stderr=subprocess.DEVNULL).decode().strip() + except: return "" + +def http_get(url, auth=None, timeout=10): + try: + cmd = f'curl -sfk --connect-timeout {timeout} -o /dev/null -w "%{{http_code}}" "{url}"' + if auth: + cmd = cmd.replace('"', '\\"') + cmd = f'curl -sfk --connect-timeout {timeout} -u "{auth}" -o /dev/null -w "%{{http_code}}" "{url}"' + r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout+2) + return r.stdout.strip() or "000" + except: + return "timeout" + +def http_get_body(url, auth=None, timeout=10): + try: + cmd = f'curl -sfk --connect-timeout {timeout} "{url}"' + if auth: + cmd = f'curl -sfk --connect-timeout {timeout} -u "{auth}" "{url}"' + r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout+2) + if r.returncode == 0: + return r.stdout.strip() + return "" + except: + return "" + +def count_in_log(filepath, pattern, minutes=15): + """Count occurrences of a pattern in a log file within the last N minutes.""" + try: + since = (NOW - datetime.timedelta(minutes=minutes)).strftime("%Y-%m-%d %H:%M") + cmd = f"""grep -a "{pattern}" {filepath} 2>/dev/null | awk '$0 >= "{since}"' | wc -l""" + count = subprocess.check_output(cmd, shell=True).decode().strip() + return int(count) + except: + return 0 + +# ── Data Collection ── + +def collect(): + report = {} + + # ── Proxmox Nodes ── + nodes = pve_get("/api2/json/nodes") + report["nodes"] = {n["node"]: { + "cpu_pct": round(n.get('cpu',0)*100, 1), + "ram": f"{n.get('mem',0)//1024//1024}/{n.get('maxmem',0)//1024//1024}MB", + "ram_pct": round(n.get('mem',0)/n.get('maxmem',1)*100, 0), + "disk": f"{n.get('disk',0)//1024//1024//1024}/{n.get('maxdisk',0)//1024//1024//1024}GB", + "disk_pct": round(n.get('disk',0)/n.get('maxdisk',1)*100, 0), + "uptime_h": n.get('uptime',0)//3600, + "status": n["status"] + } for n in nodes} + report["node_count"] = len(nodes) + report["nodes_online"] = sum(1 for n in nodes if n["status"] == "online") + + # ── VMs/CTs ── + resources = pve_get("/api2/json/cluster/resources") + vms = [r for r in resources if r.get("type") in ("qemu","lxc")] + report["total_vms"] = len(vms) + report["running_vms"] = sum(1 for v in vms if v.get("status") == "running") + stopped = [v for v in vms if v.get("status") != "running"] + report["stopped_vms"] = [f"{v['type']} {v['vmid']} {v.get('name','?')}" for v in stopped] + + report["vms_by_node"] = {} + for n in ["amdpve", "minipve", "storepve", "acerpve", "ocupve"]: + node_vms = [v for v in vms if v.get("node") == n] + node_vms.sort(key=lambda x: int(x.get("vmid",0))) + report["vms_by_node"][n] = [{ + "type": v.get("type"), "vmid": v.get("vmid"), "name": v.get("name","?"), + "status": v.get("status"), + "ram": f"{v.get('mem',0)//1024//1024}/{v.get('maxmem',0)//1024//1024}MB", + "disk": f"{v.get('disk',0)//1024//1024//1024}/{v.get('maxdisk',0)//1024//1024//1024}GB", + "cpu": v.get("cpus", v.get("maxcpu","?")), + } for v in node_vms] + + # ── Storage ── + storages = pve_get("/api2/json/nodes/storepve/storage") + report["storage"] = [] + for s in storages: + total = s.get("total",0) or 1 + used = s.get("used",0) + pct = used/total*100 + report["storage"].append({ + "name": s.get("storage","?"), "used": f"{used//1024//1024//1024}GB", + "total": f"{total//1024//1024//1024}GB", "pct": round(pct, 0), "type": s.get("type","") + }) + + # ── Docker: docker-vm (.7) ── + docker_raw = ssh("192.168.68.7", "docker ps --format '{{.Names}}|{{.Status}}|{{.Image}}'") + containers = [] + for line in docker_raw.strip().split("\n"): + if "|" in line: + parts = line.split("|", 2) + containers.append({"name": parts[0], "status": parts[1], "image": parts[2] if len(parts) > 2 else ""}) + report["docker_vm"] = {"total": len(containers), "running": sum(1 for c in containers if "Up" in c["status"]), + "unhealthy": [c for c in containers if "unhealthy" in c["status"]], "containers": containers} + report["docker_vm"]["reclaimable"] = ssh("192.168.68.7", "docker system df 2>/dev/null | tail -1 | awk '{print $5}'") + report["docker_vm"]["disk_used"] = ssh("192.168.68.7", "df -h / | tail -1 | awk '{print $5}'") + + # ── Docker: CT 116 (syslog-api — LiteLLM stack) ── + docker2_raw = ssh(LITELLM_BACKEND, "docker ps --format '{{.Names}}|{{.Status}}|{{.Image}}'") + containers2 = [] + for line in docker2_raw.strip().split("\n"): + if "|" in line: + parts = line.split("|", 2) + containers2.append({"name": parts[0], "status": parts[1], "image": parts[2] if len(parts) > 2 else ""}) + report["docker_syslog"] = {"total": len(containers2), "running": sum(1 for c in containers2 if "Up" in c["status"]), + "containers": containers2} + + # ── Docker: Netbird (72.61.0.17) ── + docker3_raw = ssh("72.61.0.17", "docker ps --format '{{.Names}}|{{.Status}}|{{.Image}}'") + containers3 = [] + for line in docker3_raw.strip().split("\n"): + if "|" in line: + parts = line.split("|", 2) + containers3.append({"name": parts[0], "status": parts[1], "image": parts[2] if len(parts) > 2 else ""}) + report["docker_netbird"] = {"total": len(containers3), "running": sum(1 for c in containers3 if "Up" in c["status"]), + "containers": containers3} + + # ── Public Endpoints ── + endpoints = [ + ("LiteLLM Admin UI", f"{LITELLM_PUBLIC}/ui/"), + ("LiteLLM Swagger", f"{LITELLM_PUBLIC}/docs"), + ("LiteLLM ReDoc", f"{LITELLM_PUBLIC}/redoc"), + ("LiteLLM OpenAPI", f"{LITELLM_PUBLIC}/openapi.json"), + ("Gitea", "https://git.sysloggh.net"), + ("Authentik", "https://auth.sysloggh.net"), + ("Zulip", "https://chat.sysloggh.net"), + ("Pulse", "https://pulse.sysloggh.net"), + ("Proxmox", "https://minipve.sysloggh.net"), + ("SearXNG", "http://192.168.68.7:8888"), + ("Firecrawl", "http://192.168.68.7:3002/health"), + ] + report["endpoints"] = [] + for name, url in endpoints: + code = http_get(url) + report["endpoints"].append({"name": name, "code": code}) + + # ── LiteLLM Specific Checks (from litellm-health prose contract) ── + report["litellm"] = {"checks": []} + + # Check 1: LiteLLM aggregate health endpoint (router binds to 127.0.0.1, check via SSH) + health_unified = ssh(LITELLM_BACKEND, "curl -sf http://127.0.0.1:9000/health/unified -o /dev/null -w '%{http_code}' 2>/dev/null") + report["litellm"]["health_unified"] = health_unified or "000" + report["litellm"]["checks"].append({"name": "unified-health", "status": "pass" if health_unified == "200" else "fail", "code": health_unified or "000"}) + + # Check 2: Nginx-proxied internal endpoints + for path, name in [("/litellm/ui/", "nginx-ui"), ("/litellm/docs", "nginx-docs")]: + code = http_get(f"http://{LITELLM_BACKEND}{path}") + report["litellm"]["checks"].append({"name": name, "status": "pass" if code == "200" else "fail", "code": code}) + + # Check 3: Docker container health for LiteLLM stack + expected_containers = ["harness-litellm", "harness-nginx", "harness-router", + "harness-postgres", "harness-redis", "harness-dashboard"] + actual_names = [c["name"] for c in containers2] + report["litellm"]["expected_containers"] = expected_containers + report["litellm"]["missing_containers"] = [e for e in expected_containers if e not in actual_names] + report["litellm"]["checks"].append({ + "name": "container-health", + "status": "pass" if not report["litellm"]["missing_containers"] else "fail", + "missing": report["litellm"]["missing_containers"], + "healthy": sum(1 for c in containers2 if "healthy" in c.get("status","")), + "total": len(expected_containers) + }) + + # Check 4: OIDC Auth endpoint + auth_code = http_get(f"https://auth.sysloggh.net") + report["litellm"]["auth_status"] = auth_code + report["litellm"]["checks"].append({"name": "oidc-auth", "status": "pass" if auth_code in ("200","302") else "fail", "code": auth_code}) + + # Check 5: Synthetic API call through LiteLLM + api_check = http_get(f"{LITELLM_PUBLIC}/v1/models", auth=SYNTHETIC_API_KEY) + report["litellm"]["api_models"] = api_check + report["litellm"]["checks"].append({"name": "api-endpoint", "status": "pass" if api_check == "200" else "fail", "code": api_check}) + + # ── NFS Mounts ── + nfs = ssh("192.168.68.7", "df -h /media/storage /media/mediastore 2>/dev/null | tail -n +2") + report["nfs"] = [] + for line in nfs.strip().split("\n"): + parts = line.split() + if len(parts) >= 6: + report["nfs"].append({"mount": parts[5], "size": parts[1], "used": parts[2], "pct": parts[4], "avail": parts[3]}) + + # ── Zulip Extension Health (from zulip-health prose contract) ── + report["zulip_ext"] = {} + + # Phase 1: Health endpoint + health_body = http_get_body("http://localhost:9200/health") + zulip_health = {} + try: + zulip_health = json.loads(health_body) if health_body else {} + except: + zulip_health = {} + report["zulip_ext"]["connected"] = zulip_health.get("connected", False) + report["zulip_ext"]["queue_id"] = zulip_health.get("queue_id") + report["zulip_ext"]["last_error"] = zulip_health.get("last_error") + report["zulip_ext"]["messages_processed"] = zulip_health.get("messages_processed", 0) + report["zulip_ext"]["retry_count"] = zulip_health.get("retry_count", 0) + + # Phase 2: PM2 process check + pm2_raw = subprocess.check_output( + "pm2 show abiba-zulip --no-color 2>/dev/null | grep -E 'status|restarts|uptime'", + shell=True).decode().strip() + pm2 = {} + for line in pm2_raw.split("\n"): + if "│" in line: + parts = line.split("│") + if len(parts) >= 3: + key = parts[1].strip() + val = parts[2].strip() + pm2[key] = val + report["zulip_ext"]["pm2"] = pm2 + report["zulip_ext"]["pm2_healthy"] = pm2.get("status") == "online" + + # Phase 3: Echo loop detection + skipped_count = count_in_log("/root/.pm2/logs/abiba-zulip-out.log", "Skipped.*bot msgs", 15) + report["zulip_ext"]["bot_skipped_15min"] = skipped_count + + # Phase 4: Response delivery stats + finalized = count_in_log("/root/.pm2/logs/abiba-zulip-out.log", "Finalized", 60) + failed = count_in_log("/root/.pm2/logs/abiba-zulip-out.log", "Failed to finalize", 60) + report["zulip_ext"]["finalized_1h"] = finalized + report["zulip_ext"]["failed_finalize_1h"] = failed + report["zulip_ext"]["finalize_fail_pct"] = round(failed / (finalized + failed) * 100, 0) if (finalized + failed) > 0 else 0 + + # Phase 5: Zulip server liveness + zulip_server_code = http_get(f"{ZULIP_SITE}/api/v1/server_settings", auth=ZULIP_AUTH) + report["zulip_ext"]["server_status"] = zulip_server_code + + # ── Agent Status ── + report["agents"] = {} + + # Abiba (pi) + report["agents"]["abiba"] = { + "platform": "pi", "ct": 100, "ip": "192.168.68.24", + "zulip_connected": zulip_health.get("connected", False), + "zulip_processed": zulip_health.get("messages_processed", 0), + "pm2_status": pm2.get("status", "unknown"), + "pm2_restarts": pm2.get("restarts", "?"), + "pm2_uptime": pm2.get("uptime", "?"), + } + + # Tanko (CT 122) + tanko_state = ssh_jerome("192.168.68.122", "cat ~/.hermes/gateway_state.json 2>/dev/null") + tanko_data = {} + try: + tanko_data = json.loads(tanko_state) if tanko_state else {} + except: + tanko_data = {} + platforms = tanko_data.get("platforms", {}) + report["agents"]["tanko"] = { + "platform": "hermes", "ct": 112, "ip": "192.168.68.122", + "gateway_state": tanko_data.get("gateway_state", "unknown"), + "zulip_state": platforms.get("zulip", {}).get("state", "unknown"), + "telegram_state": platforms.get("telegram", {}).get("state", "unknown"), + "gateway_pid": tanko_data.get("pid"), + "updated_at": tanko_data.get("updated_at"), + } + + # Mumuni (CT 114, IP 192.168.68.123) + mumuni_state = ssh("192.168.68.123", "cat ~/.hermes/gateway_state.json 2>/dev/null") + mumuni_data = {} + try: + mumuni_data = json.loads(mumuni_state) if mumuni_state else {} + except: + mumuni_data = {} + mumuni_platforms = mumuni_data.get("platforms", {}) + report["agents"]["mumuni"] = { + "platform": "hermes", "ct": 114, "ip": "192.168.68.123", + "gateway_state": mumuni_data.get("gateway_state", "unknown"), + "telegram_state": mumuni_platforms.get("telegram", {}).get("state", "unknown"), + "zulip_state": mumuni_platforms.get("zulip", {}).get("state", "not_installed"), + "email_state": mumuni_platforms.get("email", {}).get("state", "unknown"), + "hermes_version": "", + } + # Get Hermes version + ver = ssh("192.168.68.123", "hermes --version 2>/dev/null | head -1") + if ver: + report["agents"]["mumuni"]["hermes_version"] = ver.split("·")[0].replace("Hermes Agent ","").strip() + + return report + + +# ── HTML Dashboard ── + +def build_html(r): + issues = [] + + # Proxmox issues + if r["nodes_online"] < r["node_count"]: + issues.append(f"🔴 {r['node_count'] - r['nodes_online']} Proxmox node(s) offline") + if r["stopped_vms"]: + issues.append(f"🔴 {len(r['stopped_vms'])} VM(s)/CT(s) stopped — {', '.join(r['stopped_vms'][:3])}") + + # Endpoint issues + for ep in r["endpoints"]: + if ep["code"] in ("000", "timeout"): + issues.append(f"🔴 {ep['name']} — unreachable") + elif ep["code"] >= "500": + issues.append(f"🟡 {ep['name']} — HTTP {ep['code']}") + + # Docker issues + for c in r["docker_vm"].get("unhealthy", []): + issues.append(f"🟡 docker-vm: {c['name']} — unhealthy") + + # Small cluster mode warning + if r["docker_vm"].get("total", 0) == 0: + issues.append("🟡 docker-vm: no containers reported — host may be down") + + # LiteLLM issues + for check in r.get("litellm", {}).get("checks", []): + if check["status"] == "fail": + issues.append(f"🔴 LiteLLM: {check['name']} — {'missing: ' + ', '.join(check.get('missing',[])) if check.get('missing') else 'HTTP ' + check.get('code','?')}") + + # Zulip extension issues + z = r.get("zulip_ext", {}) + if not z.get("connected"): + issues.append("🔴 Zulip extension: disconnected") + if z.get("last_error"): + issues.append(f"🟡 Zulip extension: {z['last_error'][:80]}") + if z.get("retry_count", 0) >= 3: + issues.append(f"🟡 Zulip extension: {z['retry_count']} retries") + if z.get("finalize_fail_pct", 0) > 50: + issues.append(f"🔴 Zulip extension: {z['finalize_fail_pct']:.0f}% edit failures") + if z.get("bot_skipped_15min", 0) > 100: + issues.append(f"🟡 Zulip echo loop: {z['bot_skipped_15min']} bot msgs skipped in 15min") + if z.get("server_status") and z["server_status"] not in ("200", "302"): + issues.append(f"🔴 Zulip server: HTTP {z['server_status']}") + + # Agent issues + def agent_gateway(agent): + return agent.get("gateway_state") or agent.get("pm2_status") or "unknown" + for name, agent in r.get("agents", {}).items(): + gs = agent_gateway(agent) + if gs == "unknown": + issues.append(f"🟡 {name}: gateway state unknown (unreachable)") + elif gs != "running" and gs != "online": + issues.append(f"🔴 {name}: gateway {gs}") + + status = "🟢 All Healthy" if not issues else f"{'🔴' if sum(1 for i in issues if '🔴' in i) > 0 else '🟡'} {len(issues)} Issue(s)" + + # ── Build HTML ── + html = f""" + +

🏗️ Syslog Infrastructure Report

+

{TIME_STR}

+ +
+

{status}

+

+{r['node_count']} PVE nodes · {r['total_vms']} VMs/CTs · {r['running_vms']} running · +{r['docker_vm']['total'] + r['docker_syslog']['total'] + r['docker_netbird']['total']} containers · +{len(r['endpoints'])} endpoints · {len(r.get('agents',{}))} agents +

+
+""" + + # Issues list + if issues: + html += '

🚨 Issues ({})

'.format(len(issues)) + for i in issues: + html += f'

{i}

' + html += '
' + + # ── Quick Stats ── + html += '

📊 Quick Stats

' + stats = [ + ("PVE Nodes", f"{r['nodes_online']}/{r['node_count']}", "green" if r['nodes_online'] == r['node_count'] else "red"), + ("VMs/CTs", f"{r['running_vms']}/{r['total_vms']}", "green" if r['running_vms'] == r['total_vms'] else "red"), + ("Containers", f"{r['docker_vm']['running']}/{r['docker_vm']['total']}", "green" if r['docker_vm']['running'] == r['docker_vm']['total'] else "yellow"), + ("LiteLLM Ctrs", f"{r['docker_syslog']['running']}/{r['docker_syslog']['total']}", "green" if r['docker_syslog']['running'] == r['docker_syslog']['total'] else "red"), + ("Netbird Ctrs", f"{r['docker_netbird']['running']}/{r['docker_netbird']['total']}", "green" if r['docker_netbird']['running'] == r['docker_netbird']['total'] else "red"), + ("Endpoints", f"{sum(1 for e in r['endpoints'] if e['code'] in ('200','302','401'))}/{len(r['endpoints'])}", "green",), + ("Zulip Ext", "✅ Connected" if r.get('zulip_ext',{}).get('connected') else "❌ Disconnected", "green" if r.get('zulip_ext',{}).get('connected') else "red"), + ] + for label, val, color in stats: + html += f'
{val}
{label}
' + + # VLM/GPU capacity + gpu_nodes = [v for vms in r["vms_by_node"].values() for v in vms if "llm" in v["name"].lower() or "gpu" in v["name"].lower() or "ocu" in v["name"].lower()] + gpu_running = sum(1 for g in gpu_nodes if g["status"] == "running") + html += f'
{gpu_running}/{len(gpu_nodes)}
GPU VMs
' + html += '
' + + # ── LiteLLM Section ── + html += '

⚡ LiteLLM Inference Stack

' + + # Endpoint status table + html += '' + for check in r.get("litellm", {}).get("checks", []): + color = "green" if check["status"] == "pass" else "red" + detail = check.get("code", "") + if check.get("missing"): + detail = f"missing: {', '.join(check['missing'])}" + elif check.get("healthy") is not None: + detail = f"{check['healthy']}/{check['total']} healthy" + html += f'' + html += '
CheckStatus
{check["name"]}{detail}
' + + # LiteLLM container detail + html += '

Containers (CT 116)

' + for c in r["docker_syslog"]["containers"]: + color = "green" if "Up" in c["status"] and "healthy" in c["status"] else ("yellow" if "Up" in c["status"] else "red") + html += f'' + html += '
ContainerStatusImage
{c["name"]}{c["status"]}{c["image"]}
' + html += '
' + + # ── Proxmox Nodes ── + html += '

📦 Proxmox Cluster

' + for name, nd in sorted(r["nodes"].items()): + cpu_color = "green" if nd["cpu_pct"] < 50 else ("yellow" if nd["cpu_pct"] < 80 else "red") + ram_color = "green" if nd["ram_pct"] < 70 else ("yellow" if nd["ram_pct"] < 85 else "red") + disk_color = "green" if nd["disk_pct"] < 70 else ("yellow" if nd["disk_pct"] < 85 else "red") + html += f'
{nd["cpu_pct"]}%
{name}
CPU: {nd["cpu_pct"]}%
RAM: {nd["ram_pct"]:.0f}%
DISK: {nd["disk_pct"]:.0f}%
⬆ {nd["uptime_h"]}h
' + html += '
' + if r["stopped_vms"]: + html += f'

Stopped: {", ".join(r["stopped_vms"])}

' + html += '
' + + # ── VM/CT Lists per Node ── + for node_name in ["amdpve", "minipve", "storepve", "acerpve", "ocupve"]: + node_vms = r["vms_by_node"].get(node_name, []) + if not node_vms: continue + stopped_count = sum(1 for v in node_vms if v["status"] != "running") + badge = f'{len(node_vms)-stopped_count} running' + if stopped_count: + badge += f' {stopped_count} stopped' + html += f'

🔹 {node_name}

{badge}

' + html += '' + for v in node_vms: + color = "green" if v["status"] == "running" else "red" + html += f'' + html += '
IDTypeNameStatusCPURAMDisk
{v["vmid"]}{v["type"]}{v["name"]}{v["status"]}{v["cpu"]}{v["ram"]}{v["disk"]}
' + + # ── Storage ── + html += '

💾 Storage

' + for s in sorted(r["storage"], key=lambda x: -x["pct"]): + color = "green" if s["pct"] < 70 else ("yellow" if s["pct"] < 85 else "red") + html += f'' + html += '
PoolUsedTotal%Type
{s["name"]}{s["used"]}{s["total"]}{s["pct"]:.0f}%{s["type"]}
' + + # ── Docker Ecosystems ── + html += '

🐳 Docker Ecosystems

' + for host, key, label in [ + ("docker-vm (.7) — storepve", "docker_vm", f"Disk: {r['docker_vm'].get('disk_used','?')}"), + ("syslog-api (.116) — minipve (LiteLLM)", "docker_syslog", ""), + ("Netbird (72.61.0.17)", "docker_netbird", ""), + ]: + d = r[key] + html += f'

{host} — {d["running"]}/{d["total"]} running {label}

' + if d.get("reclaimable"): + html += f'

Reclaimable Docker space: {d["reclaimable"]}

' + html += '' + for c in sorted(d["containers"], key=lambda x: x["name"]): + color = "green" if "Up" in c["status"] and "healthy" in c["status"] else ("yellow" if "Up" in c["status"] else "red") + html += f'' + html += '
ContainerStatus
{c["name"]}{c["status"][:50]}
' + html += '
' + + # ── Agent Status ── + html += '

🤖 Agent Status

' + for name, agent in sorted(r.get("agents", {}).items()): + if name == "abiba": + zulip_state = "✅" if agent.get("zulip_connected") else "❌" + gateway = f"pm2:{agent.get('pm2_status','?')} (↺{agent.get('pm2_restarts','?')})" + processed = f"{agent.get('zulip_processed', 0)} msgs" + elif name == "tanko": + zulip_state = "✅" if agent.get("zulip_state") == "connected" else ("❌" if agent.get("zulip_state") == "disconnected" else "⬜") + gateway = agent.get("gateway_state", "?") + processed = agent.get("updated_at", "")[:10] + elif name == "mumuni": + zulip_state = "⬜" if agent.get("zulip_state") == "not_installed" else ("✅" if agent.get("zulip_state") == "connected" else "⬜") + gateway = agent.get("gateway_state", "?") + tg = "✅" if agent.get("telegram_state") == "connected" else "❌" + ver = agent.get("hermes_version", "") + processed = f"TG:{tg} v{ver}" + else: + zulip_state = "⬜" + gateway = agent.get("gateway_state", "?") + processed = "" + html += f'' + html += '
AgentPlatformCTGatewayZulipProcessed
{name}{agent["platform"]}{agent["ct"]}{gateway}{zulip_state}{processed}
' + + # ── Zulip Extension Health ── + html += '

💬 Zulip Extension (Abiba)

' + z = r.get("zulip_ext", {}) + + checks = [ + ("Connection", "✅ Connected" if z.get("connected") else "❌ Disconnected"), + ("Queue", f"✅ {z.get('queue_id','none')[:12]}..." if z.get("queue_id") else "❌ No queue"), + ("Messages", f"{z.get('messages_processed',0)} processed"), + ("PM2 Status", f"{z.get('pm2',{}).get('status','?')} (↺{z.get('pm2',{}).get('restarts','?')} restarts, up {z.get('pm2',{}).get('uptime','?')})"), + ("Bot Echo Loop", f"{z.get('bot_skipped_15min',0)} skipped in 15min" if z.get('bot_skipped_15min',0) > 0 else "✅ No echo loop"), + ("Edit Success", f"{z.get('finalized_1h',0)} ok / {z.get('failed_finalize_1h',0)} fail" if z.get('failed_finalize_1h',0) > 0 else f"✅ {z.get('finalized_1h',0)} ok"), + ("Zulip Server", f"✅ HTTP {z.get('server_status','?')}" if z.get('server_status') in ('200','302') else f"❌ HTTP {z.get('server_status','?')}"), + ] + for name, val in checks: + color = "green" if "✅" in val or "ok" in val.lower() else ("yellow" if "🟡" in val or "skip" in val.lower() else "red") + html += f'' + html += '
CheckStatus
{name}{val}
' + + # ── Network Endpoints ── + html += '

🌐 Network Endpoints

' + for ep in r["endpoints"]: + color = "green" if ep["code"] in ("200","302","401") else ("yellow" if ep["code"] >= "400" else "red") + html += f'' + html += '
ServiceStatus
{ep["name"]}HTTP {ep["code"]}
' + + # ── NFS ── + html += '

🗄️ NFS Mounts

' + for m in r["nfs"]: + pct_str = m.get("pct","0%") + pct_num = int(pct_str.replace("%","")) if pct_str.replace("%","").strip().isdigit() else 0 + color = "green" if pct_num < 80 else ("yellow" if pct_num < 90 else "red") + html += f'' + html += '
MountSizeUsedAvail%
{m["mount"]}{m["size"]}{m["used"]}{m.get("avail","?")}{pct_str}
' + + # ── Suggestions ── + html += '

💡 Suggestions

    ' + + added = False + + # Storage warnings + for s in r["storage"]: + if s["name"] == "mediastore" and s["pct"] > 70: + html += f'
  • 📀 mediastore at {s["pct"]:.0f}% ({s["used"]}/{s["total"]}) — plan expansion or cleanup
    • ' + added = True + + # Stopped VMs + if r["stopped_vms"]: + html += f'
  • 🛑 Stopped VMs/CTs need attention: {", ".join(r["stopped_vms"][:5])}
    • ' + added = True + + # Endpoint issues + for ep in r["endpoints"]: + if ep["code"] >= "500": + html += f'
  • 🌐 {ep["name"]} returning HTTP {ep["code"]}
    • ' + added = True + + # Zulip extension + if not z.get("connected"): + html += f'
  • 💬 Zulip extension disconnected — run: pm2 restart abiba-zulip
    • ' + added = True + if z.get("finalize_fail_pct", 0) > 50: + html += f'
  • 💬 Zulip edit failures at {z["finalize_fail_pct"]:.0f}% — check editMessage API
    • ' + added = True + + # LiteLLM + for check in r.get("litellm", {}).get("checks", []): + if check["status"] == "fail" and check.get("missing"): + html += f'
  • ⚡ LiteLLM missing containers: {", ".join(check["missing"])}
    • ' + added = True + + # Backup compliance + html += '
  • 🗄️ Verify PBS backups completed within last 48h via Proxmox Backup Server (CT 107)
    • ' + + # Heartbeat suggestion + html += '
  • 💓 All agents now have Zulip heartbeat logging — silence detection active
    • ' + + if not added: + html += '
  • ✅ Nothing flagged — infrastructure is healthy
    • ' + + html += '
' + html += '

Syslog Infrastructure Monitor v2.0 · Generated ' + TIME_STR + '

' + html += '' + return html + + +# ── Send Email ── + +def send_email(html_content, subject_prefix=""): + FROM = "abiba@sysloggh.com" + TO = "jerome@sysloggh.com" + SUBJECT = f"{subject_prefix}{'🏗️ Infrastructure Report — ' + DATE_STR}" + + msg = MIMEMultipart("alternative") + msg["From"] = FROM + msg["To"] = TO + msg["Subject"] = SUBJECT + msg.attach(MIMEText("Infrastructure report in HTML format — enable images to view.", "plain")) + msg.attach(MIMEText(html_content, "html")) + + try: + EMAIL_PASSWORD = "rgbuomwcydxwbszd" + GMAIL_EMAIL = "jtabiri@gmail.com" + + server = smtplib.SMTP("smtp.gmail.com", 587) + server.starttls() + server.login(GMAIL_EMAIL, EMAIL_PASSWORD) + server.sendmail(FROM, [TO], msg.as_string()) + server.quit() + return True, "✅ Email sent to jerome@sysloggh.com" + except Exception as e: + return False, f"❌ Email failed: {e}" + + +# ── Main ── + +if __name__ == "__main__": + is_test = "--test-email" in sys.argv + + print(f"{'🧪 TEST MODE' if is_test else '📊'} Collecting infrastructure data...") + report = collect() + + if "--json" in sys.argv: + print(json.dumps(report, indent=2, default=str)) + sys.exit(0) + + print(" Building dashboard...") + html = build_html(report) + + if is_test: + prefix = "🧪 TEST — " + print(" Sending test email...") + else: + prefix = "" + print(" Sending email...") + + ok, msg = send_email(html, subject_prefix=prefix) + print(f" {msg}") + + # Show summary + issues = sum(1 for i in ["red"] if report.get("zulip_ext", {}).get("connected") == False) + print(f"\n📋 Summary:") + print(f" Proxmox: {report['nodes_online']}/{report['node_count']} nodes online") + print(f" VMs/CTs: {report['running_vms']}/{report['total_vms']} running") + print(f" Zulip Ext: {'✅' if report.get('zulip_ext',{}).get('connected') else '❌'}") + print(f" LiteLLM: {sum(1 for c in report.get('litellm',{}).get('checks',[]) if c['status']=='pass')}/{len(report.get('litellm',{}).get('checks',[]))} checks pass") + agent_parts = [] +for k,v in report.get('agents',{}).items(): + agent_parts.append(f"{k}:{v.get('gateway_state',v.get('pm2_status','?'))}") +print(f" Agents: {', '.join(agent_parts)}") diff --git a/scripts/zulip-monitor.sh b/scripts/zulip-monitor.sh new file mode 100755 index 0000000..0d77d80 --- /dev/null +++ b/scripts/zulip-monitor.sh @@ -0,0 +1,156 @@ +#!/bin/bash +# /root/scripts/zulip-monitor.sh — Zulip Mesh Health Monitor +# Implements zulip-health.prose.md v2 +# Runs every 15 min via cron. Alerts via Telegram. +set -euo pipefail + +ZULIP_SITE="https://chat.sysloggh.net" +ZULIP_EMAIL="abiba-bot@chat.sysloggh.net" +ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8" +OWNER_ZULIP_ID="9" + +# Email config +GMAIL_USER="jtabiri@gmail.com" +GMAIL_PASS="rgbuomwcydxwbszd" +EMAIL_TO="jerome@sysloggh.com" + +notify() { + local severity="$1" msg="$2" + echo "[$severity] $msg" + + # Zulip DM to owner + local content="${severity} Zulip Monitor: ${msg}" + local form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")" + curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \ + -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \ + -d "${form}" > /dev/null 2>&1 || true + + # Email alert + local subject="${severity} Zulip Monitor Alert" + python3 -c " +import smtplib +from email.mime.text import MIMEText +m = MIMEText('''${msg}''') +m['From'] = 'abiba@sysloggh.com' +m['To'] = '${EMAIL_TO}' +m['Subject'] = '${subject}' +s = smtplib.SMTP('smtp.gmail.com', 587) +s.starttls() +s.login('${GMAIL_USER}', '${GMAIL_PASS}') +s.sendmail('abiba@sysloggh.com', ['${EMAIL_TO}'], m.as_string()) +s.quit() +" 2>/dev/null || true +} + +TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC') +ISSUES=0 +LOG="/root/zulip-health-monitor.log" + +echo "=== Zulip Health Check — $TIMESTAMP ===" >> "$LOG" + +# ── Global: Zulip Server ── +SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \ + https://chat.sysloggh.net/api/v1/server_settings \ + -u 'abiba-bot@chat.sysloggh.net:cKTDMZAPW08dk3zl05sStzO7HRztzyn8' 2>/dev/null || echo "000") +if [ "$SERVER_CODE" != "200" ]; then + notify "🔴" "Zulip server returned HTTP $SERVER_CODE" + ISSUES=$((ISSUES + 1)) +else + echo " Server: ✅ HTTP 200" >> "$LOG" +fi + +# ── Platform A: pi (Abiba) ── +PI_HEALTH=$(curl -sf --connect-timeout 5 http://localhost:9200/health 2>/dev/null || echo "{}") +PI_CONNECTED=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('connected',False))" 2>/dev/null) +PI_ERROR=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('last_error') or '')" 2>/dev/null) +PI_RETRIES=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('retry_count',0))" 2>/dev/null) + +if [ "$PI_CONNECTED" != "True" ]; then + notify "🔴" "Abiba pi extension DISCONNECTED — restarting" + pm2 restart abiba-zulip 2>/dev/null || true + ISSUES=$((ISSUES + 1)) + echo " Abiba: ❌ Disconnected — restarted" >> "$LOG" +elif [ -n "$PI_ERROR" ]; then + notify "🟡" "Abiba pi extension error: ${PI_ERROR:0:100}" + echo " Abiba: 🟡 Error: ${PI_ERROR:0:100}" >> "$LOG" +elif [ "$PI_RETRIES" -ge 3 ]; then + notify "🟡" "Abiba pi extension: $PI_RETRIES retries — restarting" + pm2 restart abiba-zulip 2>/dev/null || true + echo " Abiba: 🟡 $PI_RETRIES retries — restarted" >> "$LOG" +else + echo " Abiba: ✅ Connected (processed=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('messages_processed',0))" 2>/dev/null))" >> "$LOG" +fi + +# ── Platform B: Hermes (Tanko) ── +TANKO_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 jerome@192.168.68.122 \ + "cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}") +TANKO_ZULIP=$(echo "$TANKO_STATE" | python3 -c " +import sys,json +d=json.load(sys.stdin) +p=d.get('platforms',{}).get('zulip',{}) +print(p.get('state','unknown')) +" 2>/dev/null) + +if [ "$TANKO_ZULIP" != "connected" ]; then + notify "🔴" "Tanko (Hermes) Zulip state: $TANKO_ZULIP — needs restart" + ISSUES=$((ISSUES + 1)) + echo " Tanko: ❌ state=$TANKO_ZULIP" >> "$LOG" +else + echo " Tanko: ✅ Zulip connected" >> "$LOG" +fi + +# ── Platform B: Hermes (Mumuni) ── +MUMUNI_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.123 \ + "cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}") +MUMUNI_ZULIP=$(echo "$MUMUNI_STATE" | python3 -c " +import sys,json +d=json.load(sys.stdin) +p=d.get('platforms',{}).get('zulip',{}) +print(p.get('state','unknown')) +" 2>/dev/null) + +if [ "$MUMUNI_ZULIP" != "connected" ]; then + notify "🔴" "Mumuni (Hermes) Zulip state: $MUMUNI_ZULIP" + ISSUES=$((ISSUES + 1)) + echo " Mumuni: ❌ state=$MUMUNI_ZULIP" >> "$LOG" +else + echo " Mumuni: ✅ Zulip connected" >> "$LOG" +fi + +# ── Platform C: Agent Zero (kagentz) ── +AZ_A2A=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \ + "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json 2>/dev/null" 2>/dev/null || echo "") +AZ_ALIVE=$(echo "$AZ_A2A" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('name',''))" 2>/dev/null) + +if [ "$AZ_ALIVE" != "kagentz" ]; then + notify "🔴" "kagentz A2A server DOWN — restarting" + ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \ + "docker exec agent-zero bash -c 'pkill -9 -f a2a_agent; sleep 1; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &'" 2>/dev/null || true + ISSUES=$((ISSUES + 1)) + echo " kagentz: ❌ A2A down — restarted" >> "$LOG" +else + echo " kagentz: ✅ A2A alive" >> "$LOG" + + # Check adapter process + AZ_ADAPTER=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \ + "docker exec agent-zero ps aux 2>/dev/null | grep adapter | grep -v grep | wc -l" 2>/dev/null || echo "0") + if [ "$AZ_ADAPTER" -lt 1 ]; then + notify "🔴" "kagentz Zulip adapter DOWN — restarting" + ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \ + "docker exec agent-zero bash -c 'cd /a0/usr/kagentz-zulip && ZULIP_SITE=https://chat.sysloggh.net ZULIP_EMAIL=kagentz-bot@chat.sysloggh.net ZULIP_API_KEY=E9q9PXJTxftPYBkb5pBDWupDO7KK21ty ZULIP_AGENT_NAME=kagentz A2A_URL=http://localhost:8001/a2a A2A_TOKEN=8zNgdOEXzYxjQvTl /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &'" 2>/dev/null || true + ISSUES=$((ISSUES + 1)) + echo " kagentz: ❌ Adapter down — restarted" >> "$LOG" + else + echo " kagentz: ✅ Adapter running" >> "$LOG" + fi +fi + +# ── Summary ── +if [ "$ISSUES" -eq 0 ]; then + echo " Result: ✅ All healthy" >> "$LOG" +else + echo " Result: 🔴 $ISSUES issue(s) found" >> "$LOG" + notify "🔴" "$ISSUES issue(s) found — check /root/zulip-health-monitor.log" +fi + +echo "" >> "$LOG" From 8711840b264b9adbb610c748575269bc1674d85a Mon Sep 17 00:00:00 2001 From: Jerome Tabiri Date: Sun, 28 Jun 2026 13:20:27 -0400 Subject: [PATCH 14/16] Add memory-audit-maintenance prose contract --- memory-audit-maintenance.prose.md | 169 ++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 memory-audit-maintenance.prose.md diff --git a/memory-audit-maintenance.prose.md b/memory-audit-maintenance.prose.md new file mode 100644 index 0000000..a816180 --- /dev/null +++ b/memory-audit-maintenance.prose.md @@ -0,0 +1,169 @@ +--- +kind: responsibility +name: memory-audit-maintenance +description: > + Systematic audit and reorganization of an agent's native memory (MEMORY.md, + USER.md, SOUL.md). Categorizes entries, redistributes to correct homes, + verifies live configs, and frees up char budget. Run when memory usage + exceeds 85% or on explicit user request ("memory audit", "memory maintenance"). +--- + +## Maintains + +- last_audit: timestamp — When the last full audit was completed +- memory_usage_pct: number — Current memory store usage percentage +- user_usage_pct: number — Current user store usage percentage +- total_entries_categorized: number — How many entries were audited +- entries_moved: number — How many entries changed location +- entries_deleted: number — How many stale entries were removed +- live_configs_verified: array — Which configs were verified against live systems +- known_issues: array — Problems found during audit (stale keys, dead services) + +## Parameters + +- memory_threshold: number — Usage % that triggers audit (default: 85, max: 95) +- verify_configs: boolean — Whether to live-check configs after audit (default: true) +- compress_entries: boolean — Whether to shorten verbose entries (default: true) +- remove_session_notes: boolean — Strip "currently doing X" markers (default: true) +- max_memory_chars: number — Target max chars for MEMORY.md after audit (default: 1100) +- max_user_chars: number — Target max chars for USER.md after audit (default: 500) + +## Continuity + +The audit cycle is **event-driven**, not just cron-based: + +- **On threshold breach**: If memory usage exceeds `memory_threshold`, wake immediately +- **On user request**: `"memory audit"`, `"memory maintenance"`, `"clean up memory"` — explicit trigger +- **Retrospective**: Every 2-4 weeks if no other trigger fired — check for gradual bloat +- **On config change**: After major provider swaps or infrastructure migrations, verify live configs +- **On new agent onboarding**: Run once as part of initialization to establish clean baseline + +## Success Criteria (What Makes a Memory Audit "Good") + +The audit is considered COMPLETE when ALL of these pass: + +### Inventory Complete +- MEMORY.md content read and categorized +- USER.md content read and categorized +- In-memory `memory` and `user` store usage percentages recorded +- Every entry tagged with: `Infrastructure | Identity | Preference | Operating Rule | Historical | Session Note` + +### Redistribution Correct +- Identity entries → USER.md +- Preference entries → USER.md +- Technical config entries → MEMORY.md +- Operating rules → Skill file or SOUL.md +- Historical snapshots (dated checkpoints) → Deleted +- Session notes ("currently doing X") → Deleted + +### Files Compact +- MEMORY.md ≤ `max_memory_chars` chars +- USER.md ≤ `max_user_chars` chars +- No duplicate information across files +- No frustration signals or verbose descriptions +- § separator between entries (standard convention) + +### Configs Verified +- API keys confirmed in `.env` file (grep, not read) +- Model names in memory match live `config.yaml` +- Endpoints in memory respond to curl/health checks +- Credential pairs (username/token, email/password) match deployed state +- Any mismatch corrected in memory with ✅/❌ labels + +### In-Memory Synced +- Summary marker added to `memory` store: "MEMORY.md rewritten . Holds only tech configs." +- Summary marker added to `user` store: "Home channel: . USER.md rewritten ." +- Usage confirmed dropped below 60% after sync + +### Skill Created/Updated (if applicable) +- Operating rules that belong in a skill → created via `skill_manage` +- Existing skills verified not to conflict with memory content +- Skill category appropriate and description searchable + +## Remembers + +Each audit stores: +- Entry categorization decisions (why X went to USER instead of MEMORY) +- What was deleted (stale snapshots, session notes) +- What was verified (configs that passed live checks) +- What failed verification (configs that need user attention) +- Known tool quirks (memory replace/remove failures) +- Compression patterns used (how verbose entries were shortened) + +## Audit Rules + +### Rule 1: Read Before Delete +Never delete an entry without first reading it and categorizing it. Show the user a summary of what will change if unsure about a categorization. + +### Rule 2: Categories Are Mutual Exclusive +Each entry belongs to exactly ONE category: +- **Identity** — "Jerome, Florida, Telegram @mejerome19" +- **Preference** — "Single-account first for AWS" +- **Technical Config** — "SearXNG on storepve:8888" +- **Operating Rule** — "Never go rogue on infrastructure" +- **Historical Snapshot** — "Baseline v30 from Jun 26" (DELETE) +- **Session Note** — "Currently performing memory audit" (DELETE) + +### Rule 3: Rules Belong in Skills +If an entry is a procedure, protocol, or mandate (`"do X this way"`, `"never do Y"`) — it belongs in a skill file, not MEMORY.md. Check `skills_list` first; create via `skill_manage` if no match exists. + +### Rule 4: Verify Live Before Trusting Memory +Every config entry in MEMORY.md must be verified against the live system: +- Model name → `grep config.yaml` +- API key → `grep .env` +- Service URL → `curl` health check +- Credentials → cross-reference with deployed state + +### Rule 5: Work Around the `memory` Tool +The `memory` tool's `remove` and `replace` actions use strict text matching that can fail on special characters. If a remove/replace fails repeatedly: +- Do not retry more than 2 times +- Add a new summary marker entry instead +- Report to user: "File on disk updated; in-memory store has stale leftovers" +- The file on disk (MEMORY.md, USER.md) is the source of truth regardless + +### Rule 6: Report Before Asking +Don't ask the user "should I delete X?" for session notes or historical snapshots — those are always safe to delete. Only ask about ambiguous categorizations or cross-file moves of preference data. + +## Execution + +1. **Read state** — Read MEMORY.md, USER.md, check in-memory usage via `memory` tool +2. **Categorize** — Tag every entry with its category +3. **Prepare rewrites** — Draft compressed MEMORY.md and USER.md: + - MEMORY.md: Technical configs only (800-1100 chars) + - USER.md: Identity + preferences only (300-500 chars) +4. **Write files** — Overwrite MEMORY.md and USER.md +5. **Verify live configs** — Check every config entry against actual system state +6. **Sync in-memory** — Add summary markers to `memory` and `user` stores +7. **Create/update skills** — Move operating rules to skill files if needed +8. **Report** — Output clean status table with before/after/verification + +## Example Output + +``` +## Memory Audit Complete ✅ + +| File | Before | After | Reduction | +|---|---|---|---| +| MEMORY.md | 1,653 chars | 1,077 chars | ~35% | +| USER.md | 860 chars | 418 chars | ~51% | + +### What Moved +| Entry | From | To | +|---|---|---| +| Identity + Preferences | MEMORY.md | USER.md | +| Agent boundary rule | MEMORY.md | SOUL.md (persona) | +| Baseline v30 snapshot | MEMORY.md | Deleted | + +### Config Verification +| Config | Live Check | Status | +|---|---|---| +| DeepSeek model | config.yaml: deepseek-chat | ✅ | +| DeepSeek key | .env found | ✅ | +| SearXNG endpoint | :8888 curl 200 | ✅ | + +### Memory Usage +| Store | Before | After | Status | +|---|---|---|---| +| memory | 91% | 59% | ✅ | +| user | 85% | 41% | ✅ | +``` From 6a8a2c61dac675476587fea2c5b68829dceb88a9 Mon Sep 17 00:00:00 2001 From: Jerome Tabiri Date: Sun, 28 Jun 2026 13:33:06 -0400 Subject: [PATCH 15/16] Add hermes-config-template prose contract with shared infra + flexible models --- hermes-config-template.prose.md | 246 ++++++++++++++++++++++++++++++++ 1 file changed, 246 insertions(+) create mode 100644 hermes-config-template.prose.md diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md new file mode 100644 index 0000000..ba077a3 --- /dev/null +++ b/hermes-config-template.prose.md @@ -0,0 +1,246 @@ +--- +kind: template +name: hermes-config-template +description: > + Standard Hermes configuration template for Syslog Solution LLC agents. + Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, + RA-H OS MCP) while keeping model/provider choices flexible per agent role. + Every new agent profile should start from this template. +--- + +## Maintains + +- template_version: string — Current template version (e.g., "1.0.0") +- last_applied: timestamp — When any agent was last configured from this template +- agents_configured: array — Which agents/profiles were created from this template +- infra_endpoints_verified: array — Firecrawl, SearXNG, RA-H OS, LiteLLM endpoints last confirmed working +- known_deviations: array — Profiles that diverge from the template (and why) + +## Parameters + +- agent_name: string — Name of the agent/profile (e.g., "syslog-code", "syslog-devops") +- default_model: string — Agent's primary model (e.g., "qwen3.6-27B-code", "syslog-auto") +- default_provider: string — Inference provider for the primary model (default: "harness") +- fallback_model: string — Fallback model when primary is down (default: "gemma-4-12b") +- fallback_provider: string — Fallback provider (default: "harness") +- auxiliary_model: string — Model for vision/compression/extraction tasks (default: "gemma-4-12b") +- enable_firecrawl_searxng: boolean — Whether to configure web search/extract (default: true) +- enable_ra_h_os_mcp: boolean — Whether to wire up RA-H OS MCP bridge (default: true) +- enable_compression: boolean — Whether to enable context compression (default: true) +- custom_provider_name: string — Custom provider name (default: "harness") +- custom_provider_model: string — Custom provider model (default: same as default_model) +- custom_provider_url: string — LiteLLM base URL (default: "http://litellm.sysloggh.net/litellm/v1") +- extra_auxiliary: array — Additional auxiliary tasks to configure (vision, compression, etc.) +- output_dir: string — Where to write the config (default: "~/.hermes/profiles//config.yaml") + +## Infrastructure Stack + +This template provisions the following shared infrastructure. Every agent should point to the same endpoints unless explicitly overridden. + +| Component | Endpoint | Purpose | +|---|---|---| +| Firecrawl | `http://192.168.68.7:3002/` | Web content extraction | +| SearXNG | `http://storepve:8888` | Privacy-respecting web search | +| LiteLLM | `http://litellm.sysloggh.net/litellm/v1` | Unified model gateway | +| RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge | +| Context7 MCP | `http://localhost:8079/mcp` | Documentation queries | + +**API Key Rules:** +- `api_key: sk-_SW...B8nw` — Hardcoded LiteLLM master key for custom_providers and auxiliary tasks +- `api_key_env: DEEPSEEK_API_KEY` — Env-var based key for fallback provider +- Worker profiles should inherit proxy config via `harness` custom_provider, NOT hardcode LiteLLM keys + +**For workers that need API keys overridden:** Set `api_key: ''` in the model section (inherits from custom_providers), or use `api_key_env` for env-based auth. + +## Template Structure + +### Required Sections (every agent MUST have) + +```yaml +# ─── Model Selection (USER CHOOSES) ─── +model: + # !! CHANGE THIS for your agent !! + default: + provider: + # LiteLLM base URL (shared infra) + base_url: http://litellm.sysloggh.net/litellm/v1 + +fallback_providers: + # !! OPTIONAL: overridable fallback !! + provider: + model: + +# ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ─── +web: + backend: firecrawl + search_backend: searxng + extract_backend: firecrawl + firecrawl: + base_url: http://192.168.68.7:3002/ + +# ─── MCP Servers (SHARED INFRA — DO NOT CHANGE) ─── +mcp_servers: + context7: + connect_timeout: 60 + timeout: 300 + url: http://localhost:8079/mcp + ra-h-os: + url: http://192.168.68.65:3100/mcp + timeout: 120 + connect_timeout: 60 + +# ─── Compression (SHARED — can override model) ─── +compression: + enabled: true + provider: harness + model: # default: gemma-4-12b + threshold: 0.5 + target_ratio: 0.25 + protect_last_n: 30 + hygiene_hard_message_limit: 400 + +# ─── Auxiliary Tasks (SHARED INFRA + MODEL) ─── +auxiliary: + vision: + provider: harness + model: + web_extract: + provider: harness + model: + session_search: + provider: harness + model: + max_concurrency: 3 + +# ─── Custom Provider (SHARED INFRA — LiteLLM) ─── +custom_providers: + - name: + model: + base_url: http://litellm.sysloggh.net/litellm/v1 + api_key: + api_mode: chat_completions +``` + +### Optional Sections (agent-specific) + +- **Platform configs** (Telegram, Discord, WhatsApp bot tokens) +- **Display/skin** options (compact mode, personality) +- **Plugin lists** (zulip-platform, etc.) +- **Terminal settings** (container image, timeouts) +- **Security/approvals** (command allowlists) +- **Kanban/worker** settings (if this profile is a Kanban worker) +- **Skills auto_load** list +- **Timezone** override + +## Continuity + +Configuration drift detection is **event-driven**: + +- **On agent onboarding**: Always scaffold from this template +- **On infra change**: If Firecrawl/SearXNG/LiteLLM endpoints change, update template + all profiles +- **On new model**: When a new model is deployed on Litellm, any agent can change their `default` independently +- **Periodic**: Every 2 weeks — check if any profile's infra section has drifted from the template +- **On connection failure**: If web search/extract fails, check if that agent has missing `web.*` config + +## Success Criteria + +The configuration is CONSIDERED GOOD when ALL of these pass: + +### Infra Connectivity +- `web.backend == "firecrawl"` — Firecrawl backend configured +- `web.search_backend == "searxng"` — SearXNG search configured +- `web.extract_backend == "firecrawl"` — Firecrawl extraction configured +- `web.firecrawl.base_url` points to `192.168.68.7:3002` +- `mcp_servers.ra-h-os.url` points to `192.168.68.65:3100/mcp` +- All endpoints respond to `curl` health checks + +### Model Flexibility +- `model.default` is settable per agent (not hardcoded) +- `model.provider` is settable per agent +- `custom_providers[0].model` matches the agent's workload (code vs general) + +### Completeness +- `auxiliary.vision` configured (even if provider=auto) +- `compression.enabled` set (true for most agents) +- Fallback provider configured (deepseek or another harness model) +- `_config_version` correctly set (currently `30`) + +### No Drift +- No duplicate infra endpoints (one canonical Firecrawl URL) +- No stale endpoints (old IPs, dead services) +- All profiles consistent on shared infra + +## Configuration Rules + +### Rule 1: Shared Infra Is Locked +The following MUST be identical across ALL profiles: +- `web.backend`, `web.search_backend`, `web.extract_backend` +- `web.firecrawl.base_url` +- `mcp_servers.ra-h-os.url` +- `custom_providers[0].base_url` + +### Rule 2: Model Choice Is Free +The following are OWNED by each agent and can differ: +- `model.default` — the primary model +- `model.provider` — where it runs +- `fallback_providers.model` — backup model +- `custom_providers[0].model` — what this agent uses LiteLLM for + +### Rule 3: Workers Inherit, Not Duplicate +Worker profiles (syslog-code, syslog-devops, etc.) generally inherit from the main config. Only override what's DIFFERENT: +- Different default model → override `model.default` +- Different auxiliary model → override auxiliary sections +- No web search needed → still keep the config (tools won't enable without it) + +### Rule 4: API Key Placement +- **Custom provider keys** (`custom_providers[0].api_key`): Must be hardcoded (the LiteLLM master key or a virtual key) +- **Model section keys** (`model.api_key`): Prefer empty. Workers inherit from custom_providers. +- **Environment keys** (`api_key_env`): For fallback providers that need separate auth (DeepSeek, OpenRouter) +- **Hardcoded keys in profile**: If a worker's `model.api_key` is set, it overrides custom_providers. Only use when the worker needs a DIFFERENT provider than Litellm. + +### Rule 5: Verify After Every Config Change +After changing a profile's config: +1. `curl` the model endpoint to confirm auth works +2. `curl` the web stack endpoints (Firecrawl, SearXNG) +3. Check that `hermes gateway --restart` reloads cleanly + +## Execution + +1. **Check current config** — Read the target agent's config.yaml +2. **Compare against template** — Identify missing or divergent sections +3. **Apply shared infra** — Lock the web/MCP/compression sections to template values +4. **Apply model choice** — Set default/fallback/auxiliary models per agent's workload +5. **Preserve agent-specific** — Keep platform configs, plugins, terminal settings, skills +6. **Remove stale** — Drop any old infra endpoints that don't match the template +7. **Verify** — curl all shared endpoints, test the model +8. **Report** — What was changed, what was preserved, what's still custom + +## Example Output + +``` +## Config Template Applied: syslog-code ✅ + +### Shared Infrastructure (Locked) +| Section | Before | After | +|---|---|---| +| web.backend | firecrawl | firecrawl ✅ | +| web.search_backend | NOT SET | searxng ✅ | +| web.firecrawl.base_url | NOT SET | http://192.168.68.7:3002/ ✅ | +| mcp_servers.ra-h-os | present | present ✅ | + +### Agent Model Choices (Preserved) +| Setting | Value | +|---|---| +| model.default | qwen3.6-27B-code (code agent) | +| fallback | gemma-4-12b (harness) | +| auxiliary | gemma-4-12b (harness) | +| compression | gemma-4-12b (harness) | + +### Endpoint Verification +| Endpoint | Status | +|---|---| +| Firecrawl :3002 | ✅ 200 OK | +| SearXNG :8888 | ✅ 200 OK | +| LiteLLM | ✅ 200 OK | +| RA-H OS MCP | ✅ Connected | +``` From 94b3bf84b4443c4f094f7d11ef8fd98d30a98e1a Mon Sep 17 00:00:00 2001 From: Jerome Tabiri Date: Sun, 28 Jun 2026 13:49:06 -0400 Subject: [PATCH 16/16] Add README + Quickstart sections for agent execution --- README.md | 97 +++++++++++++++++++++++++++++++ hermes-config-template.prose.md | 61 +++++++++++++++++++ memory-audit-maintenance.prose.md | 36 ++++++++++++ 3 files changed, 194 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..d6e0b20 --- /dev/null +++ b/README.md @@ -0,0 +1,97 @@ +# Prose Contracts — Syslog Solution LLC + +Operating contracts for Hermes agents. Each `.prose.md` file defines a **responsibility** (recurring duty) or **template** (scaffold) that agents can execute via OpenProse or follow manually. + +## How Agents Run These Contracts + +### Option A: Via OpenProse CLI (preferred) + +```bash +prose run [param1=value1 param2=value2 ...] +``` + +**Examples:** + +```bash +# Run a memory audit with default parameters +prose run memory-audit-maintenance + +# Run a memory audit with custom threshold +prose run memory-audit-maintenance memory_threshold=90 verify_configs=true + +# Configure a new agent with the template +prose run hermes-config-template agent_name=syslog-devops default_model=claude-sonnet-4 + +# Configure an agent with a different auxiliary model +prose run hermes-config-template agent_name=syslog-code default_model=qwen3.6-27B-code auxiliary_model=gemma-4-12b +``` + +### Option B: Manual Execution + +If `prose` CLI isn't available, any agent can: +1. Fetch the raw contract: `curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/` +2. Read the **Execution** section +3. Follow the steps + +**Example for fetching:** +```bash +curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/memory-audit-maintenance.prose.md +``` + +## Available Contracts + +### Responsibilities (Recurring Duties) + +| Contract | What It Does | When To Run | +|---|---|---| +| `memory-audit-maintenance` | Audits & reorganizes an agent's native memory (MEMORY.md, USER.md). Categorizes entries, moves rules to skills, verifies configs, frees up char budget. | Memory >85% usage or user request: "memory audit" | +| `zulip-health` | Checks Zulip connectivity, message flow, and bot responsiveness. | On Zulip issues or periodic health check | +| `litellm-health` | Verifies LiteLLM proxy connectivity, model availability, and key rotation. | On inference failures or periodic check | +| `litellm-self-heal` | Attempts to fix common LiteLLM issues (key rotation, restart, config reload). | When litellm-health reports failures | +| `zulip-mention-reliability` | Diagnoses and fixes @mention detection issues in Zulip. | On missed @mention reports | +| `pm2-self-heal` | Restarts crashed PM2 processes and verifies recovery. | On PM2 process failure | + +### Templates (Scaffolds) + +| Contract | What It Does | Parameters | +|---|---|---| +| `hermes-config-template` | Generates a standard Hermes config for any agent. Enforces shared infra (Firecrawl, SearXNG, RA-H OS MCP, LiteLLM) while keeping model choices flexible. | `agent_name` (required), `default_model`, `default_provider`, `fallback_model`, `auxiliary_model` | + +### Scripts + +| File | What It Does | +|---|---| +| `pm2-self-heal.sh` | Shell script to restart crashed PM2 processes (companion to the prose contract) | + +## Contract Structure + +Every `.prose.md` file has the same structure: + +``` +--- +kind: +name: +description: ... +--- + +## Maintains — What state the contract tracks +## Parameters — Tunable inputs (with defaults) +## Continuity — When to run (triggers & cadence) +## Success Criteria — How to know it worked +## Remembers — What to learn between runs +## Rules — Inviolable rules for execution +## Execution — Step-by-step instructions +## Example Output — What a good run looks like +``` + +## Adding New Contracts + +1. Create a `.prose.md` file following the structure above +2. Push to this repo +3. The contract is immediately available for any agent to `prose run` + +## Repository + +- **URL:** https://git.sysloggh.net/SyslogSolution/prose-contracts +- **Branch:** master +- **Auth:** mumuni-bot (write access via PAT) diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index ba077a3..873ad56 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -33,6 +33,67 @@ description: > - extra_auxiliary: array — Additional auxiliary tasks to configure (vision, compression, etc.) - output_dir: string — Where to write the config (default: "~/.hermes/profiles//config.yaml") +## Quickstart — How to Run This Contract + +### For the Agent Running This Contract + +Read this section first. It tells you exactly what to do. + +**When to run:** +- Setting up a **new agent profile** for the first time +- An **existing profile** is missing web search, firecrawl, or MCP configs +- User says: `"apply the config template"` or `"fix my infra config"` +- User says: `"configure with "` + +**How to run (via OpenProse CLI):** +```bash +# Minimal — only agent name required +prose run hermes-config-template agent_name=syslog-code + +# Full control +prose run hermes-config-template \ + agent_name=syslog-devops \ + default_model=claude-sonnet-4 \ + auxiliary_model=gemma-4-12b \ + fallback_model=deepseek-chat \ + fallback_provider=deepseek +``` + +### How to Follow the Template (Manual) + +If `prose` CLI is not available: + +1. **Fetch this contract:** `curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/hermes-config-template.prose.md` +2. **Open the target config.yaml** — the profile you're updating +3. **Compare** each of the 5 required sections below against what exists in the profile +4. **For each section:** + - If it says **SHARED INFRA — DO NOT CHANGE**: Copy the value exactly as shown + - If it says **USER CHOOSES**: Use the agent's assigned model/provider + - If missing entirely: Add the full section +5. **Verify** every endpoint responds (see Verification section at the end) +6. **Restart** the gateway + +### Decision Tree + +``` +What kind of profile is this? +├── Main profile (~/.hermes/config.yaml) +│ └── Replace web.*, mcp_servers.*, compression.*, auxiliary.*, custom_providers.* +│ └── model.default + fallback: use current values (USER CHOOSES) +│ +├── Worker profile (~/.hermes/profiles//config.yaml) +│ ├── Does it have web.search_backend? +│ │ ├── NO → Add the full web.* section (copy exactly) +│ │ └── YES → Verify it points to the right IP +│ ├── Does it have mcp_servers.ra-h-os? +│ │ ├── NO → Add mcp_servers.* section (copy exactly) +│ │ └── YES → Verify URL +│ └── model.default: USE the worker's assigned model (don't copy main) +│ +└── Non-Syslog profile (personal/lifestyle agent) + └── Skip — use default Hermes config or Tanko's config instead +``` + ## Infrastructure Stack This template provisions the following shared infrastructure. Every agent should point to the same endpoints unless explicitly overridden. diff --git a/memory-audit-maintenance.prose.md b/memory-audit-maintenance.prose.md index a816180..d154c2d 100644 --- a/memory-audit-maintenance.prose.md +++ b/memory-audit-maintenance.prose.md @@ -8,6 +8,42 @@ description: > exceeds 85% or on explicit user request ("memory audit", "memory maintenance"). --- +## Quickstart — How to Run This Contract + +### For the Agent Running This Contract + +Read this section first. It tells you exactly what to do. + +**When to run:** +- User says: `"memory audit"`, `"memory maintenance"`, `"clean up memory"`, `"memory is full"` +- You detect that your `memory` store usage exceeds 85% in a write attempt +- It's been 2+ weeks since the last audit + +**How to run (via OpenProse CLI):** +```bash +# Default — 85% threshold, verify configs, compress entries +prose run memory-audit-maintenance + +# Softer audit — only flag issues, don't rewrite files +prose run memory-audit-maintenance compress_entries=false + +# Hard audit — stricter limits +prose run memory-audit-maintenance memory_threshold=90 max_memory_chars=800 max_user_chars=300 +``` + +### How to Follow the Template (Manual) + +If `prose` CLI is not available: + +1. **Read both memory files:** `cat ~/.hermes/memories/MEMORY.md` and `cat ~/.hermes/memories/USER.md` +2. **Check in-memory usage:** Use the `memory` tool with no args +3. **Categorize every entry** using the Categorization Guide below +4. **Rewrite MEMORY.md** — technical configs only, 800–1,100 chars max +5. **Rewrite USER.md** — identity + preferences only, 300–500 chars max +6. **Verify live configs** — curl each endpoint, check each model +7. **Sync in-memory** — Add summary markers (or just leave file as source of truth) +8. **Report** using the Example Output template at the bottom + ## Maintains - last_audit: timestamp — When the last full audit was completed