Compare commits

...
Author SHA1 Message Date
root 27bc6bafba Fix zulip-monitor.sh LOG variable scope bug
Move LOG, TIMESTAMP, and ISSUES declarations from inside notify() to
script scope so they are available throughout the script.
2026-08-29 22:09:46 +00:00
root 97dd261123 Fix zulip-monitor.sh email spam: Replace Gmail SMTP with Zulip stream post to #agent-hub on topic 'zulip-health' 2026-08-29 21:30:54 +00:00
mumuni-bot a179164f1f Merge pull request 'Ship: Koby external-agent ruling (fix api_key_env hygiene + docs)' (#46) from ship/koby-external-agent-fix into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
merge
2026-08-13 00:33:54 +00:00
mumuni-bot c4626c3512 Merge pull request 'fix: redirect pm2/zulip health-check logging to Gitea, never graph (hard rule)' (#47) from fix/health-logs-not-graph into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
merge
2026-08-13 00:33:33 +00:00
mumuni-bot b799d46596 fix: redirect pm2/zulip health-check logging to Gitea, never graph (hard rule)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- pm2-self-heal.prose.md: description said 'logs every action to the
  knowledge graph', contradicting its own execution step (line ~60) that
  says '(not knowledge graph - hard rule)'. Align description to Gitea.
- zulip-self-heal.prose.md: Reporting section said 'every cycle produces
  a knowledge graph node'. Contract is RETIRED; remove graph-node directive.
- Both now consistently log to SyslogSolution/health-logs, never the graph.
2026-08-13 00:09:40 +00:00
4 changed files with 16 additions and 30 deletions
+2 -1
View File
@@ -4,7 +4,8 @@ name: pm2-self-heal
description: >
Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner,
spoton-service, zulip-watchdog) and auto-restarts any that are stopped or
errored. Logs every action to the knowledge graph and alerts the owner via
errored. Logs every action to Gitea (SyslogSolution/health-logs — not
knowledge graph, hard rule) and alerts the owner via
Zulip DM on failures.
CRITICAL: Never restart abiba-zulip — it runs this contract.
AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative):
+11 -26
View File
@@ -9,10 +9,11 @@ ZULIP_EMAIL="abiba-bot@chat.sysloggh.net"
ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
OWNER_ZULIP_ID="9"
# Email config
GMAIL_USER="jtabiri@gmail.com"
GMAIL_PASS="rgbuomwcydxwbszd"
EMAIL_TO="jerome@sysloggh.com"
LOG="/root/zulip-health-monitor.log"
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
ISSUES=0
echo "=== Zulip Health Check — $TIMESTAMP ===" >> "$LOG"
notify() {
local severity="$1" msg="$2"
@@ -24,30 +25,14 @@ notify() {
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-d "${form}" > /dev/null 2>&1 || true
# Email alert
local subject="${severity} Zulip Monitor Alert"
python3 -c "
import smtplib
from email.mime.text import MIMEText
m = MIMEText('''${msg}''')
m['From'] = 'abiba@sysloggh.com'
m['To'] = '${EMAIL_TO}'
m['Subject'] = '${subject}'
s = smtplib.SMTP('smtp.gmail.com', 587)
s.starttls()
s.login('${GMAIL_USER}', '${GMAIL_PASS}')
s.sendmail('abiba@sysloggh.com', ['${EMAIL_TO}'], m.as_string())
s.quit()
" 2>/dev/null || true
# Zulip stream post to #agent-hub on topic 'zulip-health'
local stream_content="${severity} Zulip Monitor: ${msg}"
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-d "type=stream\&to=%5B7%5D\&topic=zulip-health\&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote(str()))")" \
> /dev/null 2>&1 || true
}
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
ISSUES=0
LOG="/root/zulip-health-monitor.log"
echo "=== Zulip Health Check — $TIMESTAMP ===" >> "$LOG"
# ── Global: Zulip Server ──
SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \
https://chat.sysloggh.net/api/v1/server_settings \
+1
View File
@@ -324,6 +324,7 @@ Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
- **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
- **Debounce gate**: Skip all checks entirely if recently restarted
4. **Zulip stream delivery** — Alerts posted to #agent-hub on topic 'zulip-health' (stream_id 7) instead of email, keeping DM to captain for critical (🔴) alerts
### Gen 4 (2026-06-29) — Response Pipeline Fix
**Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence.
+2 -3
View File
@@ -75,9 +75,8 @@ Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
## Reporting
Every cycle produces a knowledge graph node:
- Title: `[LEARN] zulip-self-heal: <timestamp>`
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
This contract is RETIRED — health-check logs are NOT knowledge graph content.
No graph nodes are created. Logs go to Gitea (SyslogSolution/health-logs).
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"