Compare commits

..
Author SHA1 Message Date
root 27bc6bafba Fix zulip-monitor.sh LOG variable scope bug
Move LOG, TIMESTAMP, and ISSUES declarations from inside notify() to
script scope so they are available throughout the script.
2026-08-29 22:09:46 +00:00
root 97dd261123 Fix zulip-monitor.sh email spam: Replace Gmail SMTP with Zulip stream post to #agent-hub on topic 'zulip-health' 2026-08-29 21:30:54 +00:00
mumuni-bot a179164f1f Merge pull request 'Ship: Koby external-agent ruling (fix api_key_env hygiene + docs)' (#46) from ship/koby-external-agent-fix into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
merge
2026-08-13 00:33:54 +00:00
mumuni-bot c4626c3512 Merge pull request 'fix: redirect pm2/zulip health-check logging to Gitea, never graph (hard rule)' (#47) from fix/health-logs-not-graph into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
merge
2026-08-13 00:33:33 +00:00
mumuni-bot b799d46596 fix: redirect pm2/zulip health-check logging to Gitea, never graph (hard rule)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- pm2-self-heal.prose.md: description said 'logs every action to the
  knowledge graph', contradicting its own execution step (line ~60) that
  says '(not knowledge graph - hard rule)'. Align description to Gitea.
- zulip-self-heal.prose.md: Reporting section said 'every cycle produces
  a knowledge graph node'. Contract is RETIRED; remove graph-node directive.
- Both now consistently log to SyslogSolution/health-logs, never the graph.
2026-08-13 00:09:40 +00:00
root 3abf784538 Fix no-mistakes warnings: remove prose from fence, reorder/renumber rules, and add Koby exception to Rule 10
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-11 18:16:12 +00:00
root d73084d481 Trivial: trigger no-mistakes re-run (all fixes applied)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-11 18:04:31 +00:00
root 9f0a940cee Fix no-mistakes warnings: prose out of YAML fence, Rule 16 rename, table cell fix
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-11 18:02:08 +00:00
root 5c3ba31b17 Ship: Koby external-agent ruling (Resolve conflict and merge master)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-11 17:48:36 +00:00
root d0144d6db6 Ship: Koby external-agent ruling (fix api_key_env hygiene + docs)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-08-11 17:22:25 +00:00
6 changed files with 29 additions and 32 deletions
+11
View File
@@ -296,6 +296,7 @@ The following MUST be identical across ALL profiles:
- See `devops-hermes-compression` skill for full reference
### Rule 10: Default Model Must Be `syslog-auto` (All Agents)
- **Koby Exception**: Per captain ruling 2026-08-11, Koby is a DeepSeek-primary external agent; its primary model remains `deepseek-v4-flash` (via api.deepseek.com to preserve DeepSeek-specific reasoning, while other sections follow Rule 10.
- **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
- **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
- `syslog-auto` is the LiteLLM routing model — it load-balances between strix-moe
@@ -423,3 +424,13 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
5. **Set model choice** — Per agent's workload
6. **Verify** — curl all shared endpoints, test the model with the new key
7. **Report** — What was changed, preserved, custom
### Rule 16: Koby Configuration (DeepSeek-primary)
(Ref: See Rule 10 for default model behavior, with Koby exception)
Koby uses a split-model architecture:
- Primary Model: `deepseek-v4-flash` via `api.deepseek.com` (for reasoning)
- Auxiliary Models: `gpu-light` (vision/web_extract) and `syslog-auto` (compression)
- Key Hygiene: `api_key_env` is strictly `LITELLM_API_KEY` or `DEEPSEEK_API_KEY`
- Constraint: Do NOT touch Koby's primary model/provider/compression settings unless explicitly ruled by the captain.
+2 -2
View File
@@ -191,7 +191,7 @@ litellm_settings:
|-------|-----|-----|---------------|------------|--------|-----------------|---------------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
| Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
| Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed (DeepSeek-primary) | `infisical run` | 23:30 UTC Jul 5 |
| Koonimo | 113 | .114 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-08-09 |
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
| Kagenz0 | 105 | .14 | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
@@ -205,7 +205,7 @@ litellm_settings:
|-------|--------------|-------------|--------|
| Mumuni | ✅ harness provider | ✅ auxiliary on /v1 (valid) | ✅ Authenticated (verified 2026-08-09) |
| Tanko | ✅ 5 sections | 0 | ✅ Migrated 2026-08-08, keys 200 |
| Koby | ✅ custom provider (harness name) | — | ✅ External DeepSeek primary (intentional) |
| Koby | ✅ custom provider (harness name) | — | ✅ External DeepSeek primary (intentional, captain ruling 2026-08-11) |
| Koonimo | ✅ .114 (baggy) | — | ✅ 128K context applied 2026-08-09 |
### Systemd Service Pattern (2026-07-11 — vault migration)
+2 -1
View File
@@ -4,7 +4,8 @@ name: pm2-self-heal
description: >
Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner,
spoton-service, zulip-watchdog) and auto-restarts any that are stopped or
errored. Logs every action to the knowledge graph and alerts the owner via
errored. Logs every action to Gitea (SyslogSolution/health-logs — not
knowledge graph, hard rule) and alerts the owner via
Zulip DM on failures.
CRITICAL: Never restart abiba-zulip — it runs this contract.
AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative):
+11 -26
View File
@@ -9,10 +9,11 @@ ZULIP_EMAIL="abiba-bot@chat.sysloggh.net"
ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
OWNER_ZULIP_ID="9"
# Email config
GMAIL_USER="jtabiri@gmail.com"
GMAIL_PASS="rgbuomwcydxwbszd"
EMAIL_TO="jerome@sysloggh.com"
LOG="/root/zulip-health-monitor.log"
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
ISSUES=0
echo "=== Zulip Health Check — $TIMESTAMP ===" >> "$LOG"
notify() {
local severity="$1" msg="$2"
@@ -24,30 +25,14 @@ notify() {
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-d "${form}" > /dev/null 2>&1 || true
# Email alert
local subject="${severity} Zulip Monitor Alert"
python3 -c "
import smtplib
from email.mime.text import MIMEText
m = MIMEText('''${msg}''')
m['From'] = 'abiba@sysloggh.com'
m['To'] = '${EMAIL_TO}'
m['Subject'] = '${subject}'
s = smtplib.SMTP('smtp.gmail.com', 587)
s.starttls()
s.login('${GMAIL_USER}', '${GMAIL_PASS}')
s.sendmail('abiba@sysloggh.com', ['${EMAIL_TO}'], m.as_string())
s.quit()
" 2>/dev/null || true
# Zulip stream post to #agent-hub on topic 'zulip-health'
local stream_content="${severity} Zulip Monitor: ${msg}"
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-d "type=stream\&to=%5B7%5D\&topic=zulip-health\&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote(str()))")" \
> /dev/null 2>&1 || true
}
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
ISSUES=0
LOG="/root/zulip-health-monitor.log"
echo "=== Zulip Health Check — $TIMESTAMP ===" >> "$LOG"
# ── Global: Zulip Server ──
SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \
https://chat.sysloggh.net/api/v1/server_settings \
+1
View File
@@ -324,6 +324,7 @@ Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
- **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
- **Debounce gate**: Skip all checks entirely if recently restarted
4. **Zulip stream delivery** — Alerts posted to #agent-hub on topic 'zulip-health' (stream_id 7) instead of email, keeping DM to captain for critical (🔴) alerts
### Gen 4 (2026-06-29) — Response Pipeline Fix
**Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence.
+2 -3
View File
@@ -75,9 +75,8 @@ Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
## Reporting
Every cycle produces a knowledge graph node:
- Title: `[LEARN] zulip-self-heal: <timestamp>`
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
This contract is RETIRED — health-check logs are NOT knowledge graph content.
No graph nodes are created. Logs go to Gitea (SyslogSolution/health-logs).
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"