1. Remove vestigial ZULIP_API_KEY requirement:
- /api/v1/server_settings is a PUBLIC endpoint (verified HTTP 200 with or without credential)
- No Zulip API key is required for this call
- If a future leg genuinely needs abiba-bot's key, it must prove it with a 200 from
/api/v1/users/me as abiba-bot and label itself degraded when it cannot
- Never fall back to the vault's shared ZULIP_API_KEY
2. Make failed sends exit non-zero:
- A degraded leg (no credential configured) must stay exit 0
- A failed send (attempted and failed) must exit 1
- This distinguishes 'not configured' from 'attempted and failed'
Test evidence:
- No-credential run: exit 0, digest still produced
- Wrong password: exit 1, labelled SMTP error
- grep -n ZULIP_API_KEY: only comment reference remains
- ZULIP_API_KEY: no longer SystemExit, now reports 'credential-missing: ZULIP_API_KEY'
- EMAIL_PASSWORD: no longer sys.exit(1), now appends to DEGRADED_LEGS and returns success
- PVE API: fixed None check in storage section
- Summary: reports degraded legs before summary
This allows the digest to be produced and emailed even when credentials are missing,
while still explicitly logging which legs are degraded.
Test: empty env produces JSON report + degraded leg labels, no SystemExit.
Scripts that had hardcoded credentials:
- scripts/zulip-monitor.sh:12 (was ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8")
- scripts/daily-infra-report.py:25 (was ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8")
Both now read from environment variable ZULIP_API_KEY (set by vault-backed start script)
with loud failure if not present.
Other credentials in scripts/:
- capture-dsh-token.sh: uses TOKEN variable with fallback (not a secret)
- pm2-self-heal.sh: reads TELEGRAM_BOT_TOKEN from /root/.pi/agent/extensions/telegram/.env (acceptable)
- prose-ai-review.sh: uses GITEA_TOKEN from .env file with LITELLM_KEY fallback (not secrets)
No other hardcoded credentials found.
Proof of behavior:
With ZULIP_API_KEY set:
bash scripts/zulip-monitor.sh → Server: HTTP 200 (authenticated)
python3 scripts/daily-infra-report.py --json → Collecting infrastructure data...
Without ZULIP_API_KEY set:
bash scripts/zulip-monitor.sh → "ZULIP_API_KEY not set — refusing to run with no credential"
python3 scripts/daily-infra-report.py --json → "ZULIP_API_KEY not set — refusing to run with no credential"
Cred source: environment variable ZULIP_API_KEY (set by vault-backed start script)
No key rotation (that is a separate decision).
- Load API key from durable file /root/.abiba-workspace/secrets/litellm-key.txt (works in cron)
- Fix http_get to use Bearer token instead of Basic Auth for API endpoint check
- All 6 LiteLLM checks now pass (was 5/6)
Captain ruling 2026-09-10: Mumuni moved off this host onto her own container
(kagentz CT 105 on minipve, 192.168.68.14, dedicated `hermes` user) and is
monitored from her side. This host must not monitor anything Mumuni.
The stale probes fired false alerts repeatedly:
* scripts/zulip-monitor.sh ssh'd to root@192.168.68.24 for the
decommissioned deployment's ~/.hermes/gateway_state.json, read "unknown"
on every run, and posted a 🔴 "Mumuni (Hermes) Zulip state: unknown" DM +
#agent-hub stream alert each cycle.
* scripts/daily-infra-report.py published a matching "mumuni:unknown" row in
every digest.
Changes:
* zulip-monitor.sh: delete the "Platform B: Hermes (Mumuni)" leg and its
notify; keep the Zulip-server, Platform A pi/Abiba (bridge), Platform B
Tanko and Platform C Agent Zero legs. A comment records why the leg is
retired so it is not re-added. Also fixes SC2155 so shellcheck is clean.
* daily-infra-report.py: delete the .24 ~/.hermes/gateway_state.json agent
probe, its hermes --version probe, and the now-dead mumuni render branch.
Abiba (CT 100, its own .24 address) and Tanko legs unchanged; the PVE API
token name is untouched.
* zulip-health.prose.md (v3.1.0): drop the Mumuni-only B4 gateway-process,
B5 heartbeat and B6 response-delivery steps and the stale 192.168.68.24
references; state explicitly that Mumuni is not monitored from this host.
Tanko/Agent-Zero/bridge steps retained.
* agent-health-check.py: correct the v2 changelog roster comment that still
placed mumuni at .24/CT100. No behavior change — the mumuni probe was
already absent from the AGENTS dict; v5 changelog notes the correction.
Tests: tests/test_mumuni_monitor_removal.py pins the removal structurally and
behaviorally — the shipped zulip-monitor.sh is run in a sandbox (only its LOG
constant rewritten) with stub ssh/curl on PATH; the ssh stub records every
target host, so "never reaches .24" and "no Mumuni notify even on the alert
path" are asserted from observed behavior. A mutation check (re-inject the old
leg) fails the suite, so the guarantee is not vacuous.
root
abiba-bot