audit-hermes-config.py Rule 8 required auxiliary.vision.model and
auxiliary.web_extract.model to equal the retired 'gpu-light', so a config
adopting the live canonical 'gpu-vision' FAILED our own audit - the audit was
enforcing a dead alias (400 Invalid model name). Rule 8 now requires
gpu-vision; retired names gpu-light/crew-auto join the raw-name rejection set;
the guidance message names the live aliases.
Sweep of the remaining references: gpu-self-heal stops canonicalizing
gpu-light; hermes-config-template, hermes-agent-baseline, hermes-key-enforcement,
inference-optimization, litellm-client-timeouts and gpu-fleet now use the live
gpu-vision alias. Where a file restated model/rpm/weight/fallback state it now
points at CT 116 /opt/inference-harness/litellm_config.yaml instead of
duplicating it. koby's .129 config is report-only and recorded, not edited.
Adds tests/test_audit_hermes_config_alias.py: executes the audit CLI and asserts
gpu-vision passes while gpu-light and gemma-4-12b fail.
- Execution step 2 now documents the public edge and the backend edge as two
distinct surfaces: public serves /ui/ and /docs (404 on the /litellm/ prefix),
backend http://192.168.68.116 serves /litellm/ui/ and /litellm/docs (with /ui/
and /docs as 301 helpers). Each probe names its surface.
- GPU topology: ocu-llm RTX 5070 now serves gpu-vision (gemma-4-12b retired).
- Fallback/timeout table rewritten to the live router_settings.fallbacks chains.
- Step 7 model list: gemma-4-12b -> gpu-vision, with a key-scoped /v1/models note
and the 2026-09-12 master-key registry snapshot.
- Remove adapter process check and restart logic
- Keep A2A probe (port 80, HTTP code check)
- The adapter code at /a0/usr/kagentz-zulip/ no longer exists
- Captain's ruling: Zulip communication with agent zero is not priority
- Load API key from durable file /root/.abiba-workspace/secrets/litellm-key.txt (works in cron)
- Fix http_get to use Bearer token instead of Basic Auth for API endpoint check
- All 6 LiteLLM checks now pass (was 5/6)
- Changed A2A probe from http://127.0.0.1:8001/.well-known/agent.json to
http://127.0.0.1:80/a2a/ inside agent-zero container
- Port 8001 does not exist inside container (nothing listens there)
- Port 80 maps to external port 50080; returns 401 (auth-gated, alive by design)
- Updated A2A_URL in adapter.py restart command to use port 80 instead of 8001
- Verified: probe now returns 401 (auth-gated) instead of 000 (connection refused)
- Before: kagentz A2A reported DOWN on every run (false positive due to stale port)
- After: kagentz A2A reports ✅ A2A alive (auth-gated 401 = healthy)
The monitor's Abiba leg parsed the :9200/health payload at the top level
(d.get('connected',False)) while the pi Zulip extension serves connection
state NESTED at zulip.connected / zulip.last_error (verified against the
extension's startHealthServer handler and the live endpoint). PI_CONNECTED
was therefore always False and every run took the DISCONNECTED path,
restarting a healthy bot: pm2 restarts=8 with the process created
2026-09-09T09:35:09Z, four ❌ Abiba verdicts today (04:23/05:35/06:55/09:35
UTC) and zero ✅, while the Zulip server answered HTTP 200 and the bot kept
heartbeating. The watchdog was the fault, not the connection.
Changes (Abiba leg only; every other leg byte-identical):
- Read the real nested shape: zulip.connected, zulip.last_error and
zulip.messages_processed. The old retry_count branch is DROPPED — the
payload exposes no retry counter (the extension keeps retryCount internal
and never serialises it), so the branch is fabricated and cannot stay.
- Fail-safe restart decision: a fetch error (HTTP 000), non-2xx response,
empty/unparseable body, or payload missing a boolean zulip.connected is a
clearly-labelled PROBE FAILURE (🟠 alert + ⚠️ log line naming reason and
HTTP code) and NEVER calls pm2 restart. pm2 restart runs only on
affirmative zulip.connected=false (🔴/❌ path unchanged in wording).
connected=true with last_error keeps the degraded 🟡 warn-no-restart path.
Regression test (new tests/zulip-monitor-abiba.sh, 36 checks): extracts the
real Abiba leg from between the # -- abiba-leg-start/-end markers in the
shipped script and executes it verbatim with stubbed curl/notify/pm2 against
fixtures of the real payload shape — asserts connected=true → no restart,
connected=false → restart, and empty/garbage/missing-key/non-boolean/HTTP
000/HTTP 500 bodies → probe-failure alert with zero restarts. The suite
fails loudly on the pre-fix base and on a marker-intact top-level-parse
variant, so this class of bug cannot return silently.
The #agent-hub / zulip-health stream post in notify() encoded content with
python quote(str()) (always empty), so every stream alert posted empty
content and Zulip rejected it silently behind '|| true'. The merged body
also used backslash-escaped ampersands inside double quotes, which curl
transmits literally (type=stream\ -> Zulip 400 'Invalid type').
Stream post now percent-encodes the real message text by piping it through
the encoder (locale-proof: quote_from_bytes on stdin.buffer), uses plain &
separators, and on failure appends one WARN line to the monitor log with
the curl exit status instead of silently swallowing it. Delivery failure
stays non-fatal and unretried. DM path, probes, alive rule, exit codes and
log format unchanged.
The no-mistakes document step had auto-updated the .8 systemd unit name
(llama-server -> llama-chat-api.service) in gpu-fleet.prose.md and
infrastructure-control.prose.md. Those contracts are co-gated with another
agent and cannot be amended inside a script-scope task (firstmate decision
2026-09-08, key prose-doc-step-scope); the unit-name doc sync is separate
follow-up work. This restores both files to their origin/master content.
The litellm-self-heal.prose.md agent-health-check v3/env.sh description
update (same document commit) is intentional and kept.
- GPU unit repoint verified live 2026-09-08: .8 rtx3090 probes
llama-chat-api.service (stale llama-server unit read inactive -> false
UNREACHABLE for a healthy process); .110 keeps llama-server.service
(ocu-llm VM), .15 keeps strix-server.service. is-active no longer
swallowed as SSH failure (|| true).
- check_agents: bind pid before the summary f-string so the non-report-only
path (abiba/koonimo) no longer raises UnboundLocalError (line 305 crash).
- abiba key leg: read LITELLM_API_KEY from /root/.pi/agent/env.sh (#735
moved creds out of shared /root/.bashrc); 'abiba NO KEY' gone on healthy
setup.
Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only), not 0.0.0.0.
They must be probed from .116 via SSH. Prometheus and Grafana remain 0.0.0.0 (LAN-reachable).
Fixes false alarm from remote probe showing connection-refused (by design for localhost binds).
Fixes the canned 'API UNREACHABLE (HTTP 000)' reporting. Adds an explicit
check-health execution section (Zulip POST, pm2, GPU exporters, Prometheus,
Grafana, LiteLLM probes) with a hard RUN LIVE, NEVER ECHO rule, mirroring
the gpu-monitor contract pattern. Captain priority 2026-08-22.
root