root
947e8b24e0
fix(zulip-monitor): stream alert body now carries real content, plain & params, WARN on delivery failure
...
The #agent-hub / zulip-health stream post in notify() encoded content with
python quote(str()) (always empty), so every stream alert posted empty
content and Zulip rejected it silently behind '|| true'. The merged body
also used backslash-escaped ampersands inside double quotes, which curl
transmits literally (type=stream\ -> Zulip 400 'Invalid type').
Stream post now percent-encodes the real message text by piping it through
the encoder (locale-proof: quote_from_bytes on stdin.buffer), uses plain &
separators, and on failure appends one WARN line to the monitor log with
the curl exit status instead of silently swallowing it. Delivery failure
stays non-fatal and unretried. DM path, probes, alive rule, exit codes and
log format unchanged.
2026-09-09 02:47:44 +00:00
root
1b8186f6b9
no-mistakes(document): docs: reword Tanko .122 SSH dependency claims in zulip-health
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-09-08 13:00:05 +00:00
root
9dd0cb18d5
no-mistakes(document): docs: align zulip-health tanko snapshot schema to dsh-web probes
2026-09-08 12:49:34 +00:00
root
4ac60e14a3
no-mistakes(review): fix zulip-monitor Tanko probe fallback-echo contamination in down detection
2026-09-08 12:38:45 +00:00
root
2e0b737f2d
revert: drop co-gated prose edits (gpu-fleet, infrastructure-control) — script-only scope per captain decision
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
The no-mistakes document step had auto-updated the .8 systemd unit name
(llama-server -> llama-chat-api.service) in gpu-fleet.prose.md and
infrastructure-control.prose.md. Those contracts are co-gated with another
agent and cannot be amended inside a script-scope task (firstmate decision
2026-09-08, key prose-doc-step-scope); the unit-name doc sync is separate
follow-up work. This restores both files to their origin/master content.
The litellm-self-heal.prose.md agent-health-check v3/env.sh description
update (same document commit) is intentional and kept.
2026-09-08 12:30:13 +00:00
root
bbe9ee533f
no-mistakes(document): Docs updated for .8 unit repoint and abiba env.sh sourcing
2026-09-08 12:18:14 +00:00
root
4684ee64e0
no-mistakes(review): Align zulip-monitor Tanko probe and B2/B3 criteria to dsh-web
2026-09-08 12:10:58 +00:00
root
af3d364242
no-mistakes(review): Bracket pgrep patterns to stop ssh wrapper self-match
2026-09-08 12:04:01 +00:00
root
2716e55c16
fix(agent-health): repoint .8 GPU unit, fix pid UnboundLocalError, source abiba key from env.sh
...
- GPU unit repoint verified live 2026-09-08: .8 rtx3090 probes
llama-chat-api.service (stale llama-server unit read inactive -> false
UNREACHABLE for a healthy process); .110 keeps llama-server.service
(ocu-llm VM), .15 keeps strix-server.service. is-active no longer
swallowed as SSH failure (|| true).
- check_agents: bind pid before the summary f-string so the non-report-only
path (abiba/koonimo) no longer raises UnboundLocalError (line 305 crash).
- abiba key leg: read LITELLM_API_KEY from /root/.pi/agent/env.sh (#735
moved creds out of shared /root/.bashrc); 'abiba NO KEY' gone on healthy
setup.
2026-09-08 11:08:28 +00:00
root
5313e6b9ba
fix: tanko zulip-health probes via amdpve pct exec (dsh-web loopback :3080 + public-URL fallback)
2026-09-08 10:26:56 +00:00
root
801de0a25c
fix: correct proxmox-monitor probe endpoints to use localhost for exporters
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only), not 0.0.0.0.
They must be probed from .116 via SSH. Prometheus and Grafana remain 0.0.0.0 (LAN-reachable).
Fixes false alarm from remote probe showing connection-refused (by design for localhost binds).
2026-09-08 08:41:21 +00:00
root
36ae464f59
fix: probe alignment across multiple monitoring contracts
...
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
- infra-monitoring: Added Router+LiteLLM+PVE API probes with expected HTTP codes (200/401/500)
- gpu-monitor: Added explicit check-health section with Router+LiteLLM probe commands
- proxmox-monitor: Added check-health section for Prometheus/Grafana/exporters (all bound to 0.0.0.0)
- zulip-health: Fixed A2A port from :8001 to :50080 and added auth-gated expectation (401)
Closes: infra-monitoring-probe-alignment, gpu-monitor-probe-alignment, proxmox-monitor-probe-alignment, zulip-health-probe-alignment
2026-09-08 07:20:41 +00:00
root
a3e97ce72b
fix: add Router+LiteLLM+PVE API probes to infrastructure-monitoring check-health
...
- Added Router health probe (http://192.168.68.116/health ) — expected 200
- Added LiteLLM health probe (http://192.168.68.116/litellm/health ) — expected 200
- Added PVE API probe (https://192.168.68.116:8006/api2/json ) — 401 expected for unauthenticated
- Clarified that 401 means API is up, 000 means unreachable, 500 means API down
Closes: infra-monitoring-probe-alignment
2026-09-08 07:20:41 +00:00
root
cc5fe0991c
Move Zulip bot creds from inline to env/vault reference
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
The abiba-bot@chat.sysloggh.net:KEY is now sourced from
/etc/litellm-monitor.env as ZULIP_USER and ZULIP_BOT_KEY.
2026-09-08 06:18:21 +00:00
root
e8b9f990b2
Fix zulip-monitor.sh LOG variable scope bug
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Move LOG, TIMESTAMP, and ISSUES declarations from inside notify() to
script scope so they are available throughout the script.
2026-08-30 12:24:06 +00:00
root
2dfc3e1530
Merged PR #50 : fix/gpu-dense-docs
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-08-30 12:16:39 +00:00
root
79af0ae7a3
Merged PR #52 : fix/tanko-runtime
2026-08-30 12:16:28 +00:00
root
994ae1b7ac
fix: standardize base URLs to authenticated /litellm/v1 path in remaining contracts
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
- Update hermes-agent-baseline.prose.md: change /v1 → /litellm/v1 in all base_url references
- Update hermes-key-enforcement.prose.md: change workaround section base_url to /litellm/v1
- Ensures all contracts consistently use the authenticated LiteLLM path
- DeepSeek harness exemption remains documented for external providers
2026-08-28 15:32:24 +00:00
root
d6ad016ac9
Add check-health section with live probes to infrastructure-monitoring contract
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 16s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 13s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Fixes the canned 'API UNREACHABLE (HTTP 000)' reporting. Adds an explicit
check-health execution section (Zulip POST, pm2, GPU exporters, Prometheus,
Grafana, LiteLLM probes) with a hard RUN LIVE, NEVER ECHO rule, mirroring
the gpu-monitor contract pattern. Captain priority 2026-08-22.
2026-08-22 23:59:28 +00:00