Commit Graph
12 Commits
Author SHA1 Message Date
agent-zero c26255f5ff fix(router): decommission legacy GPU router across contracts and fleet scripts
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Router container, image, and config are purged on CT 116 (verified: no
container, no image, inference-harness-router:latest removed, :9000 free,
11 containers healthy, 7 models, live syslog-auto completion OK). Updates:

- gpu-fleet.prose.md: topology diagram rebuilt without the router tier
- gpu-monitor / gpu-self-heal / litellm-health / litellm-self-heal: router steps dropped
- infrastructure-control.prose.md: container inventory + litellm row corrected
- scripts/daily-infra-report.py, scripts/prose-ai-review.sh: host-scoped checks

Verified: 40/40 anchors matched uniquely, 0 U+FFFD across changed files,
daily-infra-report.py compiles, no stray harness-router references remain.
2026-09-11 14:20:18 -04:00
abiba-bot 194e256ac5 no-mistakes(review): Harden health-check provenance, infisical verification, report-only JSON, tests 2026-09-10 01:35:44 +00:00
abiba-bot c66d9c1e20 fix: probe-drift round 2 — stale monitoring expectations (health check, PVE API, GPU port 80, report provenance)
Second probe-drift correction pass after #65/#66/#68. All four legs were stale
consumer expectations, not live faults.

1. scripts/agent-health-check.py (v4)
   - abiba declared pi-only runtime (harness purge): Hermes-era gateway, config
     and wrapper legs are skipped instead of failing.
   - koby declared report_only (captain ruling 2026-08-17, Rule 17): every koby
     leg is detected and reported, never counted as a fleet failure or repaired.
   - koby's CT 111 mapping corrected to storepve (.6); the old amdpve mapping
     made `pct status 111` fail and read as ct-unreachable.
   - wrapper check no longer FAILs .env-based wrappers that legitimately never
     invoke infisical (koonimo).
   - keys load in main() (load_agent_keys) so the module is importable/testable.
   - every run prints absolute execution provenance (script + cwd), in the header
     and in --json.
   Before: 6 FAILURE(S). After: 0 failures, koby reported read-only.

2. infrastructure-monitoring.prose.md
   - PVE API probe repointed from CT 116 (no pveproxy, 000) to the five real
     nodes on https://<node>:8006/api2/json/version, all 401 = alive.
   - any-HTTP-response liveness rule added (401/3xx alive; 000/timeout = DOWN).
   - LiteLLM health documented as 301 -> /litellm/health/liveliness, not bare 200.

3. gpu-monitor.prose.md
   - GPU health probes on :8080 (or router /health/unified); bare port 80 on a
     GPU host is forbidden (no listener -> false DEGRADED).
   - router /health/unified 301 -> /gpu/gpu-data documented as alive.
   - port-discipline + liveness rule + direct-fallback execution step.

4. Report provenance (all contracts)
   - docs/AUTHORING-GUIDE.md documents the rule; scripts/prose-lint.sh enforces
     that any **Report format** contract states an absolute path (pwd -P).
   - provenance added to gpu-monitor, infrastructure-monitoring, proxmox-monitor.

Tests: tests/test_probe_drift.py (23 passed); prose-lint.sh clean + shellcheck
clean; CI frontmatter validation passes. Evidence with per-leg before/after and
absolute paths: docs/probe-drift-round2-evidence.md.
2026-09-10 01:23:54 +00:00
root a3e97ce72b fix: add Router+LiteLLM+PVE API probes to infrastructure-monitoring check-health
- Added Router health probe (http://192.168.68.116/health) — expected 200
- Added LiteLLM health probe (http://192.168.68.116/litellm/health) — expected 200
- Added PVE API probe (https://192.168.68.116:8006/api2/json) — 401 expected for unauthenticated
- Clarified that 401 means API is up, 000 means unreachable, 500 means API down

Closes: infra-monitoring-probe-alignment
2026-09-08 07:20:41 +00:00
root cc5fe0991c Move Zulip bot creds from inline to env/vault reference
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
The abiba-bot@chat.sysloggh.net:KEY is now sourced from
/etc/litellm-monitor.env as ZULIP_USER and ZULIP_BOT_KEY.
2026-09-08 06:18:21 +00:00
mumuni-bot 403fbcdd9f fix: remove duplicated ## Execution block in infrastructure-monitoring
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 15s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
PR #50's content was squash-merged out-of-band (2dfc3e1, 2026-08-30) while
the PR itself stayed open, double-applying the check-health/Execution
section. This drops the second copy; keeps one canonical
Execution -> check-health -> Phases 1-4 -> Verification Commands flow.
2026-09-07 07:08:49 +00:00
root 2dfc3e1530 Merged PR #50: fix/gpu-dense-docs
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-08-30 12:16:39 +00:00
root d6ad016ac9 Add check-health section with live probes to infrastructure-monitoring contract
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 16s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 13s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Fixes the canned 'API UNREACHABLE (HTTP 000)' reporting. Adds an explicit
check-health execution section (Zulip POST, pm2, GPU exporters, Prometheus,
Grafana, LiteLLM probes) with a hard RUN LIVE, NEVER ECHO rule, mirroring
the gpu-monitor contract pattern. Captain priority 2026-08-22.
2026-08-22 23:59:28 +00:00
root 68f9ebff74 Ship: monitoring contract fixes (2026-08-09)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-09 21:57:44 +00:00
root 97977f320c fix: contract improvements from 2026-07-09 run log review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m43s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
- pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs
- disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope,
  remove jitsi (decommissioned), update access matrix with KVM VM section
- infrastructure-control: docker-vm container count 11→16, add Trove agents stack
- infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)
2026-07-09 05:32:00 +00:00
root 0d0215e9dd prose: annotate infrastructure-monitoring with current deployment status
- Core stack (Prometheus + Grafana + pve/node/docker exporters) already
  deployed by proxmox-monitor contract (Jul 2)
- GPU exporters (.8/.110/.15) NOT yet live — gpu-exporter crash-loops,
  sidecars never deployed
- Strike nginx /monitoring/ proxy suggestion (sub-path pattern already
  tried and reverted for /grafana/ per proxmox-monitor)
- Add cross-reference: this contract defines target state, proxmox-monitor
  is the as-built
2026-07-04 21:55:43 +00:00
Abiba bb20637b32 fix: align contract name fields to filenames (litellm-health, infrastructure-monitoring)
- litellm-health: 'check-litellm-health' -> 'litellm-health'
- infrastructure-monitoring: 'deploy-monitoring-stack' -> 'infrastructure-monitoring'
Fixes 'prose run <filename>' mismatch. No external references broken.
2026-07-02 20:35:44 +00:00