Commit Graph
6 Commits
Author SHA1 Message Date
root 3b6cf44a30 fix: update all Mumuni IP references from .123 to .24 (inside Abiba CT100)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Mumuni CT114 destroyed. Mumuni now runs inside Abiba CT100 at 192.168.68.24. Updated all contract files and agent-health-check.py.
2026-07-28 08:56:45 +00:00
root 0f572ff9f2 fix: remove mumuni from health check — now inside Abiba CT 100 2026-07-26 12:37:07 +00:00
root de9adb13cf fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Gaps fixed:
- Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
- Agent key lookup uses correct {NAME}_LITELLM_API_KEY format
- New CT liveness check via pct status on PVE nodes
- New config YAML integrity check via yaml.safe_load()
- New wrapper/CLI integrity check (hermes wrapper, infisical path, hermes-real)
- New vault secret non-emptiness check
- Ops escalation: failures produce ALERT lines for cron capture
2026-07-26 12:06:32 +00:00
root 17a77e6b3f fix: fleet config issues from 2026-07-18 relay review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- hermes-agent-baseline: add gpu-dense and gpu-light to models list
- hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add
  vault empty-key guard documentation in Rule 13
- litellm-api-keys: fix pgrep pattern in auditable check
- scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper

Addresses issues found during relay inbox resolution session:
1. pgrep -f 'hermes_cli.main gateway run' matches both the real python
   gateway and the infisical bash wrapper, causing false health readings
2. infisical vault stores empty key silently — no guard/monitoring
3. gpu-dense/gpu-light stable aliases missing from baseline config
2026-07-18 17:39:46 +00:00
root 79d4a73895 docs: lessons learned from 2026-07-12 session
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
2026-07-12 22:49:39 +00:00
root 79855ea9e1 feat: consolidated agent health check — key validation + GPU port conflict + streaming
Single non-disruptive script replacing 7 scattered Zulip health checks.
Checks every 10 min via cron, never restarts anything:
- LiteLLM key validation for all 4 Hermes agents
- GPU port conflict / ghost process detection
- Agent gateway liveness + Zulip streaming status
- Recent gateway error count

Port conflict detection added to all 3 GPU wrappers:
- .8 (qwen): wrapper detects ghost on port 8080 before starting
- .110 (gemma): same pattern
- .15 (ornith): port-cleanup.sh replaces blanket pkill -x llama-server
2026-07-06 02:23:54 +00:00