Commit Graph
25 Commits
Author SHA1 Message Date
agent-zero 7730cc7c03 feat(litellm): 1.99.1 update coverage, trove agent, nginx /ui /docs path fixes
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
2026-09-11 15:32:20 -04:00
agent-zero 832b184af6 fix(litellm): upgrade contract refs 1.90.0-rc.1 -> 1.99.1 and correct redis role
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
- harness-litellm pinned tag updated to 1.99.1 in infrastructure-control,
  litellm-health, litellm-self-heal container tables
- infrastructure-update MCP per-key limitation note now cites v1.99.1
- harness-redis role corrected: dead 'Router slots, circuit breakers' ->
  'LiteLLM cache + rate-limit state' (router decommissioned 2026-09-11)
- litellm-self-heal Rule 9 wording clarified for cache/rate-limit only

Live verification 2026-09-11: harness-litellm healthy, /health 200/200,
7 models, 46 keys, prisma migrations 127 -> 157.
2026-09-11 15:05:00 -04:00
root a6a459acc0 fix: correct Firecrawl GET / response code from 404 to 200 (PR #64)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-09-10 20:42:15 +00:00
agent-zero 2e70c834cb docs(infrastructure-update): add mandatory digest-pin sweep to Wave 3 verification
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Digest-pinned images (image: repo@sha256:...) are invisible to 'docker compose
pull' — the pin re-pulls the same digest forever, so new releases never appear.
Verified 2026-09-10: audiobookshelf ran 2.34.0 for 7 weeks despite weekly pulls;
un-pinned to :latest, now 2.36.0 (HTTP 200). Dockhand stack (also pinned) was
removed 2026-09-10 as unused (user decision). Weekly task qSOOVzsU now sweeps
for digest pins every run and flags them for user-approved un-pinning.
2026-09-10 06:30:07 -04:00
agent-zero bfdff13ae7 docs(infrastructure-update): point weekly trigger at scheduler task qSOOVzsU
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Weekly Sunday 03:00 America/New_York run is now implemented as Agent Zero
scheduled task 'weekly-fleet-docker-update' (qSOOVzsU) with built-in
post-update service verification and old-image pruning.
2026-09-08 05:27:38 -04:00
agent-zero 8bf32f6f0f update(infrastructure-update): v1.3.0 — full Docker ecosystem coverage from 2026-09-08 live run
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s
- Add 5 Docker ecosystems: docker-vm .7, CT 116 .116, CT 117 (zulip+jitsi),
  hwpve .11 (Authentik), NetBird VPS 72.61.0.17
- Wave 3: add trove-test, docker-stats, monitoring stack, Zulip (pct exec + compose
  recreate preserves zulip_default network), Jitsi, Authentik, NetBird
- Firecrawl test is now POST /v1/search (GET / returns 404 by design)
- Add Authentik 302 + NetBird dashboard 200 checks
- Document harness-litellm 3-5 min cold start after recreate (verified 2026-09-08)
- Add hwpve + VPS compose files to config backup list; success criteria covers 5 hosts
2026-09-08 03:23:03 -04:00
mumuni-bot aac4c7eac3 fix: repoint Mumuni refs in Normal-sensitivity contracts to kagentz CT105/.14
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Third and final PR of the post-migration contract sweep (ref PRs #54, #55).
Normal-sensitivity contracts per AGENTS.md (any registered agent):

- gpu-fleet: key roster row, Mumuni Agent Profile header + status table
  (thermal-safeguard incident note left as historical record)
- infrastructure-update: apt patch table row; Hermes config path block
  corrected to /home/hermes/.hermes + system unit /etc/systemd/system/
  hermes-gateway.service (was /root/.hermes + user unit)
- infrastructure-maintenance: Hermes gateway health-check target
- inference-optimization: apply-agent-compression host + config_path

All values verified live on kagentz 2026-08-29 (hostname, IP, user,
systemd unit, /home/hermes/.hermes). Abiba-owned refs (.24 pi agent,
GPU monitor :9100) and CRITICAL files intentionally untouched - flagged
to Abiba via relay #739.

Refs PR #54, #55; relay #738/#739.
2026-08-30 01:12:22 +00:00
kagentz-bot 44f7008302 docs: reflect hwepve removal from Tabiri cluster (5 nodes + standalone London node)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
- Remove hwepve from cluster member lists/diagrams in infrastructure-control,
  proxmox-monitor, infrastructure-update, litellm-health, mumuni-delegation
- CTs 100 (abiba) and 105 (kagentz) moved to minipve; CT 114 (mumuni) no longer
  exists (Mumuni runs inside Abiba CT 100)
- Add standalone London role + NetBird routing peer description for hwepve
- Update scripts: agent-health-check.py, pct-run.sh, prose-ai-review.sh
- Update disk-gc CT access table, hermes baselines/restore host references
2026-08-15 20:03:02 -04:00
root 8f719ca7c7 fix(contracts): Pulse port 3001→7655 (verified), jdownloader decommissioned from docker-vm → CT 118 LXC (.20)
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Skipped
2026-08-01 20:28:50 +00:00
root b3193e5e1b fix(agent-health): 14 stale-reference fixes (Mumuni CT100/.24, Koby .129, Koonimo CT113, wrapper paths)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-01 14:10:52 +00:00
root f1490b656e no-mistakes(document): docs: fix stale ornith refs to strix-server/strix-moe
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-08-01 13:35:21 +00:00
root 7c0adefdeb Merge PR #38: feat: add Level 0 auto-delete for heartbeat log orphans
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-07-29 21:50:19 +00:00
root 87b4d67067 no-mistakes(review): Fix 5 review findings: duplicate table header, hwepve specs/Wave2 target, dangling ref, Authentik port 2026-07-24 21:31:28 +00:00
root 7d1db62a8e no-mistakes(document): Fix stale CT placements across 5 docs matching infrastructure-control updates 2026-07-24 20:21:03 +00:00
root fa26b7a579 no-mistakes(test): Fixed 7 cross-table inconsistencies: kagentz placement, mumuni placement, stale 5-node references 2026-07-24 20:14:49 +00:00
root c380196fab fix: address review findings
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- Remove kagentz from amdpve node table (belongs on hwepve)
- Fix kagentz pct-run table: hwepve (was amdpve)
- Fix mumuni pct-run table: hwepve (was minipve)
- Fix Zulip recovery command: wrap in single SSH call
2026-07-24 20:11:32 +00:00
root b17c60f997 fix: contract accuracy updates post fleet-wide reboot
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
infrastructure-control.prose.md:
- Add hwepve as 6th Proxmox node
- Fix Gitea IP: .17 (was .110)
- Fix AdGuard IP: .10 on minipve (was .102 on acerpve)
- Fix Abiba placement: hwepve (was amdpve)
- Fix Mumuni placement: hwepve (was minipve)
- Fix Authentik port: add :9000
- Add CT 118 (jdownloader), CT 119 (infisical-vault)
- Add last-verified date (2026-07-24)

infrastructure-update.prose.md:
- Add post-reboot CT sweep procedure
- Add Zulip Docker network recovery steps
- Add WireGuard tunnel verification

scripts/netbird-add-domain.sh:
- New script to register domains in Netbird proxy store.db
2026-07-24 20:01:43 +00:00
root 1c44bf1259 contract updates: ornith decommissioned, 256K→128K context, Mumuni Discord disabled
Change 1: Strix Halo — ornith decommissioned
- gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout
- inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4
- gpu-monitor: ornith status → Strix Halo status
- infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4
- infrastructure-update: ornith→strix-moe via router
- proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe)

Change 2: GPU context 256K→128K fleet-wide
- hermes-agent-baseline: frontmatter description updated
- litellm-health: GPU Fleet Topology table 256K→128K
- litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert
- inference-optimization: compress threshold 256K→128K compact at 85K
- gpu-fleet: instability note updated

Change 3: Mumuni Discord platform disabled
- gpu-fleet: Mumuni platforms: removed discord
2026-07-23 09:00:42 +00:00
root 79d4a73895 docs: lessons learned from 2026-07-12 session
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
2026-07-12 22:49:39 +00:00
root 546ca79849 fix: cron alignment + VM→bare metal drift
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
- pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract)
- pm2-self-heal: added gpu-monitor to tracked PM2 processes
- infra-monitor+control: removed daytime-only restriction (now 24×7)
- infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110
- Added @reboot pm2 resurrect for persistence across reboots
- README: zulip-monitor.sh → agent-health-check.py
- Cleaned up commented-out zulip-monitor cron line

Live actions: gpu-monitor-server added to PM2 (persistent across reboots)
2026-07-09 06:27:48 +00:00
root 776df418ca gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3)
- .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache
- .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV
- .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm
- LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s
- Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma
- Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
2026-07-08 17:25:42 +00:00
Agent Zero f1a031b682 fix: Firecrawl health endpoint is / not /health
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped
2026-07-06 18:21:30 -04:00
Agent Zero f14b2fc04e infra-update contract: v1.1.0 - fix CT→VM refs, remove deprecated stacks
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped
2026-07-06 18:05:16 -04:00
Agent Zero 09ffbf10aa infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths 2026-07-06 18:05:16 -04:00
root 253e19f680 prose: review, fix, and consolidate infrastructure contracts
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00