Commit Graph
12 Commits
Author SHA1 Message Date
Abiba dc572889f8 contracts: sync to ground truth — ornith-1.0-35b→strix-moe, 256K all GPUs, real LiteLLM timeouts
Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts:
- AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist
- All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1)
- LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s)
- Added LiteLLM model surface + key scoping to litellm-self-heal
- Patched health-check script path ref

Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal,
zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline,
delegation-prose-contract, mumuni-delegation-prose-contract
2026-07-16 17:02:44 +00:00
root af41f8f57a fix: remaining template YAML and topology table corrections
- hermes-config-template: Template YAML compression model gemma→ornith.
  Context guidance updated (qwen=256K, only gemma limited to 131K).
- litellm-self-heal: GPU topology table fixed (RTX 3090: 128K→256K, parallel 2→1)
2026-07-12 22:55:24 +00:00
root be02b0e843 fix: stale dates, context values, and architecture references
- gpu-fleet: Updated all context references (128K→256K for RTX 3090).
  Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram
  (router labeled deprecated/not in path). Corrected VRAM values.
  Updated dates: June→July 2026.
- hermes-config-template: Fixed description (removed incorrect '128K on
  NVIDIA' claim). Updated to reflect verified 256K on RTX 3090.
- litellm-self-heal: Updated last-verified date.
- litellm-api-keys: Updated last-verified date.
2026-07-12 22:54:30 +00:00
root 79d4a73895 docs: lessons learned from 2026-07-12 session
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
2026-07-12 22:49:39 +00:00
root d829f595b4 fix: IP drift and container count from live testing
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 13m1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
Testing the consolidated litellm-self-heal contract against live infra found:
- IP drift: pi host is .24, not .65 (confused with ra-h-os bridge)
  - Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations)
  - Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables
- Container count: 10 running on CT 116, not 8 as contract stated
  - Added harness-docker-stats + harness-pve-exporter to litellm-self-heal
  - Updated container health check list

Verified: GPU monitor restarted on .24:9100, ornith model inference working,
all 10 containers healthy.
2026-07-09 06:23:44 +00:00
root 053d30ce7c feat: consolidate litellm-health into litellm-self-heal
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m53s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
- litellm-self-heal: merged health check as §Health Check section,
  includes full architecture diagram (v4.0.0), GPU topology, timeout
  tables, container list, and execution flow. Eliminates duplication
  between the two contracts.
- litellm-health: marked DEPRECATED, retained for reference
- README.md: updated with consolidated contract, deprecation notice,
  disk-gc-threat-response, and infrastructure-monitoring status
2026-07-09 06:01:48 +00:00
root a2c5a14c74 fix: LiteLLM contracts — align with 2026-07-08 architecture changes
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m29s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
- litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly),
  GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s),
  Strix Halo: CPU→Vulkan, context sizes + parallel slots added
- litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED
  (router not in path), GPU topology synced with gpu-fleet
- litellm-api-keys: reference gpu-fleet as source of truth for keys,
  removed qwen3.6-35B-A3B from model list (never deployed)
2026-07-09 05:37:16 +00:00
root 253e19f680 prose: review, fix, and consolidate infrastructure contracts
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00
Abiba c1d993fe01 fix: GPU table correction + OpenProse v0.15 alignment + self-heal rules 7-9
- hello-world: Ensures -> Maintains (v0.15.0 rename).
- zulip-mention-reliability: fold Success Criteria into Maintains postconditions.
- litellm-self-heal: swap gemma/qwen GPU hosts to match actual deployment;
  add Strix Halo 64GB UMA detail; add Rules 7 (roster stuck), 8 (agent keys
  invalid/401), 9 (stale Redis active counter).
2026-07-02 20:39:17 +00:00
root aa55634571 feat: tighten contracts with GPU fleet topology, model routing, Strix Halo details
litellm-health: added GPU fleet topology table, model routing map,
3-tier GPU checks (reachability, VRAM, temp, test inference)

litellm-self-heal: added GPU fleet reference, rules for GPU
unreachable and model not responding

Infrastructure topology now fully documented:
- amdpve: Strix Halo CPU (35B, 16 threads, 262K ctx, llama-server)
- llm-gpu: RTX 3090 24GB (Dense tier)
- ocu-llm: RTX 5070 12GB (MoE + Light tiers)
2026-06-29 04:07:05 +00:00
root 41bd3c5748 feat: add daily/weekly digest reporting to self-heal contract
- Hourly cycle reports: fix alerts, escalation alerts, 10-cycle all-clear
- Daily digest: 24h summary with counts, top actions, uptime
- Weekly digest: 7-day summary + KG node for trending
- All delivered via Zulip DM to owner
2026-06-26 15:23:21 +00:00
root fb27c99c3d feat: litellm-self-heal — proactive remediation responsibility 2026-06-26 14:51:04 +00:00