root
1c44bf1259
contract updates: ornith decommissioned, 256K→128K context, Mumuni Discord disabled
...
Change 1: Strix Halo — ornith decommissioned
- gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout
- inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4
- gpu-monitor: ornith status → Strix Halo status
- infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4
- infrastructure-update: ornith→strix-moe via router
- proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe)
Change 2: GPU context 256K→128K fleet-wide
- hermes-agent-baseline: frontmatter description updated
- litellm-health: GPU Fleet Topology table 256K→128K
- litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert
- inference-optimization: compress threshold 256K→128K compact at 85K
- gpu-fleet: instability note updated
Change 3: Mumuni Discord platform disabled
- gpu-fleet: Mumuni platforms: removed discord
2026-07-23 09:00:42 +00:00
Abiba
cd52deda92
contracts: stable alias chain + no-master-key-for-inference rule
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Agent configs aligned to stable aliases (gpu-light/gpu-dense/strix-moe/syslog-auto):
- Tanko fixed: base_url :4000->/v1 (nginx), max_context_window 131072->262144,
aux.compression gemma->strix-moe, aux gemma->gpu-light
- Mumuni aligned: aux gemma->gpu-light, delegation/x_search qwen->gpu-dense
- Koby/Koonimo already used gpu-light (validated)
- Zero raw model names remain in any agent config
No-master-key rule: litellm_proxy_master_key (sk-litellm-...) is admin-only
(/key/list, /key/generate). All inference uses agent keys. Health-check model
tests use dedicated monitor key (/etc/litellm-monitor.env on CT 116).
Fixed: daily-infra-report.py, gpu-self-heal.py, litellm-health-check.sh.
Fixed wrong key-scope note: qwen3.6-35B-udq4 IS in scope for baggy/koby/mumuni/
abiba-pi keys (not 'NOT in allowlist' as previously documented).
2026-07-16 17:39:05 +00:00
Abiba
97f1cd77e4
litellm-self-heal: document 2026-07-16 ops sync (script fixes, systemd gpu-monitor, live-key agent check)
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Health-check script: critical-only gpu-fleet alerting, strix-moe model test.
gpu-monitor: systemd unit (was bare &), Strix counted in gpu_count, VRAM
thresholds raised (93/97 — 256K ctx steady-state ~96% on RTX 3090).
agent-health-check.py: reads live LITELLM_API_KEY from gateway env (no hardcoded keys).
daily-infra-report.py: stale SYNTHETIC key → env-based master key.
2026-07-16 17:23:37 +00:00
Abiba
dc572889f8
contracts: sync to ground truth — ornith-1.0-35b→strix-moe, 256K all GPUs, real LiteLLM timeouts
...
Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts:
- AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist
- All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1)
- LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s)
- Added LiteLLM model surface + key scoping to litellm-self-heal
- Patched health-check script path ref
Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal,
zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline,
delegation-prose-contract, mumuni-delegation-prose-contract
2026-07-16 17:02:44 +00:00
root
af41f8f57a
fix: remaining template YAML and topology table corrections
...
- hermes-config-template: Template YAML compression model gemma→ornith.
Context guidance updated (qwen=256K, only gemma limited to 131K).
- litellm-self-heal: GPU topology table fixed (RTX 3090: 128K→256K, parallel 2→1)
2026-07-12 22:55:24 +00:00
root
be02b0e843
fix: stale dates, context values, and architecture references
...
- gpu-fleet: Updated all context references (128K→256K for RTX 3090).
Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram
(router labeled deprecated/not in path). Corrected VRAM values.
Updated dates: June→July 2026.
- hermes-config-template: Fixed description (removed incorrect '128K on
NVIDIA' claim). Updated to reflect verified 256K on RTX 3090.
- litellm-self-heal: Updated last-verified date.
- litellm-api-keys: Updated last-verified date.
2026-07-12 22:54:30 +00:00
root
79d4a73895
docs: lessons learned from 2026-07-12 session
...
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
Updated context values (RTX 3090=256K, not 128K). Added api-key
standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
L2: Fallback chain cascading failure loop detection
L3: Verify running state, not documentation
L4: Infisical fallback requirement (.env must have uncommented key)
L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
2026-07-12 22:49:39 +00:00
root
d829f595b4
fix: IP drift and container count from live testing
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 13m1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
Testing the consolidated litellm-self-heal contract against live infra found:
- IP drift: pi host is .24, not .65 (confused with ra-h-os bridge)
- Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations)
- Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables
- Container count: 10 running on CT 116, not 8 as contract stated
- Added harness-docker-stats + harness-pve-exporter to litellm-self-heal
- Updated container health check list
Verified: GPU monitor restarted on .24:9100, ornith model inference working,
all 10 containers healthy.
2026-07-09 06:23:44 +00:00
root
053d30ce7c
feat: consolidate litellm-health into litellm-self-heal
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m53s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
- litellm-self-heal: merged health check as §Health Check section,
includes full architecture diagram (v4.0.0), GPU topology, timeout
tables, container list, and execution flow. Eliminates duplication
between the two contracts.
- litellm-health: marked DEPRECATED, retained for reference
- README.md: updated with consolidated contract, deprecation notice,
disk-gc-threat-response, and infrastructure-monitoring status
2026-07-09 06:01:48 +00:00
root
a2c5a14c74
fix: LiteLLM contracts — align with 2026-07-08 architecture changes
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m29s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
- litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly),
GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s),
Strix Halo: CPU→Vulkan, context sizes + parallel slots added
- litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED
(router not in path), GPU topology synced with gpu-fleet
- litellm-api-keys: reference gpu-fleet as source of truth for keys,
removed qwen3.6-35B-A3B from model list (never deployed)
2026-07-09 05:37:16 +00:00
root
253e19f680
prose: review, fix, and consolidate infrastructure contracts
...
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)
All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00
Abiba
c1d993fe01
fix: GPU table correction + OpenProse v0.15 alignment + self-heal rules 7-9
...
- hello-world: Ensures -> Maintains (v0.15.0 rename).
- zulip-mention-reliability: fold Success Criteria into Maintains postconditions.
- litellm-self-heal: swap gemma/qwen GPU hosts to match actual deployment;
add Strix Halo 64GB UMA detail; add Rules 7 (roster stuck), 8 (agent keys
invalid/401), 9 (stale Redis active counter).
2026-07-02 20:39:17 +00:00
root
aa55634571
feat: tighten contracts with GPU fleet topology, model routing, Strix Halo details
...
litellm-health: added GPU fleet topology table, model routing map,
3-tier GPU checks (reachability, VRAM, temp, test inference)
litellm-self-heal: added GPU fleet reference, rules for GPU
unreachable and model not responding
Infrastructure topology now fully documented:
- amdpve: Strix Halo CPU (35B, 16 threads, 262K ctx, llama-server)
- llm-gpu: RTX 3090 24GB (Dense tier)
- ocu-llm: RTX 5070 12GB (MoE + Light tiers)
2026-06-29 04:07:05 +00:00
root
41bd3c5748
feat: add daily/weekly digest reporting to self-heal contract
...
- Hourly cycle reports: fix alerts, escalation alerts, 10-cycle all-clear
- Daily digest: 24h summary with counts, top actions, uptime
- Weekly digest: 7-day summary + KG node for trending
- All delivered via Zulip DM to owner
2026-06-26 15:23:21 +00:00
root
fb27c99c3d
feat: litellm-self-heal — proactive remediation responsibility
2026-06-26 14:51:04 +00:00