docs: lessons learned from 2026-07-12 session

- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
This commit is contained in:
root
2026-07-12 22:49:39 +00:00
parent 19b6db9891
commit 79d4a73895
14 changed files with 372 additions and 132 deletions
+6 -4
View File
@@ -1,11 +1,13 @@
---
kind: responsibility
name: litellm-self-heal
status: manual-only
status: deployed
note: >
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
This contract is now manual-only — triggers require explicit user request.
Consider reimplementing as a standalone cron job or prose contract.
DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * *
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
now reimplemented as `litellm-health-check.sh` on CT 116.
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
GPU monitoring integrated from gpu-monitor on .24:9100.
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
duplication of architecture diagrams, GPU topology, timeout tables, and container