Commit Graph
18 Commits
Author SHA1 Message Date
Abiba cd52deda92 contracts: stable alias chain + no-master-key-for-inference rule
Agent configs aligned to stable aliases (gpu-light/gpu-dense/strix-moe/syslog-auto):
- Tanko fixed: base_url :4000->/v1 (nginx), max_context_window 131072->262144,
  aux.compression gemma->strix-moe, aux gemma->gpu-light
- Mumuni aligned: aux gemma->gpu-light, delegation/x_search qwen->gpu-dense
- Koby/Koonimo already used gpu-light (validated)
- Zero raw model names remain in any agent config

No-master-key rule: litellm_proxy_master_key (sk-litellm-...) is admin-only
(/key/list, /key/generate). All inference uses agent keys. Health-check model
tests use dedicated monitor key (/etc/litellm-monitor.env on CT 116).
Fixed: daily-infra-report.py, gpu-self-heal.py, litellm-health-check.sh.

Fixed wrong key-scope note: qwen3.6-35B-udq4 IS in scope for baggy/koby/mumuni/
abiba-pi keys (not 'NOT in allowlist' as previously documented).
2026-07-16 17:39:05 +00:00
Abiba d0feb7881e Contracts: Rule 13 two key-injection patterns + koby/baggy rotation log (WAL #1300) 2026-07-16 15:10:50 +00:00
Abiba 9fd8c68bd2 Contracts: replace remaining ornith-1.0-35b refs with strix-moe 2026-07-16 13:57:57 +00:00
Abiba 39e2fa0cfa Contracts: Mumuni context-fix (WAL #1300) — strix-moe, 256K all GPUs, Rule 12/13, machine-identity vault procedure 2026-07-16 13:57:31 +00:00
root af41f8f57a fix: remaining template YAML and topology table corrections
- hermes-config-template: Template YAML compression model gemma→ornith.
  Context guidance updated (qwen=256K, only gemma limited to 131K).
- litellm-self-heal: GPU topology table fixed (RTX 3090: 128K→256K, parallel 2→1)
2026-07-12 22:55:24 +00:00
root be02b0e843 fix: stale dates, context values, and architecture references
- gpu-fleet: Updated all context references (128K→256K for RTX 3090).
  Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram
  (router labeled deprecated/not in path). Corrected VRAM values.
  Updated dates: June→July 2026.
- hermes-config-template: Fixed description (removed incorrect '128K on
  NVIDIA' claim). Updated to reflect verified 256K on RTX 3090.
- litellm-self-heal: Updated last-verified date.
- litellm-api-keys: Updated last-verified date.
2026-07-12 22:54:30 +00:00
root 79d4a73895 docs: lessons learned from 2026-07-12 session
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
2026-07-12 22:49:39 +00:00
root 19b6db9891 feat: GPU workload redistribution — compression → Strix Halo
- Move compression model from gemma-4-12b (RTX 5070) to ornith-1.0-35b (Strix Halo)
- Add Rule 8: GPU Workload Distribution — per-GPU role assignment
- Add Rule 9: Compression Threshold for 256K models
- Update Rule 7: Auxiliary Model Consistency with new compression routing
- Add gpu-self-heal.prose.md contract with 10 remediation rules
- Strix Halo (64GB, 256K, 72.4 tok/s) → compression specialist
- RTX 5070 (12GB) → vision/web search specialist
- RTX 3090 (24GB, 256K) → heavy reasoning specialist
- All rules grilled and confirmed with Kwame 2026-07-12
2026-07-12 22:27:15 +00:00
root 776df418ca gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3)
- .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache
- .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV
- .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm
- LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s
- Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma
- Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
2026-07-08 17:25:42 +00:00
root 5c90eb8f6c rename: Koonimo→Baggy, Koby→Tdunna — match hostnames
Standardize agent names across all contracts and pct-run.sh.
No functional changes — same CT IDs, same keys, same nodes.
2026-07-05 20:04:22 +00:00
root 0db95a4d1c fix: correct IPs, CT IDs, and key status across all contracts
hermes-key-enforcement.prose.md:
  - Add IP column to verified agents table
  - Fix Tanko status: was using master key via systemd drop-in (not compliant)
  - Add detection query step 2: check systemd drop-ins for master key leaks
  - Add step 3: verify running process env against dedicated key
  - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)

gpu-fleet.prose.md:
  - Add CT ID column to agent table for cross-reference
  - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
  - Update Abiba LiteLLM key to match actual models.json key
  - Set Koonimo IP to unknown (was .114 — incorrect per user)
  - Fix pi-specific paths .24 → .65

hermes-config-template.prose.md:
  - Set Koonimo IP to unknown

infrastructure-control.prose.md:
  - Set Koonimo IP to unknown
  - Fix Koonimo CT reference
2026-07-05 19:58:14 +00:00
jerome 0a3b62a598 hermes-config-template: update to v2.1.0 — max_tokens, auxiliary consistency, compression threshold
- Add model.max_tokens: 4096 (thermal safety — Rule 6)
- Add context_length: 262144 to template
- Add Rule 7: Auxiliary model consistency (gemma-4-12b + base_url + api_key_env)
- Add Rule 8: Compression threshold for 256K models (0.65, not 0.25)
- Fix compression section: threshold 0.65, max_context_window 262144, protect_last_n 40
- Remove stray api_key_env from compression block (not propagated)
- Update agent Keys table to show key aliases instead of redacted values
- Add base_url and timeout to all auxiliary service configs
- Add auxiliary: compression: section matching main compression config
- Update version 2.0.0 → 2.1.0
2026-07-04 17:54:43 -04:00
root 253e19f680 prose: review, fix, and consolidate infrastructure contracts
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00
Abiba c1bdbbc491 2026-07-03: Hermes key enforcement contract + Stirling-PDF docs + config standardization
- hermes-key-enforcement.prose.md: NEW contract enforcing api_key_env for all harness/LiteLLM providers, exempting external providers
- hermes-config-template.prose.md: Updated with full fix log (15 total fixes across Koby/Koonimo/Mumuni/Tanko), model:auto detection rule, key-to-agent mapping
- infrastructure-control.prose.md: Added Stirling-PDF service details (port 8989, credentials, API key, Swagger URL) replacing Bento-PDF
2026-07-03 18:19:41 +00:00
jerome a985748994 Update hermes-config-template.prose.md
update wrong searxng address
2026-07-03 17:59:39 +00:00
Abiba e991fc7ea0 fix(hermes-config-template): remove stale key table, add Key Management section
- REMOVED hardcoded Agent Keys table (keys became stale after LiteLLM downgrade).
- ADDED Key Management section: verify loop, rotate + deploy procedure, key
  deployment rules (plaintext in /etc/environment only, api_key_env pattern),
  current aliases (2026-07-02 rotation).
- LiteLLM DB is the single source of truth for keys; contracts never store plaintext.
- Mumuni sub-agent profiles documented (6, all inherit auth from main config).
2026-07-02 20:49:14 +00:00
jerome 94b3bf84b4 Add README + Quickstart sections for agent execution 2026-06-28 13:49:06 -04:00
jerome 6a8a2c61da Add hermes-config-template prose contract with shared infra + flexible models 2026-06-28 13:33:06 -04:00