18 Commits

Include renames

Author	SHA1	Message	Date
Abiba	39e2fa0cfa	Contracts: Mumuni context-fix (WAL #1300) — strix-moe, 256K all GPUs, Rule 12/13, machine-identity vault procedure	2026-07-16 13:57:31 +00:00
root	622cf7b176	UPDATE 2026-07-15: Full fleet rebuild ... - Stable role-based aliases: strix-moe, gpu-dense, gpu-light - Strix Halo: ornith-1.0-35b -> unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 256K) - RTX 5070: Q4_K_M -> IQ4_NL+MTP (256K, 125-213 tok/s, 2x faster) - RTX 3090: bumped to 256K context - Provider rename: harness -> litellm (all agents) - syslog-auto: weighted pool restored with direct GPU routing - Mumuni agent profile documented - Self-heal thresholds updated for 256K context - All model contexts corrected (RTX 3090 was 131K, not 256K)	2026-07-15 16:04:21 +00:00
root	0c298eb9d9	feat: add routing configuration with RPM caps (ornith syslog-auto: 40→60) ... - Strix Halo ornith RPM increased from 40 to 60 inside syslog-auto pool - Prevented unbounded ornith traffic when multiple agents use syslog-auto - Direct ornith remains at RPM=40 - All models verified healthy (ornith 80ms, gemma 110ms, qwen 840ms) - Added full routing table to gpu-fleet contract	2026-07-12 23:38:21 +00:00
root	be02b0e843	fix: stale dates, context values, and architecture references ... - gpu-fleet: Updated all context references (128K→256K for RTX 3090). Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram (router labeled deprecated/not in path). Corrected VRAM values. Updated dates: June→July 2026. - hermes-config-template: Fixed description (removed incorrect '128K on NVIDIA' claim). Updated to reflect verified 256K on RTX 3090. - litellm-self-heal: Updated last-verified date. - litellm-api-keys: Updated last-verified date.	2026-07-12 22:54:30 +00:00
root	79d4a73895	docs: lessons learned from 2026-07-12 session ... - gpu-fleet: Corrected architecture (direct GPU, no router in path). Updated context values (RTX 3090=256K, not 128K). Added api-key standardization requirement. - gpu-self-heal: Added Lessons Learned section with 5 critical findings: L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed) L2: Fallback chain cascading failure loop detection L3: Verify running state, not documentation L4: Infisical fallback requirement (.env must have uncommented key) L5: Zulip event queue can silently die after ~40 reconnects - litellm-self-heal: Updated status manual-only→deployed, cron schedule - litellm-api-keys: Added Infisical token expiry warning + .env fallback - hermes-config-template: Rule 3 updated with .env fallback requirement	2026-07-12 22:49:39 +00:00
root	d829f595b4	fix: IP drift and container count from live testing ... Testing the consolidated litellm-self-heal contract against live infra found: - IP drift: pi host is .24, not .65 (confused with ra-h-os bridge) - Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations) - Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables - Container count: 10 running on CT 116, not 8 as contract stated - Added harness-docker-stats + harness-pve-exporter to litellm-self-heal - Updated container health check list Verified: GPU monitor restarted on .24:9100, ornith model inference working, all 10 containers healthy.	2026-07-09 06:23:44 +00:00
root	776df418ca	gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes ... - All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3) - .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache - .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV - .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm - LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s - Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma - Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)	2026-07-08 17:25:42 +00:00
root	5aa93117de	docs: capture all session changes — port conflict, health check, streaming ... Updated contracts to reflect all infrastructure changes made 2026-07-05/06: gpu-fleet.prose.md: - Port conflict detection on all 3 GPU wrappers - Ghost process root cause documented (.8 stale pid 25836) infrastructure-control.prose.md: - Section 7: Agent Health Check (consolidated, non-disruptive) - Disabled scripts documented (zulip-watchdog, zulip-monitor) hermes-agent-baseline.prose.md: - Health verification section + GPU port conflict detection - Change log updated zulip-health.prose.md: - Streaming support section (edit_message + PATCH API)	2026-07-06 02:46:41 +00:00
root	35e864bdb7	audit: all 4 Hermes agents verified — no master key, api_key workaround applied ... Agent audit (2026-07-05): Tanko CT 112: sk-CggiHWlamQyShxWC3Hx6uw — vision+compression patched Mumuni CT 114: sk-VrqCNlwUgzoNGOpikJ7nwQ — vision patched Tdunna CT 111: sk-6sbCNjz2T6lTVDBdlNHXsA — vision+compression patched Baggy CT 113: sk-krnw_zGBwvvL5b7l2t-s-A — vision+compression patched All 4 agents: zero master key references, dedicated LiteLLM keys, api_key workaround applied to bypass auxiliary_client bug.	2026-07-05 23:36:00 +00:00
root	c8dcd8b63d	fix: Tanko key regenerated, Mumuni key verified from /etc/environment ... Tanko: old key sk-620a05e95a... was from docker-compose router config, not LiteLLM key DB. Deleted + regenerated via LiteLLM API: sk-CggiHWlamQyShxWC3Hx6uw (alias: tanko, budget: $100) Mumuni: verified via pct-run 114 — key is sk-VrqCNlwUgzoNGOpikJ7nwQ (was previously listed as sk-XY2aUfvy2... in contracts)	2026-07-05 20:18:05 +00:00
root	5c90eb8f6c	rename: Koonimo→Baggy, Koby→Tdunna — match hostnames ... Standardize agent names across all contracts and pct-run.sh. No functional changes — same CT IDs, same keys, same nodes.	2026-07-05 20:04:22 +00:00
root	0db95a4d1c	fix: correct IPs, CT IDs, and key status across all contracts ... hermes-key-enforcement.prose.md: - Add IP column to verified agents table - Fix Tanko status: was using master key via systemd drop-in (not compliant) - Add detection query step 2: check systemd drop-ins for master key leaks - Add step 3: verify running process env against dedicated key - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only) gpu-fleet.prose.md: - Add CT ID column to agent table for cross-reference - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host) - Update Abiba LiteLLM key to match actual models.json key - Set Koonimo IP to unknown (was .114 — incorrect per user) - Fix pi-specific paths .24 → .65 hermes-config-template.prose.md: - Set Koonimo IP to unknown infrastructure-control.prose.md: - Set Koonimo IP to unknown - Fix Koonimo CT reference	2026-07-05 19:58:14 +00:00
root	263c4080f3	ci: add automated PR validation pipeline + fix regressions ... NEW: Gitea Actions CI pipeline (.gitea/workflows/pr-pipeline.yaml) - Stage 1 (validate): YAML frontmatter validation for all .prose.md files - Stage 2 (lint): Structural checks + regression detection * /grafana/ nginx route (reverted 2026-07-02) * Stale CT IDs (122→112, no 123) * Verified: .19/.122/.123 = correct bridge IPs - Stage 3 (ai-review): LiteLLM-powered diff review against ground truth - Stage 4 (auto-merge): Gate — safe to merge when all green NEW: scripts/prose-lint.sh — contract structure + regression checker NEW: scripts/prose-ai-review.sh — AI review via syslog-auto model FIXES FOUND BY LINT: - gpu-fleet.prose.md: removed /grafana/ from nginx routing diagram - gpu-fleet.prose.md: removed /grafana/ from config file description - infrastructure-control.prose.md: CT 122→112 in topology diagram - scripts/prose-lint.sh: updated to not flag .19 (verified correct IP)	2026-07-04 22:13:51 +00:00
root	b15771bfd9	prose: add Strix Halo thermal safeguard (-n 8192 cap) to gpu-fleet ... Runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (9°C past critical) and throttling 70->29 t/s. Server-side -n 8192 hard cap now bounds generation. Documented connection source (all from .116/LiteLLM) and the thermal constraint (fanless APU).	2026-07-02 03:51:36 +00:00
root	a07ff2d96a	prose: align all contracts with Vulkan rebuild + Koby key regen ... - gpu-fleet: Koby key updated (sk-lDTsWy7H3T...), config table (ornith service + masked services), benchmarks (prompt 209→532 tok/s), ROCm→Vulkan status - gpu-monitor: topology (sidecars :8090 removed, .15 via router), subsystem polling (router /health/unified as source of truth), execution steps, alert threshold (sidecar-unreachable downgraded from critical)	2026-07-02 00:29:10 +00:00
root	49e8d968f5	gpu-fleet: Strix Halo Vulkan rebuild + LiteLLM alignment (2026-07-01) ... - ornith now on Vulkan build 4fc4ec5, 256K ctx, ~70 tok/s - Fixed GPU_MOE_URL bug in docker-compose (.110→.15) - Added router sidecar fallback for health-checks - Updated roster context 4096→262144 + correct model_path - Documented port 8080 firewall (restricted to .116)	2026-07-02 00:01:56 +00:00
root	38d2f7d956	feat: update contracts with GPU optimizations, Strix Halo/ROCm status, alert migration ... - gpu-fleet: documented parallel=2/batch=4096 optimizations for RTX 3090/5070 - gpu-fleet: added Strix Halo ROCm status (installed, blocked by HSA runtime on Debian 13) - gpu-fleet: documented alert migration from DM → #agent-hub stream topics - gpu-monitor: added Alert Delivery section with stream topic conventions - All GPU alerts now go to #agent-hub for cross-agent visibility	2026-07-01 22:25:26 +00:00
root	76fc2595ea	feat(gpu-fleet): GPU fleet management prose contract ... Declarative GPU roster replaces hardcoded router.py model configs. Operations: add-model, remove-model, heal, sync-keys, list. Documents agent key architecture with LITELLM_API_KEY env var. Covers all 3 GPU hosts (RTX 5070, RTX 3090, Strix Halo).	2026-06-28 16:24:40 +00:00