26 Commits

Include renames

Author	SHA1	Message	Date
root	3b6cf44a30	fix: update all Mumuni IP references from .123 to .24 (inside Abiba CT100) ... Mumuni CT114 destroyed. Mumuni now runs inside Abiba CT100 at 192.168.68.24. Updated all contract files and agent-health-check.py.	2026-07-28 08:56:45 +00:00
root	22b015e182	fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)	2026-07-27 20:25:51 +00:00
root	179529de71	fix: swap gpu-dense to SmartCode-Fable-5 (ThinkingCap + Fable 5 CoT) ... gpu-dense on RTX 3090 swapped from qwen3.6-27B-code to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled (UD-Q4_K_XL, ~17.9GB).\n\nImprovements:\n- ~50% fewer thinking tokens via ThinkingCap finetune\n- Fable 5 CoT distillation for improved coding reasoning\n- Same 27B base, fits RTX 3090 at ~73% VRAM\n- Recommended samplers: temp 0.9, top-p 0.95, top-k 60\n- Context: 128K (fleet standard)	2026-07-27 16:43:29 +00:00
root	2b9b545ca9	no-mistakes(document): Fixed 2 stale model-name references in doc files	2026-07-23 09:10:43 +00:00
root	1c44bf1259	contract updates: ornith decommissioned, 256K→128K context, Mumuni Discord disabled ... Change 1: Strix Halo — ornith decommissioned - gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout - inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4 - gpu-monitor: ornith status → Strix Halo status - infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4 - infrastructure-update: ornith→strix-moe via router - proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe) Change 2: GPU context 256K→128K fleet-wide - hermes-agent-baseline: frontmatter description updated - litellm-health: GPU Fleet Topology table 256K→128K - litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert - inference-optimization: compress threshold 256K→128K compact at 85K - gpu-fleet: instability note updated Change 3: Mumuni Discord platform disabled - gpu-fleet: Mumuni platforms: removed discord	2026-07-23 09:00:42 +00:00
root	aebc98ead6	fix: remove trailing whitespace from litellm-api-keys.prose.md	2026-07-18 17:48:52 +00:00
root	31ec70ae36	gpu-fleet: RTX 3090 swap to ThinkingCap-Qwen3.6-27B Q4_K_M ... - Model: Qwopus Q4_K_M (16GB, 63 tok/s) → ThinkingCap Q4_K_M (15.7GB, 68 tok/s) - RL-finetuned: 50% fewer thinking tokens, 0.85 MMLU-Pro (vs 0.83 base) - Self-spec MTP (n=4) REQUIRED for stability — segfaults without it - Added vision via mmproj (0.9GB) — new capability for this GPU - VRAM: 20.9/24.6GB (85%), tighter but stable - Outputs reasoning_content (hidden from Hermes agent)	2026-07-17 15:50:50 +00:00
root	5eb6d3bfbd	gpu-fleet: RTX 5070 swap to HauhauCS Gemma4-12B QAT Uncensored Balanced ... - Model: IQ4_NL (6.3GB, 191 tok/s) → Q4_K_M QAT (6.9GB, 87 tok/s) - MTP draft: Q8_0 (444MB) → tuned draft (242MB), saves 200MB VRAM - mmproj: F16 → BF16 (same size, matched to new model) - Benefits: 0/465 refusals, agent-optimized tuning, QAT quality - Trade: 54% slower generation (acceptable for gpu-light role) - Config: --parallel 1, --ctx-size 131072, single-slot full 128K - VRAM: 10.0/12.2GB (82%), healthy headroom	2026-07-17 15:24:13 +00:00
root	b9149bce47	feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo ... GPU Changes: - All 3 GPUs reduced from 256K (-c 262144) to 128K (-c 131072) for stability - Observed instability near 100K at 256K — 128K is the stable ceiling - VRAM improved: RTX 3090 ~70% (was 90%), RTX 5070 ~65% (was 88%) - Strix Halo swapped to LuffyTheFox/Genesis Hermes V3 APEX - Hermes agent fine-tune, tensor repair (3 SSM layers, 76% W1 improvement) - Uncensored (0/465 refusals), multimodal (mmproj F16) - Speed: 65 tok/s gen, 140 tok/s prompt - Alias strix-moe maintained Agent Updates: - Mumuni: max_context_window 262144→131072, already aligned on strix-moe/0.65 - Tanko: max_context_window 262144→131072 - Koonimo: max_context_window + context_length 262144→131072 - CT114 SSH access confirmed (was 'Zulip only') LiteLLM (CT116): - Updated backend model references qwen3.6-35B-udq4→strix-moe - Removed stale ornith-1.0-35b from model_cost - Fallback chains updated Contracts Updated: - gpu-fleet.prose.md: topology, VRAM, benchmarks, config lines, model assignments - gpu-self-heal.prose.md: Rule 9/10 context targets - hermes-config-template.prose.md: template values, Rules 7-9, compression thresholds - inference-optimization.prose.md: added to repo, 128K recommendation Compression: 0.65 fires at ~85K (~43K headroom before 128K ceiling) For >128K workloads: route to external providers (deepseek)	2026-07-17 10:59:45 +00:00
root	b7e23e2592	vault: Infisical cleanup + contract sync (WAL #1316) ... - litellm-api-keys: vault audit, tanko/koby/koonimo migration, key rotation log - gpu-fleet: ornith-1.0-35b→strix-moe (remaining refs) - hermes-agent-baseline: 256K all GPUs, CT IPs updated, Shumba retired - delegation-prose-contract: removed (renamed to mumuni-delegation) - contract-registry.yaml + cron-prompts-review.md: from feat/contract-registry	2026-07-16 21:17:09 +00:00
root	0c298eb9d9	feat: add routing configuration with RPM caps (ornith syslog-auto: 40→60) ... - Strix Halo ornith RPM increased from 40 to 60 inside syslog-auto pool - Prevented unbounded ornith traffic when multiple agents use syslog-auto - Direct ornith remains at RPM=40 - All models verified healthy (ornith 80ms, gemma 110ms, qwen 840ms) - Added full routing table to gpu-fleet contract	2026-07-12 23:38:21 +00:00
root	be02b0e843	fix: stale dates, context values, and architecture references ... - gpu-fleet: Updated all context references (128K→256K for RTX 3090). Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram (router labeled deprecated/not in path). Corrected VRAM values. Updated dates: June→July 2026. - hermes-config-template: Fixed description (removed incorrect '128K on NVIDIA' claim). Updated to reflect verified 256K on RTX 3090. - litellm-self-heal: Updated last-verified date. - litellm-api-keys: Updated last-verified date.	2026-07-12 22:54:30 +00:00
root	79d4a73895	docs: lessons learned from 2026-07-12 session ... - gpu-fleet: Corrected architecture (direct GPU, no router in path). Updated context values (RTX 3090=256K, not 128K). Added api-key standardization requirement. - gpu-self-heal: Added Lessons Learned section with 5 critical findings: L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed) L2: Fallback chain cascading failure loop detection L3: Verify running state, not documentation L4: Infisical fallback requirement (.env must have uncommented key) L5: Zulip event queue can silently die after ~40 reconnects - litellm-self-heal: Updated status manual-only→deployed, cron schedule - litellm-api-keys: Added Infisical token expiry warning + .env fallback - hermes-config-template: Rule 3 updated with .env fallback requirement	2026-07-12 22:49:39 +00:00
root	d829f595b4	fix: IP drift and container count from live testing ... Testing the consolidated litellm-self-heal contract against live infra found: - IP drift: pi host is .24, not .65 (confused with ra-h-os bridge) - Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations) - Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables - Container count: 10 running on CT 116, not 8 as contract stated - Added harness-docker-stats + harness-pve-exporter to litellm-self-heal - Updated container health check list Verified: GPU monitor restarted on .24:9100, ornith model inference working, all 10 containers healthy.	2026-07-09 06:23:44 +00:00
root	776df418ca	gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes ... - All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3) - .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache - .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV - .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm - LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s - Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma - Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)	2026-07-08 17:25:42 +00:00
root	5aa93117de	docs: capture all session changes — port conflict, health check, streaming ... Updated contracts to reflect all infrastructure changes made 2026-07-05/06: gpu-fleet.prose.md: - Port conflict detection on all 3 GPU wrappers - Ghost process root cause documented (.8 stale pid 25836) infrastructure-control.prose.md: - Section 7: Agent Health Check (consolidated, non-disruptive) - Disabled scripts documented (zulip-watchdog, zulip-monitor) hermes-agent-baseline.prose.md: - Health verification section + GPU port conflict detection - Change log updated zulip-health.prose.md: - Streaming support section (edit_message + PATCH API)	2026-07-06 02:46:41 +00:00
root	35e864bdb7	audit: all 4 Hermes agents verified — no master key, api_key workaround applied ... Agent audit (2026-07-05): Tanko CT 112: sk-CggiHWlamQyShxWC3Hx6uw — vision+compression patched Mumuni CT 114: sk-VrqCNlwUgzoNGOpikJ7nwQ — vision patched Tdunna CT 111: sk-6sbCNjz2T6lTVDBdlNHXsA — vision+compression patched Baggy CT 113: sk-krnw_zGBwvvL5b7l2t-s-A — vision+compression patched All 4 agents: zero master key references, dedicated LiteLLM keys, api_key workaround applied to bypass auxiliary_client bug.	2026-07-05 23:36:00 +00:00
root	c8dcd8b63d	fix: Tanko key regenerated, Mumuni key verified from /etc/environment ... Tanko: old key sk-620a05e95a... was from docker-compose router config, not LiteLLM key DB. Deleted + regenerated via LiteLLM API: sk-CggiHWlamQyShxWC3Hx6uw (alias: tanko, budget: $100) Mumuni: verified via pct-run 114 — key is sk-VrqCNlwUgzoNGOpikJ7nwQ (was previously listed as sk-XY2aUfvy2... in contracts)	2026-07-05 20:18:05 +00:00
root	5c90eb8f6c	rename: Koonimo→Baggy, Koby→Tdunna — match hostnames ... Standardize agent names across all contracts and pct-run.sh. No functional changes — same CT IDs, same keys, same nodes.	2026-07-05 20:04:22 +00:00
root	0db95a4d1c	fix: correct IPs, CT IDs, and key status across all contracts ... hermes-key-enforcement.prose.md: - Add IP column to verified agents table - Fix Tanko status: was using master key via systemd drop-in (not compliant) - Add detection query step 2: check systemd drop-ins for master key leaks - Add step 3: verify running process env against dedicated key - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only) gpu-fleet.prose.md: - Add CT ID column to agent table for cross-reference - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host) - Update Abiba LiteLLM key to match actual models.json key - Set Koonimo IP to unknown (was .114 — incorrect per user) - Fix pi-specific paths .24 → .65 hermes-config-template.prose.md: - Set Koonimo IP to unknown infrastructure-control.prose.md: - Set Koonimo IP to unknown - Fix Koonimo CT reference	2026-07-05 19:58:14 +00:00
root	263c4080f3	ci: add automated PR validation pipeline + fix regressions ... NEW: Gitea Actions CI pipeline (.gitea/workflows/pr-pipeline.yaml) - Stage 1 (validate): YAML frontmatter validation for all .prose.md files - Stage 2 (lint): Structural checks + regression detection * /grafana/ nginx route (reverted 2026-07-02) * Stale CT IDs (122→112, no 123) * Verified: .19/.122/.123 = correct bridge IPs - Stage 3 (ai-review): LiteLLM-powered diff review against ground truth - Stage 4 (auto-merge): Gate — safe to merge when all green NEW: scripts/prose-lint.sh — contract structure + regression checker NEW: scripts/prose-ai-review.sh — AI review via syslog-auto model FIXES FOUND BY LINT: - gpu-fleet.prose.md: removed /grafana/ from nginx routing diagram - gpu-fleet.prose.md: removed /grafana/ from config file description - infrastructure-control.prose.md: CT 122→112 in topology diagram - scripts/prose-lint.sh: updated to not flag .19 (verified correct IP)	2026-07-04 22:13:51 +00:00
root	b15771bfd9	prose: add Strix Halo thermal safeguard (-n 8192 cap) to gpu-fleet ... Runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (9°C past critical) and throttling 70->29 t/s. Server-side -n 8192 hard cap now bounds generation. Documented connection source (all from .116/LiteLLM) and the thermal constraint (fanless APU).	2026-07-02 03:51:36 +00:00
root	a07ff2d96a	prose: align all contracts with Vulkan rebuild + Koby key regen ... - gpu-fleet: Koby key updated (sk-lDTsWy7H3T...), config table (ornith service + masked services), benchmarks (prompt 209→532 tok/s), ROCm→Vulkan status - gpu-monitor: topology (sidecars :8090 removed, .15 via router), subsystem polling (router /health/unified as source of truth), execution steps, alert threshold (sidecar-unreachable downgraded from critical)	2026-07-02 00:29:10 +00:00
root	49e8d968f5	gpu-fleet: Strix Halo Vulkan rebuild + LiteLLM alignment (2026-07-01) ... - ornith now on Vulkan build 4fc4ec5, 256K ctx, ~70 tok/s - Fixed GPU_MOE_URL bug in docker-compose (.110→.15) - Added router sidecar fallback for health-checks - Updated roster context 4096→262144 + correct model_path - Documented port 8080 firewall (restricted to .116)	2026-07-02 00:01:56 +00:00
root	38d2f7d956	feat: update contracts with GPU optimizations, Strix Halo/ROCm status, alert migration ... - gpu-fleet: documented parallel=2/batch=4096 optimizations for RTX 3090/5070 - gpu-fleet: added Strix Halo ROCm status (installed, blocked by HSA runtime on Debian 13) - gpu-fleet: documented alert migration from DM → #agent-hub stream topics - gpu-monitor: added Alert Delivery section with stream topic conventions - All GPU alerts now go to #agent-hub for cross-agent visibility	2026-07-01 22:25:26 +00:00
root	76fc2595ea	feat(gpu-fleet): GPU fleet management prose contract ... Declarative GPU roster replaces hardcoded router.py model configs. Operations: add-model, remove-model, heal, sync-keys, list. Documents agent key architecture with LITELLM_API_KEY env var. Covers all 3 GPU hosts (RTX 5070, RTX 3090, Strix Halo).	2026-06-28 16:24:40 +00:00