153 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
root	23f3f378c5	zulip-resilience: add fleet-wide audit findings and fixes from 2026-07-18 ... - Added Incident Log section documenting fleet-wide Zulip audit - Abiba: poll timeout AbortError fix (returns [] instead of error) - Abiba: credential fallback .env file for Infisical outages - Tanko: full gateway restart to recover Zulip connection - Fleet health metrics summary table - Hermes agent improvement recommendations (circuit breaker, credential fallback, queue re-registration, watchdog) - Abiba pi extension v2.1 resilience feature matrix	2026-07-18 16:33:12 +00:00
root	bddbb22f03	gpu-self-heal: refresh to current fleet baseline and topology ... - Synced model assignments to 2026-07-17 swaps (ThinkingCap, HauhauCS QAT, Genesis Hermes V3) - Added stable role-based aliases from gpu-fleet (gpu-dense, gpu-light, strix-moe) - Updated benchmark baselines to live values (74.9/169.6/62.9 tok/s) - Replaced router (port 9000) references with LiteLLM + direct routing - Replaced Prometheus exporter rule with sidecar health probe - Updated VRAM thresholds to match operational data (300/300/200 MB/h) - Added response size limit (1MB) to prevent OOM crashes - Added Lessons L5 (response size crash) and L6 (stable aliases) - Removed deprecated Rules 11-12 (router-specific distribution balance)	2026-07-18 08:07:03 +00:00
root	31ec70ae36	gpu-fleet: RTX 3090 swap to ThinkingCap-Qwen3.6-27B Q4_K_M ... - Model: Qwopus Q4_K_M (16GB, 63 tok/s) → ThinkingCap Q4_K_M (15.7GB, 68 tok/s) - RL-finetuned: 50% fewer thinking tokens, 0.85 MMLU-Pro (vs 0.83 base) - Self-spec MTP (n=4) REQUIRED for stability — segfaults without it - Added vision via mmproj (0.9GB) — new capability for this GPU - VRAM: 20.9/24.6GB (85%), tighter but stable - Outputs reasoning_content (hidden from Hermes agent)	2026-07-17 15:50:50 +00:00
root	5eb6d3bfbd	gpu-fleet: RTX 5070 swap to HauhauCS Gemma4-12B QAT Uncensored Balanced ... - Model: IQ4_NL (6.3GB, 191 tok/s) → Q4_K_M QAT (6.9GB, 87 tok/s) - MTP draft: Q8_0 (444MB) → tuned draft (242MB), saves 200MB VRAM - mmproj: F16 → BF16 (same size, matched to new model) - Benefits: 0/465 refusals, agent-optimized tuning, QAT quality - Trade: 54% slower generation (acceptable for gpu-light role) - Config: --parallel 1, --ctx-size 131072, single-slot full 128K - VRAM: 10.0/12.2GB (82%), healthy headroom	2026-07-17 15:24:13 +00:00
jerome	33cb88d571	Merge pull request 'feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo' (#20) from feat/gpu-128k-genesis-hermes-v3-20260717 into master ... Reviewed-on: #20	2026-07-17 11:21:32 +00:00
root	4a1f476623	fix: add missing description to inference-optimization frontmatter	2026-07-17 11:11:27 +00:00
root	ba2c55c7e6	ci: re-trigger pipeline for PR #20	2026-07-17 11:10:29 +00:00
root	b9149bce47	feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo ... GPU Changes: - All 3 GPUs reduced from 256K (-c 262144) to 128K (-c 131072) for stability - Observed instability near 100K at 256K — 128K is the stable ceiling - VRAM improved: RTX 3090 ~70% (was 90%), RTX 5070 ~65% (was 88%) - Strix Halo swapped to LuffyTheFox/Genesis Hermes V3 APEX - Hermes agent fine-tune, tensor repair (3 SSM layers, 76% W1 improvement) - Uncensored (0/465 refusals), multimodal (mmproj F16) - Speed: 65 tok/s gen, 140 tok/s prompt - Alias strix-moe maintained Agent Updates: - Mumuni: max_context_window 262144→131072, already aligned on strix-moe/0.65 - Tanko: max_context_window 262144→131072 - Koonimo: max_context_window + context_length 262144→131072 - CT114 SSH access confirmed (was 'Zulip only') LiteLLM (CT116): - Updated backend model references qwen3.6-35B-udq4→strix-moe - Removed stale ornith-1.0-35b from model_cost - Fallback chains updated Contracts Updated: - gpu-fleet.prose.md: topology, VRAM, benchmarks, config lines, model assignments - gpu-self-heal.prose.md: Rule 9/10 context targets - hermes-config-template.prose.md: template values, Rules 7-9, compression thresholds - inference-optimization.prose.md: added to repo, 128K recommendation Compression: 0.65 fires at ~85K (~43K headroom before 128K ceiling) For >128K workloads: route to external providers (deepseek)	2026-07-17 10:59:45 +00:00
root	65c99dab50	contract: update litellm-api-keys to v2026-07-17 — fleet standardization ... - Document canonical systemd drop-in pattern (ExecStart= reset + wrapper) - Hardcode venv paths — never use variables in single-quoted bash -c - Update migration status: all 4 agents on while-true wrapper + st.8e848433 - Infisical CLI update procedure (0.38.0 → 0.43.109 via artifacts-cli) - Service token inventory + .env fallback inventory - Key rotation log: fleet standardize + tanko fix-zulip entries - Remove git merge conflict artifacts - Add fleet-wide standardization lessons section	2026-07-17 10:17:23 +00:00
jerome	20cbb96e2d	Merge pull request 'Vault cleanup + contract sync (WAL #1316)' (#19) from fix/vault-cleanup-contract-sync into master ... Reviewed-on: #19	2026-07-16 21:24:59 +00:00
root	8215b84f88	merge: resolve conflicts with master (delegation delete + gpu-fleet)	2026-07-16 21:22:34 +00:00
jerome	24cd7f72ac	Merge branch 'master' into feat/zulip-v3-resilience	2026-07-16 21:20:24 +00:00
jerome	d6e85e68e9	Merge pull request 'UPDATE 2026-07-15: Full GPU fleet rebuild + stable aliases' (#18) from feat/gpu-fleet-rebuild-20260715 into master ... Reviewed-on: #18	2026-07-16 21:17:12 +00:00
root	b7e23e2592	vault: Infisical cleanup + contract sync (WAL #1316) ... - litellm-api-keys: vault audit, tanko/koby/koonimo migration, key rotation log - gpu-fleet: ornith-1.0-35b→strix-moe (remaining refs) - hermes-agent-baseline: 256K all GPUs, CT IPs updated, Shumba retired - delegation-prose-contract: removed (renamed to mumuni-delegation) - contract-registry.yaml + cron-prompts-review.md: from feat/contract-registry	2026-07-16 21:17:09 +00:00
root	369c312ccb	vault: BAGGY_LITELLM_API_KEY deleted — one secret per agent	2026-07-16 19:54:51 +00:00
root	7f39a444d6	vault: BAGGY+LITELLM synced, deletion needs Infisical UI	2026-07-16 19:28:28 +00:00
Abiba	17f4c79fe4	vault: sync BAGGY_LITELLM_API_KEY = KOONIMO_LITELLM_API_KEY (same agent CT113) ... Baggy = Koonimo (CT 113). The vault had two divergent keys: - BAGGY_LITELLM_API_KEY = sk-QT-kDt8Szo... (stale, 401) - KOONIMO_LITELLM_API_KEY = sk-OEK7z26n6... (valid, 200, session-13 rotation) Synced BAGGY to match KOONIMO (the valid key). Added contract note: both secrets MUST mirror each other. This was the root cause of Koonimo's 401 after migration — the ghost process had BAGGY in its env instead of KOONIMO.	2026-07-16 19:22:48 +00:00
Abiba	a9caf5b216	vault: fix koby platform connectivity + document migration lessons ... Koby was broken after migration — Zulip credentials lost (only in process memory), Telegram token overwritten (only existed in old .env.bak-20260603). Fix: Koby shares Tanko's Zulip bot (tanko-bot@, TANKO_ZULIP_API_KEY in vault). Telegram token recovered from .env.bak-20260603. Both platforms now connected. Added Koby migration lessons to contract (back up .env before migration, inject ALL platform env vars, cat /proc/pid/environ before killing old gateway). Added KOBY_ZULIP_API_KEY to vault (shares Tanko's key).	2026-07-16 18:19:27 +00:00
Abiba	15a826457e	vault: canonical Production Vault Access Process + sync vault + migrate koby/koonimo ... - Vault SYNCED: mumuni/koby/koonimo rotated keys written to Infisical (all validate 200). Vault is source of truth again (was stale since session-13 rotations). - Koby (.129) + Koonimo (.114) migrated from hardcoded systemd drop-ins to infisical-gateway.sh wrapper (live vault injection, .env fallback safety net). 4/5 agents now vault-backed (abiba/mumuni/koby/koonimo). - New § Production Vault Access Process: canonical 7-step pattern, migration status table, key rotation procedure, why-it's-non-fail rationale. - Fixed stale notes: 'vault sync PENDING' (now synced), 'Abiba key IS master key' (now a proper agent key sk-sxbphLvk1OU). - Tanko (user jerome, not systemd) documented as pending migration. - hermes-key-enforcement: cross-ref to canonical process, drop-ins deprecated.	2026-07-16 17:51:53 +00:00
Abiba	cd52deda92	contracts: stable alias chain + no-master-key-for-inference rule ... Agent configs aligned to stable aliases (gpu-light/gpu-dense/strix-moe/syslog-auto): - Tanko fixed: base_url :4000->/v1 (nginx), max_context_window 131072->262144, aux.compression gemma->strix-moe, aux gemma->gpu-light - Mumuni aligned: aux gemma->gpu-light, delegation/x_search qwen->gpu-dense - Koby/Koonimo already used gpu-light (validated) - Zero raw model names remain in any agent config No-master-key rule: litellm_proxy_master_key (sk-litellm-...) is admin-only (/key/list, /key/generate). All inference uses agent keys. Health-check model tests use dedicated monitor key (/etc/litellm-monitor.env on CT 116). Fixed: daily-infra-report.py, gpu-self-heal.py, litellm-health-check.sh. Fixed wrong key-scope note: qwen3.6-35B-udq4 IS in scope for baggy/koby/mumuni/ abiba-pi keys (not 'NOT in allowlist' as previously documented).	2026-07-16 17:39:05 +00:00
Abiba	97f1cd77e4	litellm-self-heal: document 2026-07-16 ops sync (script fixes, systemd gpu-monitor, live-key agent check) ... Health-check script: critical-only gpu-fleet alerting, strix-moe model test. gpu-monitor: systemd unit (was bare &), Strix counted in gpu_count, VRAM thresholds raised (93/97 — 256K ctx steady-state ~96% on RTX 3090). agent-health-check.py: reads live LITELLM_API_KEY from gateway env (no hardcoded keys). daily-infra-report.py: stale SYNTHETIC key → env-based master key.	2026-07-16 17:23:37 +00:00
Abiba	dc572889f8	contracts: sync to ground truth — ornith-1.0-35b→strix-moe, 256K all GPUs, real LiteLLM timeouts ... Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts: - AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist - All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1) - LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s) - Added LiteLLM model surface + key scoping to litellm-self-heal - Patched health-check script path ref Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal, zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline, delegation-prose-contract, mumuni-delegation-prose-contract	2026-07-16 17:02:44 +00:00
Abiba	d0feb7881e	Contracts: Rule 13 two key-injection patterns + koby/baggy rotation log (WAL #1300)	2026-07-16 15:10:50 +00:00
Abiba	9fd8c68bd2	Contracts: replace remaining ornith-1.0-35b refs with strix-moe	2026-07-16 13:57:57 +00:00
Abiba	39e2fa0cfa	Contracts: Mumuni context-fix (WAL #1300) — strix-moe, 256K all GPUs, Rule 12/13, machine-identity vault procedure	2026-07-16 13:57:31 +00:00
root	622cf7b176	UPDATE 2026-07-15: Full fleet rebuild ... - Stable role-based aliases: strix-moe, gpu-dense, gpu-light - Strix Halo: ornith-1.0-35b -> unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 256K) - RTX 5070: Q4_K_M -> IQ4_NL+MTP (256K, 125-213 tok/s, 2x faster) - RTX 3090: bumped to 256K context - Provider rename: harness -> litellm (all agents) - syslog-auto: weighted pool restored with direct GPU routing - Mumuni agent profile documented - Self-heal thresholds updated for 256K context - All model contexts corrected (RTX 3090 was 131K, not 256K)	2026-07-15 16:04:21 +00:00
root	30bf42b841	feat: Zulip v3 resilience contract + restore playbook ... - zulip-resilience-v3.prose.md: Production resilience rewrite contract covering circuit breaker, retry with jitter, queue lifecycle management, supervisor watchdog, and PM2 hardening. Research-backed from Zulip event system docs. - abiba-zulip-restore.prose.md: Quick-restore playbook for recovery scenarios.	2026-07-13 21:29:35 +00:00
abiba-bot	190ceb9be4	Merge pull request 'GPU workload redistribution + lessons learned July 2026' (#15) from feat/gpu-workload-compression-jul2026 into master	2026-07-12 23:40:12 +00:00
root	0c298eb9d9	feat: add routing configuration with RPM caps (ornith syslog-auto: 40→60) ... - Strix Halo ornith RPM increased from 40 to 60 inside syslog-auto pool - Prevented unbounded ornith traffic when multiple agents use syslog-auto - Direct ornith remains at RPM=40 - All models verified healthy (ornith 80ms, gemma 110ms, qwen 840ms) - Added full routing table to gpu-fleet contract	2026-07-12 23:38:21 +00:00
root	af41f8f57a	fix: remaining template YAML and topology table corrections ... - hermes-config-template: Template YAML compression model gemma→ornith. Context guidance updated (qwen=256K, only gemma limited to 131K). - litellm-self-heal: GPU topology table fixed (RTX 3090: 128K→256K, parallel 2→1)	2026-07-12 22:55:24 +00:00
root	be02b0e843	fix: stale dates, context values, and architecture references ... - gpu-fleet: Updated all context references (128K→256K for RTX 3090). Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram (router labeled deprecated/not in path). Corrected VRAM values. Updated dates: June→July 2026. - hermes-config-template: Fixed description (removed incorrect '128K on NVIDIA' claim). Updated to reflect verified 256K on RTX 3090. - litellm-self-heal: Updated last-verified date. - litellm-api-keys: Updated last-verified date.	2026-07-12 22:54:30 +00:00
root	79d4a73895	docs: lessons learned from 2026-07-12 session ... - gpu-fleet: Corrected architecture (direct GPU, no router in path). Updated context values (RTX 3090=256K, not 128K). Added api-key standardization requirement. - gpu-self-heal: Added Lessons Learned section with 5 critical findings: L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed) L2: Fallback chain cascading failure loop detection L3: Verify running state, not documentation L4: Infisical fallback requirement (.env must have uncommented key) L5: Zulip event queue can silently die after ~40 reconnects - litellm-self-heal: Updated status manual-only→deployed, cron schedule - litellm-api-keys: Added Infisical token expiry warning + .env fallback - hermes-config-template: Rule 3 updated with .env fallback requirement	2026-07-12 22:49:39 +00:00
root	19b6db9891	feat: GPU workload redistribution — compression → Strix Halo ... - Move compression model from gemma-4-12b (RTX 5070) to ornith-1.0-35b (Strix Halo) - Add Rule 8: GPU Workload Distribution — per-GPU role assignment - Add Rule 9: Compression Threshold for 256K models - Update Rule 7: Auxiliary Model Consistency with new compression routing - Add gpu-self-heal.prose.md contract with 10 remediation rules - Strix Halo (64GB, 256K, 72.4 tok/s) → compression specialist - RTX 5070 (12GB) → vision/web search specialist - RTX 3090 (24GB, 256K) → heavy reasoning specialist - All rules grilled and confirmed with Kwame 2026-07-12	2026-07-12 22:27:15 +00:00
jerome	22eaaf4254	Merge pull request 'fix: add data source integrity rule to delegation contract' (#14) from feat/data-source-integrity-rule into master ... Reviewed-on: #14	2026-07-12 18:09:10 +00:00
jerome	fdb22948d9	fix: add data source integrity rule to delegation contract ... Workers MUST use provided input data, not fetch external sources. Adds: Data Source Integrity section, dispatch guidance example, anti-pattern entry. Root cause of e2e test discrepancies where writer worker queried Proxmox API instead of using raw data.	2026-07-11 18:51:30 -04:00
jerome	e9968e165d	Merge pull request 'feat: add RA-H OS Custodianship Contract for shared memory protocol' (#13) from feat/ra-h-os-custodianship-contract into master ... Reviewed-on: #13	2026-07-10 23:39:30 +00:00
mumuni-bot	44ba53cf71	fix: add YAML frontmatter to RA-H OS Custodianship Contract ... - Added required frontmatter block with kind=pattern, name, and description - Fixes CI validate step that was blocking the PR pipeline - All other CI checks (lint, ai-review) were skipped due to validate failure	2026-07-10 19:16:30 -04:00
jerome	b96b334283	feat: add RA-H OS Custodianship Contract for shared memory protocol	2026-07-10 09:43:49 -04:00
jerome	dc829bb09d	Merge pull request 'feat: mumuni-delegation-prose-contract — manager operating doctrine for worker delegation' (#12) from feat/delegation-prose-contract into master ... Reviewed-on: #12	2026-07-09 07:43:26 +00:00
jerome	a8354e88bc	rename: delegation-prose-contract → mumuni-delegation-prose-contract ... Contract is Mumuni-specific operating doctrine — different agents have different architectures and don't need this. Renamed file and frontmatter to reflect that.	2026-07-09 03:18:29 -04:00
jerome	98d8ce6772	fix: add topology section to satisfy lint checks ... Added cluster/node topology section referencing all 5 Proxmox nodes and clarifying that the contract is infrastructure-agnostic — defines the who and when, not the where.	2026-07-09 02:53:44 -04:00
jerome	548e421fa1	feat: add delegation-prose-contract pattern — manager operating doctrine for worker delegation ... Defines when to delegate, worker selection matrix, verification protocol, kanban board, and failure handling. Addresses the context overflow and iteration exhaustion issues where the manager was doing all work itself instead of delegating to the 6 worker profiles.	2026-07-09 02:29:42 -04:00
abiba-bot	0134ad8be6	Merge pull request 'fix: cron alignment + VM→bare metal drift' (#11) from fix/cron-alignment-jul2026 into master	2026-07-09 06:28:04 +00:00
root	546ca79849	fix: cron alignment + VM→bare metal drift ... - pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract) - pm2-self-heal: added gpu-monitor to tracked PM2 processes - infra-monitor+control: removed daytime-only restriction (now 24×7) - infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110 - Added @reboot pm2 resurrect for persistence across reboots - README: zulip-monitor.sh → agent-health-check.py - Cleaned up commented-out zulip-monitor cron line Live actions: gpu-monitor-server added to PM2 (persistent across reboots)	2026-07-09 06:27:48 +00:00
abiba-bot	c31dbad95f	Merge pull request 'fix: IP drift and container count from live testing' (#10) from fix/testing-drift-jul2026 into master	2026-07-09 06:24:07 +00:00
root	d829f595b4	fix: IP drift and container count from live testing ... Testing the consolidated litellm-self-heal contract against live infra found: - IP drift: pi host is .24, not .65 (confused with ra-h-os bridge) - Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations) - Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables - Container count: 10 running on CT 116, not 8 as contract stated - Added harness-docker-stats + harness-pve-exporter to litellm-self-heal - Updated container health check list Verified: GPU monitor restarted on .24:9100, ornith model inference working, all 10 containers healthy.	2026-07-09 06:23:44 +00:00
abiba-bot	4164ee5ed0	Merge pull request 'feat: consolidate litellm-health into litellm-self-heal' (#9) from feat/consolidate-litellm-contracts into master	2026-07-09 06:02:47 +00:00
root	053d30ce7c	feat: consolidate litellm-health into litellm-self-heal ... - litellm-self-heal: merged health check as §Health Check section, includes full architecture diagram (v4.0.0), GPU topology, timeout tables, container list, and execution flow. Eliminates duplication between the two contracts. - litellm-health: marked DEPRECATED, retained for reference - README.md: updated with consolidated contract, deprecation notice, disk-gc-threat-response, and infrastructure-monitoring status	2026-07-09 06:01:48 +00:00
abiba-bot	e4ab43f1c6	Merge pull request 'fix: LiteLLM contracts — align with 2026-07-08 architecture changes' (#8) from fix/litellm-contracts-jul2026 into master	2026-07-09 05:37:48 +00:00
root	a2c5a14c74	fix: LiteLLM contracts — align with 2026-07-08 architecture changes ... - litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly), GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s), Strix Halo: CPU→Vulkan, context sizes + parallel slots added - litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED (router not in path), GPU topology synced with gpu-fleet - litellm-api-keys: reference gpu-fleet as source of truth for keys, removed qwen3.6-35B-A3B from model list (never deployed)	2026-07-09 05:37:16 +00:00