167 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
root	1c44bf1259	contract updates: ornith decommissioned, 256K→128K context, Mumuni Discord disabled ... Change 1: Strix Halo — ornith decommissioned - gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout - inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4 - gpu-monitor: ornith status → Strix Halo status - infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4 - infrastructure-update: ornith→strix-moe via router - proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe) Change 2: GPU context 256K→128K fleet-wide - hermes-agent-baseline: frontmatter description updated - litellm-health: GPU Fleet Topology table 256K→128K - litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert - inference-optimization: compress threshold 256K→128K compact at 85K - gpu-fleet: instability note updated Change 3: Mumuni Discord platform disabled - gpu-fleet: Mumuni platforms: removed discord	2026-07-23 09:00:42 +00:00
jerome	fc88265e76	Merge pull request 'tune: switch compression model from strix-moe to syslog-auto' (#25) from tune/compression-syslog-auto into master ... Reviewed-on: #25	2026-07-19 01:32:44 +00:00
jerome	51da19d92d	Merge pull request 'feat(contracts): add infrastructure-maintenance contract' (#24) from fm/infra-maint-contract into master ... Reviewed-on: #24	2026-07-19 01:32:29 +00:00
root	6570fd60e7	tune: switch compression model from strix-moe to syslog-auto ... Relieve Strix Halo pressure by distributing compression across the syslog-auto weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070). Updates: - compression.model: strix-moe -> syslog-auto - auxiliary.compression.model: strix-moe -> syslog-auto - Rule 7: Updated for syslog-auto compression, removed aux prohibition - Rule 8: Updated GPU workload distribution - All docs/comments updated to reflect the change	2026-07-18 23:28:07 +00:00
jerome	5d2ecbace6	Merge branch 'master' into fm/infra-maint-contract	2026-07-18 23:12:20 +00:00
root	99789a00a1	no-mistakes(document): Reframed infra-maintenance contract as deliberate partition	2026-07-18 22:16:09 +00:00
root	a68879e904	no-mistakes(review): fix docker rollback command to pin digest then compose up	2026-07-18 22:07:15 +00:00
jerome	06d2bcbc9e	Merge pull request 'fix: fleet config issues from 2026-07-18 relay review' (#23) from fix/fleet-config-issues-20260718 into master ... Reviewed-on: #23	2026-07-18 22:06:48 +00:00
jerome	c4a8c45835	Merge pull request 'zulip-resilience: fleet-wide audit findings and fixes 2026-07-18' (#22) from feat/zulip-resilience-audit-20260718 into master ... Reviewed-on: #22	2026-07-18 22:06:34 +00:00
root	1d027f71f6	feat(contracts): add infrastructure-maintenance contract ... New responsibility contract consolidating host-level system maintenance and Docker image lifecycle management, filling the gap left by infrastructure-update (which owns cluster-wide apt waves). Scope: - OS package updates on primary host with pre-update snapshot/backup check - Docker image pulls for LiteLLM, SearXNG, and other running containers - Container restarts with per-stack health verification - Post-update verification: LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways - Rollback on failure (image/apt/config restore) with circuit breaker Owner: ops (firstmate secondmate). Trigger: weekly Sunday 2am ET. Escalation: warning/critical->abiba+mumuni, fatal->abiba+mumuni+kwame. circuit_breaker: max_retries 2, window 7200, trip_action escalate_to_fatal. depends_on: infrastructure-monitoring (pre-update health baseline). Registry: - Add infrastructure-maintenance to by_category.maintenance, by_domain.infrastructure, by_owner.ops (new), by_trigger.scheduled, by_sensitivity.high - Add 'ops' to owners list - Move infrastructure-update owner abiba -> ops (in contracts entry + by_owner index) Also adds '## Maintaining this file' section to AGENTS.md per fm-ensure-agents-md.	2026-07-18 22:02:45 +00:00
jerome	a74229ee74	Merge branch 'master' into fix/fleet-config-issues-20260718	2026-07-18 18:06:19 +00:00
root	aebc98ead6	fix: remove trailing whitespace from litellm-api-keys.prose.md	2026-07-18 17:48:52 +00:00
root	17a77e6b3f	fix: fleet config issues from 2026-07-18 relay review ... - hermes-agent-baseline: add gpu-dense and gpu-light to models list - hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add vault empty-key guard documentation in Rule 13 - litellm-api-keys: fix pgrep pattern in auditable check - scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper Addresses issues found during relay inbox resolution session: 1. pgrep -f 'hermes_cli.main gateway run' matches both the real python gateway and the infisical bash wrapper, causing false health readings 2. infisical vault stores empty key silently — no guard/monitoring 3. gpu-dense/gpu-light stable aliases missing from baseline config	2026-07-18 17:39:46 +00:00
root	23f3f378c5	zulip-resilience: add fleet-wide audit findings and fixes from 2026-07-18 ... - Added Incident Log section documenting fleet-wide Zulip audit - Abiba: poll timeout AbortError fix (returns [] instead of error) - Abiba: credential fallback .env file for Infisical outages - Tanko: full gateway restart to recover Zulip connection - Fleet health metrics summary table - Hermes agent improvement recommendations (circuit breaker, credential fallback, queue re-registration, watchdog) - Abiba pi extension v2.1 resilience feature matrix	2026-07-18 16:33:12 +00:00
jerome	14d27a09b5	Merge pull request 'gpu-self-heal: refresh to current fleet baseline and topology' (#21) from feat/gpu-self-heal-refresh-20260718 into master ... Reviewed-on: #21	2026-07-18 08:36:37 +00:00
root	bddbb22f03	gpu-self-heal: refresh to current fleet baseline and topology ... - Synced model assignments to 2026-07-17 swaps (ThinkingCap, HauhauCS QAT, Genesis Hermes V3) - Added stable role-based aliases from gpu-fleet (gpu-dense, gpu-light, strix-moe) - Updated benchmark baselines to live values (74.9/169.6/62.9 tok/s) - Replaced router (port 9000) references with LiteLLM + direct routing - Replaced Prometheus exporter rule with sidecar health probe - Updated VRAM thresholds to match operational data (300/300/200 MB/h) - Added response size limit (1MB) to prevent OOM crashes - Added Lessons L5 (response size crash) and L6 (stable aliases) - Removed deprecated Rules 11-12 (router-specific distribution balance)	2026-07-18 08:07:03 +00:00
root	31ec70ae36	gpu-fleet: RTX 3090 swap to ThinkingCap-Qwen3.6-27B Q4_K_M ... - Model: Qwopus Q4_K_M (16GB, 63 tok/s) → ThinkingCap Q4_K_M (15.7GB, 68 tok/s) - RL-finetuned: 50% fewer thinking tokens, 0.85 MMLU-Pro (vs 0.83 base) - Self-spec MTP (n=4) REQUIRED for stability — segfaults without it - Added vision via mmproj (0.9GB) — new capability for this GPU - VRAM: 20.9/24.6GB (85%), tighter but stable - Outputs reasoning_content (hidden from Hermes agent)	2026-07-17 15:50:50 +00:00
root	5eb6d3bfbd	gpu-fleet: RTX 5070 swap to HauhauCS Gemma4-12B QAT Uncensored Balanced ... - Model: IQ4_NL (6.3GB, 191 tok/s) → Q4_K_M QAT (6.9GB, 87 tok/s) - MTP draft: Q8_0 (444MB) → tuned draft (242MB), saves 200MB VRAM - mmproj: F16 → BF16 (same size, matched to new model) - Benefits: 0/465 refusals, agent-optimized tuning, QAT quality - Trade: 54% slower generation (acceptable for gpu-light role) - Config: --parallel 1, --ctx-size 131072, single-slot full 128K - VRAM: 10.0/12.2GB (82%), healthy headroom	2026-07-17 15:24:13 +00:00
jerome	33cb88d571	Merge pull request 'feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo' (#20) from feat/gpu-128k-genesis-hermes-v3-20260717 into master ... Reviewed-on: #20	2026-07-17 11:21:32 +00:00
root	4a1f476623	fix: add missing description to inference-optimization frontmatter	2026-07-17 11:11:27 +00:00
root	ba2c55c7e6	ci: re-trigger pipeline for PR #20	2026-07-17 11:10:29 +00:00
root	b9149bce47	feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo ... GPU Changes: - All 3 GPUs reduced from 256K (-c 262144) to 128K (-c 131072) for stability - Observed instability near 100K at 256K — 128K is the stable ceiling - VRAM improved: RTX 3090 ~70% (was 90%), RTX 5070 ~65% (was 88%) - Strix Halo swapped to LuffyTheFox/Genesis Hermes V3 APEX - Hermes agent fine-tune, tensor repair (3 SSM layers, 76% W1 improvement) - Uncensored (0/465 refusals), multimodal (mmproj F16) - Speed: 65 tok/s gen, 140 tok/s prompt - Alias strix-moe maintained Agent Updates: - Mumuni: max_context_window 262144→131072, already aligned on strix-moe/0.65 - Tanko: max_context_window 262144→131072 - Koonimo: max_context_window + context_length 262144→131072 - CT114 SSH access confirmed (was 'Zulip only') LiteLLM (CT116): - Updated backend model references qwen3.6-35B-udq4→strix-moe - Removed stale ornith-1.0-35b from model_cost - Fallback chains updated Contracts Updated: - gpu-fleet.prose.md: topology, VRAM, benchmarks, config lines, model assignments - gpu-self-heal.prose.md: Rule 9/10 context targets - hermes-config-template.prose.md: template values, Rules 7-9, compression thresholds - inference-optimization.prose.md: added to repo, 128K recommendation Compression: 0.65 fires at ~85K (~43K headroom before 128K ceiling) For >128K workloads: route to external providers (deepseek)	2026-07-17 10:59:45 +00:00
root	65c99dab50	contract: update litellm-api-keys to v2026-07-17 — fleet standardization ... - Document canonical systemd drop-in pattern (ExecStart= reset + wrapper) - Hardcode venv paths — never use variables in single-quoted bash -c - Update migration status: all 4 agents on while-true wrapper + st.8e848433 - Infisical CLI update procedure (0.38.0 → 0.43.109 via artifacts-cli) - Service token inventory + .env fallback inventory - Key rotation log: fleet standardize + tanko fix-zulip entries - Remove git merge conflict artifacts - Add fleet-wide standardization lessons section	2026-07-17 10:17:23 +00:00
jerome	20cbb96e2d	Merge pull request 'Vault cleanup + contract sync (WAL #1316)' (#19) from fix/vault-cleanup-contract-sync into master ... Reviewed-on: #19	2026-07-16 21:24:59 +00:00
root	8215b84f88	merge: resolve conflicts with master (delegation delete + gpu-fleet)	2026-07-16 21:22:34 +00:00
jerome	24cd7f72ac	Merge branch 'master' into feat/zulip-v3-resilience	2026-07-16 21:20:24 +00:00
jerome	d6e85e68e9	Merge pull request 'UPDATE 2026-07-15: Full GPU fleet rebuild + stable aliases' (#18) from feat/gpu-fleet-rebuild-20260715 into master ... Reviewed-on: #18	2026-07-16 21:17:12 +00:00
root	b7e23e2592	vault: Infisical cleanup + contract sync (WAL #1316) ... - litellm-api-keys: vault audit, tanko/koby/koonimo migration, key rotation log - gpu-fleet: ornith-1.0-35b→strix-moe (remaining refs) - hermes-agent-baseline: 256K all GPUs, CT IPs updated, Shumba retired - delegation-prose-contract: removed (renamed to mumuni-delegation) - contract-registry.yaml + cron-prompts-review.md: from feat/contract-registry	2026-07-16 21:17:09 +00:00
root	369c312ccb	vault: BAGGY_LITELLM_API_KEY deleted — one secret per agent	2026-07-16 19:54:51 +00:00
root	7f39a444d6	vault: BAGGY+LITELLM synced, deletion needs Infisical UI	2026-07-16 19:28:28 +00:00
Abiba	17f4c79fe4	vault: sync BAGGY_LITELLM_API_KEY = KOONIMO_LITELLM_API_KEY (same agent CT113) ... Baggy = Koonimo (CT 113). The vault had two divergent keys: - BAGGY_LITELLM_API_KEY = sk-QT-kDt8Szo... (stale, 401) - KOONIMO_LITELLM_API_KEY = sk-OEK7z26n6... (valid, 200, session-13 rotation) Synced BAGGY to match KOONIMO (the valid key). Added contract note: both secrets MUST mirror each other. This was the root cause of Koonimo's 401 after migration — the ghost process had BAGGY in its env instead of KOONIMO.	2026-07-16 19:22:48 +00:00
Abiba	a9caf5b216	vault: fix koby platform connectivity + document migration lessons ... Koby was broken after migration — Zulip credentials lost (only in process memory), Telegram token overwritten (only existed in old .env.bak-20260603). Fix: Koby shares Tanko's Zulip bot (tanko-bot@, TANKO_ZULIP_API_KEY in vault). Telegram token recovered from .env.bak-20260603. Both platforms now connected. Added Koby migration lessons to contract (back up .env before migration, inject ALL platform env vars, cat /proc/pid/environ before killing old gateway). Added KOBY_ZULIP_API_KEY to vault (shares Tanko's key).	2026-07-16 18:19:27 +00:00
Abiba	15a826457e	vault: canonical Production Vault Access Process + sync vault + migrate koby/koonimo ... - Vault SYNCED: mumuni/koby/koonimo rotated keys written to Infisical (all validate 200). Vault is source of truth again (was stale since session-13 rotations). - Koby (.129) + Koonimo (.114) migrated from hardcoded systemd drop-ins to infisical-gateway.sh wrapper (live vault injection, .env fallback safety net). 4/5 agents now vault-backed (abiba/mumuni/koby/koonimo). - New § Production Vault Access Process: canonical 7-step pattern, migration status table, key rotation procedure, why-it's-non-fail rationale. - Fixed stale notes: 'vault sync PENDING' (now synced), 'Abiba key IS master key' (now a proper agent key sk-sxbphLvk1OU). - Tanko (user jerome, not systemd) documented as pending migration. - hermes-key-enforcement: cross-ref to canonical process, drop-ins deprecated.	2026-07-16 17:51:53 +00:00
Abiba	cd52deda92	contracts: stable alias chain + no-master-key-for-inference rule ... Agent configs aligned to stable aliases (gpu-light/gpu-dense/strix-moe/syslog-auto): - Tanko fixed: base_url :4000->/v1 (nginx), max_context_window 131072->262144, aux.compression gemma->strix-moe, aux gemma->gpu-light - Mumuni aligned: aux gemma->gpu-light, delegation/x_search qwen->gpu-dense - Koby/Koonimo already used gpu-light (validated) - Zero raw model names remain in any agent config No-master-key rule: litellm_proxy_master_key (sk-litellm-...) is admin-only (/key/list, /key/generate). All inference uses agent keys. Health-check model tests use dedicated monitor key (/etc/litellm-monitor.env on CT 116). Fixed: daily-infra-report.py, gpu-self-heal.py, litellm-health-check.sh. Fixed wrong key-scope note: qwen3.6-35B-udq4 IS in scope for baggy/koby/mumuni/ abiba-pi keys (not 'NOT in allowlist' as previously documented).	2026-07-16 17:39:05 +00:00
Abiba	97f1cd77e4	litellm-self-heal: document 2026-07-16 ops sync (script fixes, systemd gpu-monitor, live-key agent check) ... Health-check script: critical-only gpu-fleet alerting, strix-moe model test. gpu-monitor: systemd unit (was bare &), Strix counted in gpu_count, VRAM thresholds raised (93/97 — 256K ctx steady-state ~96% on RTX 3090). agent-health-check.py: reads live LITELLM_API_KEY from gateway env (no hardcoded keys). daily-infra-report.py: stale SYNTHETIC key → env-based master key.	2026-07-16 17:23:37 +00:00
Abiba	dc572889f8	contracts: sync to ground truth — ornith-1.0-35b→strix-moe, 256K all GPUs, real LiteLLM timeouts ... Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts: - AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist - All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1) - LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s) - Added LiteLLM model surface + key scoping to litellm-self-heal - Patched health-check script path ref Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal, zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline, delegation-prose-contract, mumuni-delegation-prose-contract	2026-07-16 17:02:44 +00:00
Abiba	d0feb7881e	Contracts: Rule 13 two key-injection patterns + koby/baggy rotation log (WAL #1300)	2026-07-16 15:10:50 +00:00
Abiba	9fd8c68bd2	Contracts: replace remaining ornith-1.0-35b refs with strix-moe	2026-07-16 13:57:57 +00:00
Abiba	39e2fa0cfa	Contracts: Mumuni context-fix (WAL #1300) — strix-moe, 256K all GPUs, Rule 12/13, machine-identity vault procedure	2026-07-16 13:57:31 +00:00
root	622cf7b176	UPDATE 2026-07-15: Full fleet rebuild ... - Stable role-based aliases: strix-moe, gpu-dense, gpu-light - Strix Halo: ornith-1.0-35b -> unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 256K) - RTX 5070: Q4_K_M -> IQ4_NL+MTP (256K, 125-213 tok/s, 2x faster) - RTX 3090: bumped to 256K context - Provider rename: harness -> litellm (all agents) - syslog-auto: weighted pool restored with direct GPU routing - Mumuni agent profile documented - Self-heal thresholds updated for 256K context - All model contexts corrected (RTX 3090 was 131K, not 256K)	2026-07-15 16:04:21 +00:00
root	30bf42b841	feat: Zulip v3 resilience contract + restore playbook ... - zulip-resilience-v3.prose.md: Production resilience rewrite contract covering circuit breaker, retry with jitter, queue lifecycle management, supervisor watchdog, and PM2 hardening. Research-backed from Zulip event system docs. - abiba-zulip-restore.prose.md: Quick-restore playbook for recovery scenarios.	2026-07-13 21:29:35 +00:00
abiba-bot	190ceb9be4	Merge pull request 'GPU workload redistribution + lessons learned July 2026' (#15) from feat/gpu-workload-compression-jul2026 into master	2026-07-12 23:40:12 +00:00
root	0c298eb9d9	feat: add routing configuration with RPM caps (ornith syslog-auto: 40→60) ... - Strix Halo ornith RPM increased from 40 to 60 inside syslog-auto pool - Prevented unbounded ornith traffic when multiple agents use syslog-auto - Direct ornith remains at RPM=40 - All models verified healthy (ornith 80ms, gemma 110ms, qwen 840ms) - Added full routing table to gpu-fleet contract	2026-07-12 23:38:21 +00:00
root	af41f8f57a	fix: remaining template YAML and topology table corrections ... - hermes-config-template: Template YAML compression model gemma→ornith. Context guidance updated (qwen=256K, only gemma limited to 131K). - litellm-self-heal: GPU topology table fixed (RTX 3090: 128K→256K, parallel 2→1)	2026-07-12 22:55:24 +00:00
root	be02b0e843	fix: stale dates, context values, and architecture references ... - gpu-fleet: Updated all context references (128K→256K for RTX 3090). Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram (router labeled deprecated/not in path). Corrected VRAM values. Updated dates: June→July 2026. - hermes-config-template: Fixed description (removed incorrect '128K on NVIDIA' claim). Updated to reflect verified 256K on RTX 3090. - litellm-self-heal: Updated last-verified date. - litellm-api-keys: Updated last-verified date.	2026-07-12 22:54:30 +00:00
root	79d4a73895	docs: lessons learned from 2026-07-12 session ... - gpu-fleet: Corrected architecture (direct GPU, no router in path). Updated context values (RTX 3090=256K, not 128K). Added api-key standardization requirement. - gpu-self-heal: Added Lessons Learned section with 5 critical findings: L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed) L2: Fallback chain cascading failure loop detection L3: Verify running state, not documentation L4: Infisical fallback requirement (.env must have uncommented key) L5: Zulip event queue can silently die after ~40 reconnects - litellm-self-heal: Updated status manual-only→deployed, cron schedule - litellm-api-keys: Added Infisical token expiry warning + .env fallback - hermes-config-template: Rule 3 updated with .env fallback requirement	2026-07-12 22:49:39 +00:00
root	19b6db9891	feat: GPU workload redistribution — compression → Strix Halo ... - Move compression model from gemma-4-12b (RTX 5070) to ornith-1.0-35b (Strix Halo) - Add Rule 8: GPU Workload Distribution — per-GPU role assignment - Add Rule 9: Compression Threshold for 256K models - Update Rule 7: Auxiliary Model Consistency with new compression routing - Add gpu-self-heal.prose.md contract with 10 remediation rules - Strix Halo (64GB, 256K, 72.4 tok/s) → compression specialist - RTX 5070 (12GB) → vision/web search specialist - RTX 3090 (24GB, 256K) → heavy reasoning specialist - All rules grilled and confirmed with Kwame 2026-07-12	2026-07-12 22:27:15 +00:00
jerome	22eaaf4254	Merge pull request 'fix: add data source integrity rule to delegation contract' (#14) from feat/data-source-integrity-rule into master ... Reviewed-on: #14	2026-07-12 18:09:10 +00:00
jerome	fdb22948d9	fix: add data source integrity rule to delegation contract ... Workers MUST use provided input data, not fetch external sources. Adds: Data Source Integrity section, dispatch guidance example, anti-pattern entry. Root cause of e2e test discrepancies where writer worker queried Proxmox API instead of using raw data.	2026-07-11 18:51:30 -04:00
jerome	e9968e165d	Merge pull request 'feat: add RA-H OS Custodianship Contract for shared memory protocol' (#13) from feat/ra-h-os-custodianship-contract into master ... Reviewed-on: #13	2026-07-10 23:39:30 +00:00