189 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
root	3b6cf44a30	fix: update all Mumuni IP references from .123 to .24 (inside Abiba CT100) ... Mumuni CT114 destroyed. Mumuni now runs inside Abiba CT100 at 192.168.68.24. Updated all contract files and agent-health-check.py.	2026-07-28 08:56:45 +00:00
root	22b015e182	fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)	2026-07-27 20:25:51 +00:00
mumuni-bot	29e32340a4	Merge pull request 'fix: swap gpu-dense to SmartCode-Fable-5 (ThinkingCap + Fable 5 CoT)' (#33) from fix/gpu-dense-smartcode-fable5 into master	2026-07-27 16:43:46 +00:00
root	179529de71	fix: swap gpu-dense to SmartCode-Fable-5 (ThinkingCap + Fable 5 CoT) ... gpu-dense on RTX 3090 swapped from qwen3.6-27B-code to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled (UD-Q4_K_XL, ~17.9GB).\n\nImprovements:\n- ~50% fewer thinking tokens via ThinkingCap finetune\n- Fable 5 CoT distillation for improved coding reasoning\n- Same 27B base, fits RTX 3090 at ~73% VRAM\n- Recommended samplers: temp 0.9, top-p 0.95, top-k 60\n- Context: 128K (fleet standard)	2026-07-27 16:43:29 +00:00
mumuni-bot	fb45007ace	Merge pull request 'fix: remove mumuni from health check — now inside Abiba CT 100' (#32) from fix/update-health-check-remove-mumuni into master	2026-07-26 12:37:35 +00:00
root	0f572ff9f2	fix: remove mumuni from health check — now inside Abiba CT 100	2026-07-26 12:37:07 +00:00
mumuni-bot	3a8e7d9b3a	Merge pull request 'fix: remove CT 114 (mumuni) — destroyed, moved inside CT 100' (#31) from fix/remove-mumuni-ct114 into master	2026-07-26 12:36:31 +00:00
root	0fb54926f6	fix: remove CT 114 (mumuni) — destroyed, moved inside CT 100	2026-07-26 12:36:02 +00:00
mumuni-bot	146abf7f82	Merge pull request 'fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness' (#30) from fix/agent-health-check-v2 into master	2026-07-26 12:27:20 +00:00
root	c869e75c61	ci: trigger recheck	2026-07-26 12:24:05 +00:00
root	47aa92bee3	fix: verification protocol findings — CT 118 static IP, .110 llama-server restored ... Verification protocol run 2026-07-26: - CT 118 (jdownloader): DHCP had moved it to .131 post-reboot. Now set to static .20. - RTX 5070 (.110) llama-server: was stopped (disabled unit). Started and verified — gemma-4-12b responding through LiteLLM. - All 6 PVE nodes confirmed at correct IPs. - All 18 CTs confirmed at correct IPs on correct nodes. - All public endpoints responding (meet, chat, git, litellm, vault, auth). - Container counts verified: docker-vm 14 ctrs, CT 116 10 ctrs, VPS 5 ctrs.	2026-07-26 12:20:58 +00:00
root	de9adb13cf	fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness ... Gaps fixed: - Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped - Agent key lookup uses correct {NAME}_LITELLM_API_KEY format - New CT liveness check via pct status on PVE nodes - New config YAML integrity check via yaml.safe_load() - New wrapper/CLI integrity check (hermes wrapper, infisical path, hermes-real) - New vault secret non-emptiness check - Ops escalation: failures produce ALERT lines for cron capture	2026-07-26 12:06:32 +00:00
root	c380196fab	fix: address review findings ... - Remove kagentz from amdpve node table (belongs on hwepve) - Fix kagentz pct-run table: hwepve (was amdpve) - Fix mumuni pct-run table: hwepve (was minipve) - Fix Zulip recovery command: wrap in single SSH call	2026-07-24 20:11:32 +00:00
root	b17c60f997	fix: contract accuracy updates post fleet-wide reboot ... infrastructure-control.prose.md: - Add hwepve as 6th Proxmox node - Fix Gitea IP: .17 (was .110) - Fix AdGuard IP: .10 on minipve (was .102 on acerpve) - Fix Abiba placement: hwepve (was amdpve) - Fix Mumuni placement: hwepve (was minipve) - Fix Authentik port: add :9000 - Add CT 118 (jdownloader), CT 119 (infisical-vault) - Add last-verified date (2026-07-24) infrastructure-update.prose.md: - Add post-reboot CT sweep procedure - Add Zulip Docker network recovery steps - Add WireGuard tunnel verification scripts/netbird-add-domain.sh: - New script to register domains in Netbird proxy store.db	2026-07-24 20:01:43 +00:00
abiba-bot	2f0c3c1850	fix: update contracts for hwepve 6th node + Mumuni recovery	2026-07-23 18:47:17 +00:00
jerome	cedbdc465d	Merge branch 'master' into fm/mumuni-recovery-hwepve	2026-07-23 18:11:06 +00:00
abiba-bot	8b09c78efd	Merge: PR #26 conflict resolution — hwepve 6th node + qwen model ref	2026-07-23 18:10:36 +00:00
root	767ab25128	merge: resolve PR #26 conflict — keep qwen model ref + add hwepve node	2026-07-23 18:10:18 +00:00
abiba-bot	75381f9737	Captain decision batch 2026-07-21: ornith decommission, 256K→128K context, Mumuni Discord disable ... Three contract updates per captain decision batch. Validated through no-mistakes pipeline (green).	2026-07-23 09:13:06 +00:00
root	2b9b545ca9	no-mistakes(document): Fixed 2 stale model-name references in doc files	2026-07-23 09:10:43 +00:00
root	e6c52bf071	no-mistakes(review): Fix F1 model rename and F2 128K compaction math in contracts	2026-07-23 09:04:51 +00:00
root	1c44bf1259	contract updates: ornith decommissioned, 256K→128K context, Mumuni Discord disabled ... Change 1: Strix Halo — ornith decommissioned - gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout - inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4 - gpu-monitor: ornith status → Strix Halo status - infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4 - infrastructure-update: ornith→strix-moe via router - proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe) Change 2: GPU context 256K→128K fleet-wide - hermes-agent-baseline: frontmatter description updated - litellm-health: GPU Fleet Topology table 256K→128K - litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert - inference-optimization: compress threshold 256K→128K compact at 85K - gpu-fleet: instability note updated Change 3: Mumuni Discord platform disabled - gpu-fleet: Mumuni platforms: removed discord	2026-07-23 09:00:42 +00:00
root	9f0e04f22c	fix: update contracts for hwepve 6th node + Mumuni migration (lxc/114 on hwepve) ... - proxmox-monitor: 5→6 nodes, add hwepve (.4, Huawei Matebook 16), note Mumuni (lxc/114) placement - zulip-health: add dual-gateway collision detection to B2, note Mumuni on hwepve - hermes-agent-baseline: fix Mumuni node minipve→hwepve - hermes-zulip-restore: add hwepve Proxmox column for Mumuni - mumuni-delegation: fix CT 118/storepve/.6 → lxc/114/hwepve/.123, 5→6 nodes - pm2-self-heal: verified — no Mumuni references, no changes needed Post-migration discoveries: - Mumuni container migrated from minipve to hwepve (6th node, 192.168.68.4) - IP preserved (192.168.68.123), hostname 'mumuni' unchanged - Dual gateway collision (PID 1442 --replace + PID 2924 --force) resolved by container restart at 10:07; gateway restarted clean at 13:06 with fresh Zulip queue (errors=0, reconnects=0)	2026-07-20 17:08:58 +00:00
jerome	fc88265e76	Merge pull request 'tune: switch compression model from strix-moe to syslog-auto' (#25) from tune/compression-syslog-auto into master ... Reviewed-on: #25	2026-07-19 01:32:44 +00:00
jerome	51da19d92d	Merge pull request 'feat(contracts): add infrastructure-maintenance contract' (#24) from fm/infra-maint-contract into master ... Reviewed-on: #24	2026-07-19 01:32:29 +00:00
root	6570fd60e7	tune: switch compression model from strix-moe to syslog-auto ... Relieve Strix Halo pressure by distributing compression across the syslog-auto weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070). Updates: - compression.model: strix-moe -> syslog-auto - auxiliary.compression.model: strix-moe -> syslog-auto - Rule 7: Updated for syslog-auto compression, removed aux prohibition - Rule 8: Updated GPU workload distribution - All docs/comments updated to reflect the change	2026-07-18 23:28:07 +00:00
jerome	5d2ecbace6	Merge branch 'master' into fm/infra-maint-contract	2026-07-18 23:12:20 +00:00
root	99789a00a1	no-mistakes(document): Reframed infra-maintenance contract as deliberate partition	2026-07-18 22:16:09 +00:00
root	a68879e904	no-mistakes(review): fix docker rollback command to pin digest then compose up	2026-07-18 22:07:15 +00:00
jerome	06d2bcbc9e	Merge pull request 'fix: fleet config issues from 2026-07-18 relay review' (#23) from fix/fleet-config-issues-20260718 into master ... Reviewed-on: #23	2026-07-18 22:06:48 +00:00
jerome	c4a8c45835	Merge pull request 'zulip-resilience: fleet-wide audit findings and fixes 2026-07-18' (#22) from feat/zulip-resilience-audit-20260718 into master ... Reviewed-on: #22	2026-07-18 22:06:34 +00:00
root	1d027f71f6	feat(contracts): add infrastructure-maintenance contract ... New responsibility contract consolidating host-level system maintenance and Docker image lifecycle management, filling the gap left by infrastructure-update (which owns cluster-wide apt waves). Scope: - OS package updates on primary host with pre-update snapshot/backup check - Docker image pulls for LiteLLM, SearXNG, and other running containers - Container restarts with per-stack health verification - Post-update verification: LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways - Rollback on failure (image/apt/config restore) with circuit breaker Owner: ops (firstmate secondmate). Trigger: weekly Sunday 2am ET. Escalation: warning/critical->abiba+mumuni, fatal->abiba+mumuni+kwame. circuit_breaker: max_retries 2, window 7200, trip_action escalate_to_fatal. depends_on: infrastructure-monitoring (pre-update health baseline). Registry: - Add infrastructure-maintenance to by_category.maintenance, by_domain.infrastructure, by_owner.ops (new), by_trigger.scheduled, by_sensitivity.high - Add 'ops' to owners list - Move infrastructure-update owner abiba -> ops (in contracts entry + by_owner index) Also adds '## Maintaining this file' section to AGENTS.md per fm-ensure-agents-md.	2026-07-18 22:02:45 +00:00
jerome	a74229ee74	Merge branch 'master' into fix/fleet-config-issues-20260718	2026-07-18 18:06:19 +00:00
root	aebc98ead6	fix: remove trailing whitespace from litellm-api-keys.prose.md	2026-07-18 17:48:52 +00:00
root	17a77e6b3f	fix: fleet config issues from 2026-07-18 relay review ... - hermes-agent-baseline: add gpu-dense and gpu-light to models list - hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add vault empty-key guard documentation in Rule 13 - litellm-api-keys: fix pgrep pattern in auditable check - scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper Addresses issues found during relay inbox resolution session: 1. pgrep -f 'hermes_cli.main gateway run' matches both the real python gateway and the infisical bash wrapper, causing false health readings 2. infisical vault stores empty key silently — no guard/monitoring 3. gpu-dense/gpu-light stable aliases missing from baseline config	2026-07-18 17:39:46 +00:00
root	23f3f378c5	zulip-resilience: add fleet-wide audit findings and fixes from 2026-07-18 ... - Added Incident Log section documenting fleet-wide Zulip audit - Abiba: poll timeout AbortError fix (returns [] instead of error) - Abiba: credential fallback .env file for Infisical outages - Tanko: full gateway restart to recover Zulip connection - Fleet health metrics summary table - Hermes agent improvement recommendations (circuit breaker, credential fallback, queue re-registration, watchdog) - Abiba pi extension v2.1 resilience feature matrix	2026-07-18 16:33:12 +00:00
jerome	14d27a09b5	Merge pull request 'gpu-self-heal: refresh to current fleet baseline and topology' (#21) from feat/gpu-self-heal-refresh-20260718 into master ... Reviewed-on: #21	2026-07-18 08:36:37 +00:00
root	bddbb22f03	gpu-self-heal: refresh to current fleet baseline and topology ... - Synced model assignments to 2026-07-17 swaps (ThinkingCap, HauhauCS QAT, Genesis Hermes V3) - Added stable role-based aliases from gpu-fleet (gpu-dense, gpu-light, strix-moe) - Updated benchmark baselines to live values (74.9/169.6/62.9 tok/s) - Replaced router (port 9000) references with LiteLLM + direct routing - Replaced Prometheus exporter rule with sidecar health probe - Updated VRAM thresholds to match operational data (300/300/200 MB/h) - Added response size limit (1MB) to prevent OOM crashes - Added Lessons L5 (response size crash) and L6 (stable aliases) - Removed deprecated Rules 11-12 (router-specific distribution balance)	2026-07-18 08:07:03 +00:00
root	31ec70ae36	gpu-fleet: RTX 3090 swap to ThinkingCap-Qwen3.6-27B Q4_K_M ... - Model: Qwopus Q4_K_M (16GB, 63 tok/s) → ThinkingCap Q4_K_M (15.7GB, 68 tok/s) - RL-finetuned: 50% fewer thinking tokens, 0.85 MMLU-Pro (vs 0.83 base) - Self-spec MTP (n=4) REQUIRED for stability — segfaults without it - Added vision via mmproj (0.9GB) — new capability for this GPU - VRAM: 20.9/24.6GB (85%), tighter but stable - Outputs reasoning_content (hidden from Hermes agent)	2026-07-17 15:50:50 +00:00
root	5eb6d3bfbd	gpu-fleet: RTX 5070 swap to HauhauCS Gemma4-12B QAT Uncensored Balanced ... - Model: IQ4_NL (6.3GB, 191 tok/s) → Q4_K_M QAT (6.9GB, 87 tok/s) - MTP draft: Q8_0 (444MB) → tuned draft (242MB), saves 200MB VRAM - mmproj: F16 → BF16 (same size, matched to new model) - Benefits: 0/465 refusals, agent-optimized tuning, QAT quality - Trade: 54% slower generation (acceptable for gpu-light role) - Config: --parallel 1, --ctx-size 131072, single-slot full 128K - VRAM: 10.0/12.2GB (82%), healthy headroom	2026-07-17 15:24:13 +00:00
jerome	33cb88d571	Merge pull request 'feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo' (#20) from feat/gpu-128k-genesis-hermes-v3-20260717 into master ... Reviewed-on: #20	2026-07-17 11:21:32 +00:00
root	4a1f476623	fix: add missing description to inference-optimization frontmatter	2026-07-17 11:11:27 +00:00
root	ba2c55c7e6	ci: re-trigger pipeline for PR #20	2026-07-17 11:10:29 +00:00
root	b9149bce47	feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo ... GPU Changes: - All 3 GPUs reduced from 256K (-c 262144) to 128K (-c 131072) for stability - Observed instability near 100K at 256K — 128K is the stable ceiling - VRAM improved: RTX 3090 ~70% (was 90%), RTX 5070 ~65% (was 88%) - Strix Halo swapped to LuffyTheFox/Genesis Hermes V3 APEX - Hermes agent fine-tune, tensor repair (3 SSM layers, 76% W1 improvement) - Uncensored (0/465 refusals), multimodal (mmproj F16) - Speed: 65 tok/s gen, 140 tok/s prompt - Alias strix-moe maintained Agent Updates: - Mumuni: max_context_window 262144→131072, already aligned on strix-moe/0.65 - Tanko: max_context_window 262144→131072 - Koonimo: max_context_window + context_length 262144→131072 - CT114 SSH access confirmed (was 'Zulip only') LiteLLM (CT116): - Updated backend model references qwen3.6-35B-udq4→strix-moe - Removed stale ornith-1.0-35b from model_cost - Fallback chains updated Contracts Updated: - gpu-fleet.prose.md: topology, VRAM, benchmarks, config lines, model assignments - gpu-self-heal.prose.md: Rule 9/10 context targets - hermes-config-template.prose.md: template values, Rules 7-9, compression thresholds - inference-optimization.prose.md: added to repo, 128K recommendation Compression: 0.65 fires at ~85K (~43K headroom before 128K ceiling) For >128K workloads: route to external providers (deepseek)	2026-07-17 10:59:45 +00:00
root	65c99dab50	contract: update litellm-api-keys to v2026-07-17 — fleet standardization ... - Document canonical systemd drop-in pattern (ExecStart= reset + wrapper) - Hardcode venv paths — never use variables in single-quoted bash -c - Update migration status: all 4 agents on while-true wrapper + st.8e848433 - Infisical CLI update procedure (0.38.0 → 0.43.109 via artifacts-cli) - Service token inventory + .env fallback inventory - Key rotation log: fleet standardize + tanko fix-zulip entries - Remove git merge conflict artifacts - Add fleet-wide standardization lessons section	2026-07-17 10:17:23 +00:00
jerome	20cbb96e2d	Merge pull request 'Vault cleanup + contract sync (WAL #1316)' (#19) from fix/vault-cleanup-contract-sync into master ... Reviewed-on: #19	2026-07-16 21:24:59 +00:00
root	8215b84f88	merge: resolve conflicts with master (delegation delete + gpu-fleet)	2026-07-16 21:22:34 +00:00
jerome	24cd7f72ac	Merge branch 'master' into feat/zulip-v3-resilience	2026-07-16 21:20:24 +00:00
jerome	d6e85e68e9	Merge pull request 'UPDATE 2026-07-15: Full GPU fleet rebuild + stable aliases' (#18) from feat/gpu-fleet-rebuild-20260715 into master ... Reviewed-on: #18	2026-07-16 21:17:12 +00:00
root	b7e23e2592	vault: Infisical cleanup + contract sync (WAL #1316) ... - litellm-api-keys: vault audit, tanko/koby/koonimo migration, key rotation log - gpu-fleet: ornith-1.0-35b→strix-moe (remaining refs) - hermes-agent-baseline: 256K all GPUs, CT IPs updated, Shumba retired - delegation-prose-contract: removed (renamed to mumuni-delegation) - contract-registry.yaml + cron-prompts-review.md: from feat/contract-registry	2026-07-16 21:17:09 +00:00