180 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
root	87b4d67067	no-mistakes(review): Fix 5 review findings: duplicate table header, hwepve specs/Wave2 target, dangling ref, Authentik port	2026-07-24 21:31:28 +00:00
root	7d1db62a8e	no-mistakes(document): Fix stale CT placements across 5 docs matching infrastructure-control updates	2026-07-24 20:21:03 +00:00
root	9943be5e68	no-mistakes(test): Lint, shell syntax, and all 13 user-intent constraints pass on 3 changed files. One mode fix: netbird-add-domain.sh 644→755	2026-07-24 20:16:57 +00:00
root	fa26b7a579	no-mistakes(test): Fixed 7 cross-table inconsistencies: kagentz placement, mumuni placement, stale 5-node references	2026-07-24 20:14:49 +00:00
root	b17c60f997	fix: contract accuracy updates post fleet-wide reboot ... infrastructure-control.prose.md: - Add hwepve as 6th Proxmox node - Fix Gitea IP: .17 (was .110) - Fix AdGuard IP: .10 on minipve (was .102 on acerpve) - Fix Abiba placement: hwepve (was amdpve) - Fix Mumuni placement: hwepve (was minipve) - Fix Authentik port: add :9000 - Add CT 118 (jdownloader), CT 119 (infisical-vault) - Add last-verified date (2026-07-24) infrastructure-update.prose.md: - Add post-reboot CT sweep procedure - Add Zulip Docker network recovery steps - Add WireGuard tunnel verification scripts/netbird-add-domain.sh: - New script to register domains in Netbird proxy store.db	2026-07-24 20:01:43 +00:00
abiba-bot	2f0c3c1850	fix: update contracts for hwepve 6th node + Mumuni recovery	2026-07-23 18:47:17 +00:00
jerome	cedbdc465d	Merge branch 'master' into fm/mumuni-recovery-hwepve	2026-07-23 18:11:06 +00:00
abiba-bot	8b09c78efd	Merge: PR #26 conflict resolution — hwepve 6th node + qwen model ref	2026-07-23 18:10:36 +00:00
root	767ab25128	merge: resolve PR #26 conflict — keep qwen model ref + add hwepve node	2026-07-23 18:10:18 +00:00
abiba-bot	75381f9737	Captain decision batch 2026-07-21: ornith decommission, 256K→128K context, Mumuni Discord disable ... Three contract updates per captain decision batch. Validated through no-mistakes pipeline (green).	2026-07-23 09:13:06 +00:00
root	2b9b545ca9	no-mistakes(document): Fixed 2 stale model-name references in doc files	2026-07-23 09:10:43 +00:00
root	e6c52bf071	no-mistakes(review): Fix F1 model rename and F2 128K compaction math in contracts	2026-07-23 09:04:51 +00:00
root	1c44bf1259	contract updates: ornith decommissioned, 256K→128K context, Mumuni Discord disabled ... Change 1: Strix Halo — ornith decommissioned - gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout - inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4 - gpu-monitor: ornith status → Strix Halo status - infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4 - infrastructure-update: ornith→strix-moe via router - proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe) Change 2: GPU context 256K→128K fleet-wide - hermes-agent-baseline: frontmatter description updated - litellm-health: GPU Fleet Topology table 256K→128K - litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert - inference-optimization: compress threshold 256K→128K compact at 85K - gpu-fleet: instability note updated Change 3: Mumuni Discord platform disabled - gpu-fleet: Mumuni platforms: removed discord	2026-07-23 09:00:42 +00:00
root	9f0e04f22c	fix: update contracts for hwepve 6th node + Mumuni migration (lxc/114 on hwepve) ... - proxmox-monitor: 5→6 nodes, add hwepve (.4, Huawei Matebook 16), note Mumuni (lxc/114) placement - zulip-health: add dual-gateway collision detection to B2, note Mumuni on hwepve - hermes-agent-baseline: fix Mumuni node minipve→hwepve - hermes-zulip-restore: add hwepve Proxmox column for Mumuni - mumuni-delegation: fix CT 118/storepve/.6 → lxc/114/hwepve/.123, 5→6 nodes - pm2-self-heal: verified — no Mumuni references, no changes needed Post-migration discoveries: - Mumuni container migrated from minipve to hwepve (6th node, 192.168.68.4) - IP preserved (192.168.68.123), hostname 'mumuni' unchanged - Dual gateway collision (PID 1442 --replace + PID 2924 --force) resolved by container restart at 10:07; gateway restarted clean at 13:06 with fresh Zulip queue (errors=0, reconnects=0)	2026-07-20 17:08:58 +00:00
jerome	fc88265e76	Merge pull request 'tune: switch compression model from strix-moe to syslog-auto' (#25) from tune/compression-syslog-auto into master ... Reviewed-on: #25	2026-07-19 01:32:44 +00:00
jerome	51da19d92d	Merge pull request 'feat(contracts): add infrastructure-maintenance contract' (#24) from fm/infra-maint-contract into master ... Reviewed-on: #24	2026-07-19 01:32:29 +00:00
root	6570fd60e7	tune: switch compression model from strix-moe to syslog-auto ... Relieve Strix Halo pressure by distributing compression across the syslog-auto weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070). Updates: - compression.model: strix-moe -> syslog-auto - auxiliary.compression.model: strix-moe -> syslog-auto - Rule 7: Updated for syslog-auto compression, removed aux prohibition - Rule 8: Updated GPU workload distribution - All docs/comments updated to reflect the change	2026-07-18 23:28:07 +00:00
jerome	5d2ecbace6	Merge branch 'master' into fm/infra-maint-contract	2026-07-18 23:12:20 +00:00
root	99789a00a1	no-mistakes(document): Reframed infra-maintenance contract as deliberate partition	2026-07-18 22:16:09 +00:00
root	a68879e904	no-mistakes(review): fix docker rollback command to pin digest then compose up	2026-07-18 22:07:15 +00:00
jerome	06d2bcbc9e	Merge pull request 'fix: fleet config issues from 2026-07-18 relay review' (#23) from fix/fleet-config-issues-20260718 into master ... Reviewed-on: #23	2026-07-18 22:06:48 +00:00
jerome	c4a8c45835	Merge pull request 'zulip-resilience: fleet-wide audit findings and fixes 2026-07-18' (#22) from feat/zulip-resilience-audit-20260718 into master ... Reviewed-on: #22	2026-07-18 22:06:34 +00:00
root	1d027f71f6	feat(contracts): add infrastructure-maintenance contract ... New responsibility contract consolidating host-level system maintenance and Docker image lifecycle management, filling the gap left by infrastructure-update (which owns cluster-wide apt waves). Scope: - OS package updates on primary host with pre-update snapshot/backup check - Docker image pulls for LiteLLM, SearXNG, and other running containers - Container restarts with per-stack health verification - Post-update verification: LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways - Rollback on failure (image/apt/config restore) with circuit breaker Owner: ops (firstmate secondmate). Trigger: weekly Sunday 2am ET. Escalation: warning/critical->abiba+mumuni, fatal->abiba+mumuni+kwame. circuit_breaker: max_retries 2, window 7200, trip_action escalate_to_fatal. depends_on: infrastructure-monitoring (pre-update health baseline). Registry: - Add infrastructure-maintenance to by_category.maintenance, by_domain.infrastructure, by_owner.ops (new), by_trigger.scheduled, by_sensitivity.high - Add 'ops' to owners list - Move infrastructure-update owner abiba -> ops (in contracts entry + by_owner index) Also adds '## Maintaining this file' section to AGENTS.md per fm-ensure-agents-md.	2026-07-18 22:02:45 +00:00
jerome	a74229ee74	Merge branch 'master' into fix/fleet-config-issues-20260718	2026-07-18 18:06:19 +00:00
root	aebc98ead6	fix: remove trailing whitespace from litellm-api-keys.prose.md	2026-07-18 17:48:52 +00:00
root	17a77e6b3f	fix: fleet config issues from 2026-07-18 relay review ... - hermes-agent-baseline: add gpu-dense and gpu-light to models list - hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add vault empty-key guard documentation in Rule 13 - litellm-api-keys: fix pgrep pattern in auditable check - scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper Addresses issues found during relay inbox resolution session: 1. pgrep -f 'hermes_cli.main gateway run' matches both the real python gateway and the infisical bash wrapper, causing false health readings 2. infisical vault stores empty key silently — no guard/monitoring 3. gpu-dense/gpu-light stable aliases missing from baseline config	2026-07-18 17:39:46 +00:00
root	23f3f378c5	zulip-resilience: add fleet-wide audit findings and fixes from 2026-07-18 ... - Added Incident Log section documenting fleet-wide Zulip audit - Abiba: poll timeout AbortError fix (returns [] instead of error) - Abiba: credential fallback .env file for Infisical outages - Tanko: full gateway restart to recover Zulip connection - Fleet health metrics summary table - Hermes agent improvement recommendations (circuit breaker, credential fallback, queue re-registration, watchdog) - Abiba pi extension v2.1 resilience feature matrix	2026-07-18 16:33:12 +00:00
jerome	14d27a09b5	Merge pull request 'gpu-self-heal: refresh to current fleet baseline and topology' (#21) from feat/gpu-self-heal-refresh-20260718 into master ... Reviewed-on: #21	2026-07-18 08:36:37 +00:00
root	bddbb22f03	gpu-self-heal: refresh to current fleet baseline and topology ... - Synced model assignments to 2026-07-17 swaps (ThinkingCap, HauhauCS QAT, Genesis Hermes V3) - Added stable role-based aliases from gpu-fleet (gpu-dense, gpu-light, strix-moe) - Updated benchmark baselines to live values (74.9/169.6/62.9 tok/s) - Replaced router (port 9000) references with LiteLLM + direct routing - Replaced Prometheus exporter rule with sidecar health probe - Updated VRAM thresholds to match operational data (300/300/200 MB/h) - Added response size limit (1MB) to prevent OOM crashes - Added Lessons L5 (response size crash) and L6 (stable aliases) - Removed deprecated Rules 11-12 (router-specific distribution balance)	2026-07-18 08:07:03 +00:00
root	31ec70ae36	gpu-fleet: RTX 3090 swap to ThinkingCap-Qwen3.6-27B Q4_K_M ... - Model: Qwopus Q4_K_M (16GB, 63 tok/s) → ThinkingCap Q4_K_M (15.7GB, 68 tok/s) - RL-finetuned: 50% fewer thinking tokens, 0.85 MMLU-Pro (vs 0.83 base) - Self-spec MTP (n=4) REQUIRED for stability — segfaults without it - Added vision via mmproj (0.9GB) — new capability for this GPU - VRAM: 20.9/24.6GB (85%), tighter but stable - Outputs reasoning_content (hidden from Hermes agent)	2026-07-17 15:50:50 +00:00
root	5eb6d3bfbd	gpu-fleet: RTX 5070 swap to HauhauCS Gemma4-12B QAT Uncensored Balanced ... - Model: IQ4_NL (6.3GB, 191 tok/s) → Q4_K_M QAT (6.9GB, 87 tok/s) - MTP draft: Q8_0 (444MB) → tuned draft (242MB), saves 200MB VRAM - mmproj: F16 → BF16 (same size, matched to new model) - Benefits: 0/465 refusals, agent-optimized tuning, QAT quality - Trade: 54% slower generation (acceptable for gpu-light role) - Config: --parallel 1, --ctx-size 131072, single-slot full 128K - VRAM: 10.0/12.2GB (82%), healthy headroom	2026-07-17 15:24:13 +00:00
jerome	33cb88d571	Merge pull request 'feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo' (#20) from feat/gpu-128k-genesis-hermes-v3-20260717 into master ... Reviewed-on: #20	2026-07-17 11:21:32 +00:00
root	4a1f476623	fix: add missing description to inference-optimization frontmatter	2026-07-17 11:11:27 +00:00
root	ba2c55c7e6	ci: re-trigger pipeline for PR #20	2026-07-17 11:10:29 +00:00
root	b9149bce47	feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo ... GPU Changes: - All 3 GPUs reduced from 256K (-c 262144) to 128K (-c 131072) for stability - Observed instability near 100K at 256K — 128K is the stable ceiling - VRAM improved: RTX 3090 ~70% (was 90%), RTX 5070 ~65% (was 88%) - Strix Halo swapped to LuffyTheFox/Genesis Hermes V3 APEX - Hermes agent fine-tune, tensor repair (3 SSM layers, 76% W1 improvement) - Uncensored (0/465 refusals), multimodal (mmproj F16) - Speed: 65 tok/s gen, 140 tok/s prompt - Alias strix-moe maintained Agent Updates: - Mumuni: max_context_window 262144→131072, already aligned on strix-moe/0.65 - Tanko: max_context_window 262144→131072 - Koonimo: max_context_window + context_length 262144→131072 - CT114 SSH access confirmed (was 'Zulip only') LiteLLM (CT116): - Updated backend model references qwen3.6-35B-udq4→strix-moe - Removed stale ornith-1.0-35b from model_cost - Fallback chains updated Contracts Updated: - gpu-fleet.prose.md: topology, VRAM, benchmarks, config lines, model assignments - gpu-self-heal.prose.md: Rule 9/10 context targets - hermes-config-template.prose.md: template values, Rules 7-9, compression thresholds - inference-optimization.prose.md: added to repo, 128K recommendation Compression: 0.65 fires at ~85K (~43K headroom before 128K ceiling) For >128K workloads: route to external providers (deepseek)	2026-07-17 10:59:45 +00:00
root	65c99dab50	contract: update litellm-api-keys to v2026-07-17 — fleet standardization ... - Document canonical systemd drop-in pattern (ExecStart= reset + wrapper) - Hardcode venv paths — never use variables in single-quoted bash -c - Update migration status: all 4 agents on while-true wrapper + st.8e848433 - Infisical CLI update procedure (0.38.0 → 0.43.109 via artifacts-cli) - Service token inventory + .env fallback inventory - Key rotation log: fleet standardize + tanko fix-zulip entries - Remove git merge conflict artifacts - Add fleet-wide standardization lessons section	2026-07-17 10:17:23 +00:00
jerome	20cbb96e2d	Merge pull request 'Vault cleanup + contract sync (WAL #1316)' (#19) from fix/vault-cleanup-contract-sync into master ... Reviewed-on: #19	2026-07-16 21:24:59 +00:00
root	8215b84f88	merge: resolve conflicts with master (delegation delete + gpu-fleet)	2026-07-16 21:22:34 +00:00
jerome	24cd7f72ac	Merge branch 'master' into feat/zulip-v3-resilience	2026-07-16 21:20:24 +00:00
jerome	d6e85e68e9	Merge pull request 'UPDATE 2026-07-15: Full GPU fleet rebuild + stable aliases' (#18) from feat/gpu-fleet-rebuild-20260715 into master ... Reviewed-on: #18	2026-07-16 21:17:12 +00:00
root	b7e23e2592	vault: Infisical cleanup + contract sync (WAL #1316) ... - litellm-api-keys: vault audit, tanko/koby/koonimo migration, key rotation log - gpu-fleet: ornith-1.0-35b→strix-moe (remaining refs) - hermes-agent-baseline: 256K all GPUs, CT IPs updated, Shumba retired - delegation-prose-contract: removed (renamed to mumuni-delegation) - contract-registry.yaml + cron-prompts-review.md: from feat/contract-registry	2026-07-16 21:17:09 +00:00
root	369c312ccb	vault: BAGGY_LITELLM_API_KEY deleted — one secret per agent	2026-07-16 19:54:51 +00:00
root	7f39a444d6	vault: BAGGY+LITELLM synced, deletion needs Infisical UI	2026-07-16 19:28:28 +00:00
Abiba	17f4c79fe4	vault: sync BAGGY_LITELLM_API_KEY = KOONIMO_LITELLM_API_KEY (same agent CT113) ... Baggy = Koonimo (CT 113). The vault had two divergent keys: - BAGGY_LITELLM_API_KEY = sk-QT-kDt8Szo... (stale, 401) - KOONIMO_LITELLM_API_KEY = sk-OEK7z26n6... (valid, 200, session-13 rotation) Synced BAGGY to match KOONIMO (the valid key). Added contract note: both secrets MUST mirror each other. This was the root cause of Koonimo's 401 after migration — the ghost process had BAGGY in its env instead of KOONIMO.	2026-07-16 19:22:48 +00:00
Abiba	a9caf5b216	vault: fix koby platform connectivity + document migration lessons ... Koby was broken after migration — Zulip credentials lost (only in process memory), Telegram token overwritten (only existed in old .env.bak-20260603). Fix: Koby shares Tanko's Zulip bot (tanko-bot@, TANKO_ZULIP_API_KEY in vault). Telegram token recovered from .env.bak-20260603. Both platforms now connected. Added Koby migration lessons to contract (back up .env before migration, inject ALL platform env vars, cat /proc/pid/environ before killing old gateway). Added KOBY_ZULIP_API_KEY to vault (shares Tanko's key).	2026-07-16 18:19:27 +00:00
Abiba	15a826457e	vault: canonical Production Vault Access Process + sync vault + migrate koby/koonimo ... - Vault SYNCED: mumuni/koby/koonimo rotated keys written to Infisical (all validate 200). Vault is source of truth again (was stale since session-13 rotations). - Koby (.129) + Koonimo (.114) migrated from hardcoded systemd drop-ins to infisical-gateway.sh wrapper (live vault injection, .env fallback safety net). 4/5 agents now vault-backed (abiba/mumuni/koby/koonimo). - New § Production Vault Access Process: canonical 7-step pattern, migration status table, key rotation procedure, why-it's-non-fail rationale. - Fixed stale notes: 'vault sync PENDING' (now synced), 'Abiba key IS master key' (now a proper agent key sk-sxbphLvk1OU). - Tanko (user jerome, not systemd) documented as pending migration. - hermes-key-enforcement: cross-ref to canonical process, drop-ins deprecated.	2026-07-16 17:51:53 +00:00
Abiba	cd52deda92	contracts: stable alias chain + no-master-key-for-inference rule ... Agent configs aligned to stable aliases (gpu-light/gpu-dense/strix-moe/syslog-auto): - Tanko fixed: base_url :4000->/v1 (nginx), max_context_window 131072->262144, aux.compression gemma->strix-moe, aux gemma->gpu-light - Mumuni aligned: aux gemma->gpu-light, delegation/x_search qwen->gpu-dense - Koby/Koonimo already used gpu-light (validated) - Zero raw model names remain in any agent config No-master-key rule: litellm_proxy_master_key (sk-litellm-...) is admin-only (/key/list, /key/generate). All inference uses agent keys. Health-check model tests use dedicated monitor key (/etc/litellm-monitor.env on CT 116). Fixed: daily-infra-report.py, gpu-self-heal.py, litellm-health-check.sh. Fixed wrong key-scope note: qwen3.6-35B-udq4 IS in scope for baggy/koby/mumuni/ abiba-pi keys (not 'NOT in allowlist' as previously documented).	2026-07-16 17:39:05 +00:00
Abiba	97f1cd77e4	litellm-self-heal: document 2026-07-16 ops sync (script fixes, systemd gpu-monitor, live-key agent check) ... Health-check script: critical-only gpu-fleet alerting, strix-moe model test. gpu-monitor: systemd unit (was bare &), Strix counted in gpu_count, VRAM thresholds raised (93/97 — 256K ctx steady-state ~96% on RTX 3090). agent-health-check.py: reads live LITELLM_API_KEY from gateway env (no hardcoded keys). daily-infra-report.py: stale SYNTHETIC key → env-based master key.	2026-07-16 17:23:37 +00:00
Abiba	dc572889f8	contracts: sync to ground truth — ornith-1.0-35b→strix-moe, 256K all GPUs, real LiteLLM timeouts ... Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts: - AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist - All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1) - LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s) - Added LiteLLM model surface + key scoping to litellm-self-heal - Patched health-check script path ref Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal, zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline, delegation-prose-contract, mumuni-delegation-prose-contract	2026-07-16 17:02:44 +00:00
Abiba	d0feb7881e	Contracts: Rule 13 two key-injection patterns + koby/baggy rotation log (WAL #1300)	2026-07-16 15:10:50 +00:00