UPDATE 2026-07-15: Full GPU fleet rebuild + stable aliases #18

Merged

jerome merged 13 commits from feat/gpu-fleet-rebuild-20260715 into master 2026-07-16 21:17:12 +00:00

Conversation 0 Commits 13 Files Changed 13

+1136 -107

abiba-bot commented 2026-07-15 16:04:55 +00:00

Owner

Copy Link

Copy Source

Changes

Stable Role-Based Aliases (Critical Architecture)

• New: strix-moe, gpu-dense, gpu-light — never change when models swap
• Agent configs now use aliases, not model-specific names
• Old names preserved for backward compatibility

Model Changes

• Strix Halo: ornith-1.0-35b → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 256K)
• RTX 5070: Q4_K_M → IQ4_NL + MTP draft (256K, 125-213 tok/s, 2x faster)
• RTX 3090: Bumped to 256K context (was 131K — corrected in docs)
• All 3 GPUs now at 256K context

Provider Rename

• harness → litellm across all agent configs
• External providers (DeepSeek, OpenRouter) untouched

LiteLLM Config

• syslog-auto weighted pool restored with direct GPU routing (55/30/15)
• All models use api_key: not-needed
• Timeouts: RTX 3090=90s, Strix=300s, RTX 5070=300s

Mumuni Agent Profile

• Full profile documented in contract

Self-Heal

• VRAM leak thresholds: 50→300 MB/h (accommodates 256K context)
• RTX 5070 config corrected (context, model desc, target tps)

## Changes ### Stable Role-Based Aliases (Critical Architecture) - New: `strix-moe`, `gpu-dense`, `gpu-light` — never change when models swap - Agent configs now use aliases, not model-specific names - Old names preserved for backward compatibility ### Model Changes - **Strix Halo**: ornith-1.0-35b → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 256K) - **RTX 5070**: Q4_K_M → IQ4_NL + MTP draft (256K, 125-213 tok/s, 2x faster) - **RTX 3090**: Bumped to 256K context (was 131K — corrected in docs) - All 3 GPUs now at 256K context ### Provider Rename - `harness` → `litellm` across all agent configs - External providers (DeepSeek, OpenRouter) untouched ### LiteLLM Config - syslog-auto weighted pool restored with direct GPU routing (55/30/15) - All models use `api_key: not-needed` - Timeouts: RTX 3090=90s, Strix=300s, RTX 5070=300s ### Mumuni Agent Profile - Full profile documented in contract ### Self-Heal - VRAM leak thresholds: 50→300 MB/h (accommodates 256K context) - RTX 5070 config corrected (context, model desc, target tps)

abiba-bot added 2 commits 2026-07-15 16:04:56 +00:00

feat: Zulip v3 resilience contract + restore playbook ... 30bf42b841

- zulip-resilience-v3.prose.md: Production resilience rewrite contract covering
  circuit breaker, retry with jitter, queue lifecycle management, supervisor
  watchdog, and PM2 hardening. Research-backed from Zulip event system docs.
- abiba-zulip-restore.prose.md: Quick-restore playbook for recovery scenarios.

UPDATE 2026-07-15: Full fleet rebuild ... 622cf7b176

- Stable role-based aliases: strix-moe, gpu-dense, gpu-light
- Strix Halo: ornith-1.0-35b -> unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 256K)
- RTX 5070: Q4_K_M -> IQ4_NL+MTP (256K, 125-213 tok/s, 2x faster)
- RTX 3090: bumped to 256K context
- Provider rename: harness -> litellm (all agents)
- syslog-auto: weighted pool restored with direct GPU routing
- Mumuni agent profile documented
- Self-heal thresholds updated for 256K context
- All model contexts corrected (RTX 3090 was 131K, not 256K)

abiba-bot added 5 commits 2026-07-16 17:23:58 +00:00

Contracts: Mumuni context-fix (WAL #1300) — strix-moe, 256K all GPUs, Rule 12/13, machine-identity vault procedure 39e2fa0cfa

Contracts: replace remaining ornith-1.0-35b refs with strix-moe 9fd8c68bd2

Contracts: Rule 13 two key-injection patterns + koby/baggy rotation log (WAL #1300) d0feb7881e

contracts: sync to ground truth — ornith-1.0-35b→strix-moe, 256K all GPUs, real LiteLLM timeouts ... dc572889f8

Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts:
- AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist
- All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1)
- LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s)
- Added LiteLLM model surface + key scoping to litellm-self-heal
- Patched health-check script path ref

Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal,
zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline,
delegation-prose-contract, mumuni-delegation-prose-contract

litellm-self-heal: document 2026-07-16 ops sync (script fixes, systemd gpu-monitor, live-key agent check) ... 97f1cd77e4

Health-check script: critical-only gpu-fleet alerting, strix-moe model test.
gpu-monitor: systemd unit (was bare &), Strix counted in gpu_count, VRAM
thresholds raised (93/97 — 256K ctx steady-state ~96% on RTX 3090).
agent-health-check.py: reads live LITELLM_API_KEY from gateway env (no hardcoded keys).
daily-infra-report.py: stale SYNTHETIC key → env-based master key.

abiba-bot added 1 commit 2026-07-16 17:39:07 +00:00

contracts: stable alias chain + no-master-key-for-inference rule ... cd52deda92

Agent configs aligned to stable aliases (gpu-light/gpu-dense/strix-moe/syslog-auto):
- Tanko fixed: base_url :4000->/v1 (nginx), max_context_window 131072->262144,
  aux.compression gemma->strix-moe, aux gemma->gpu-light
- Mumuni aligned: aux gemma->gpu-light, delegation/x_search qwen->gpu-dense
- Koby/Koonimo already used gpu-light (validated)
- Zero raw model names remain in any agent config

No-master-key rule: litellm_proxy_master_key (sk-litellm-...) is admin-only
(/key/list, /key/generate). All inference uses agent keys. Health-check model
tests use dedicated monitor key (/etc/litellm-monitor.env on CT 116).
Fixed: daily-infra-report.py, gpu-self-heal.py, litellm-health-check.sh.

Fixed wrong key-scope note: qwen3.6-35B-udq4 IS in scope for baggy/koby/mumuni/
abiba-pi keys (not 'NOT in allowlist' as previously documented).

abiba-bot added 1 commit 2026-07-16 17:51:56 +00:00

vault: canonical Production Vault Access Process + sync vault + migrate koby/koonimo ... 15a826457e

- Vault SYNCED: mumuni/koby/koonimo rotated keys written to Infisical (all validate 200).
  Vault is source of truth again (was stale since session-13 rotations).
- Koby (.129) + Koonimo (.114) migrated from hardcoded systemd drop-ins to
  infisical-gateway.sh wrapper (live vault injection, .env fallback safety net).
  4/5 agents now vault-backed (abiba/mumuni/koby/koonimo).
- New § Production Vault Access Process: canonical 7-step pattern, migration status
  table, key rotation procedure, why-it's-non-fail rationale.
- Fixed stale notes: 'vault sync PENDING' (now synced), 'Abiba key IS master key'
  (now a proper agent key sk-sxbphLvk1OU).
- Tanko (user jerome, not systemd) documented as pending migration.
- hermes-key-enforcement: cross-ref to canonical process, drop-ins deprecated.

abiba-bot added 1 commit 2026-07-16 18:19:30 +00:00

vault: fix koby platform connectivity + document migration lessons ... a9caf5b216

Koby was broken after migration — Zulip credentials lost (only in process memory),
Telegram token overwritten (only existed in old .env.bak-20260603).

Fix: Koby shares Tanko's Zulip bot (tanko-bot@, TANKO_ZULIP_API_KEY in vault).
Telegram token recovered from .env.bak-20260603. Both platforms now connected.

Added Koby migration lessons to contract (back up .env before migration,
inject ALL platform env vars, cat /proc/pid/environ before killing old gateway).
Added KOBY_ZULIP_API_KEY to vault (shares Tanko's key).

abiba-bot added 1 commit 2026-07-16 19:22:51 +00:00

vault: sync BAGGY_LITELLM_API_KEY = KOONIMO_LITELLM_API_KEY (same agent CT113) ... 17f4c79fe4

Baggy = Koonimo (CT 113). The vault had two divergent keys:
- BAGGY_LITELLM_API_KEY = sk-QT-kDt8Szo... (stale, 401)
- KOONIMO_LITELLM_API_KEY = sk-OEK7z26n6... (valid, 200, session-13 rotation)

Synced BAGGY to match KOONIMO (the valid key).
Added contract note: both secrets MUST mirror each other.
This was the root cause of Koonimo's 401 after migration —
the ghost process had BAGGY in its env instead of KOONIMO.

abiba-bot added 1 commit 2026-07-16 19:28:31 +00:00

vault: BAGGY+LITELLM synced, deletion needs Infisical UI 7f39a444d6

abiba-bot added 1 commit 2026-07-16 19:54:55 +00:00

vault: BAGGY_LITELLM_API_KEY deleted — one secret per agent 369c312ccb

jerome merged commit d6e85e68e9 into master 2026-07-16 21:17:12 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

abiba-bot jerome (Jerome Kwame Tabiri) kagentz-bot kwame (Kwame Tabiri) mumuni-bot (Mumuni 🦅 (Syslog Falcon)) tanko-bot theo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#18