Compare commits

...
Author SHA1 Message Date
root fb52aff6a0 fix: update Mumuni CT reference to 100 (Hermes gateway on CT 100) 2026-07-28 15:49:51 +00:00
root 3b6cf44a30 fix: update all Mumuni IP references from .123 to .24 (inside Abiba CT100)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Mumuni CT114 destroyed. Mumuni now runs inside Abiba CT100 at 192.168.68.24. Updated all contract files and agent-health-check.py.
2026-07-28 08:56:45 +00:00
root 22b015e182 fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
2026-07-27 20:25:51 +00:00
mumuni-bot 29e32340a4 Merge pull request 'fix: swap gpu-dense to SmartCode-Fable-5 (ThinkingCap + Fable 5 CoT)' (#33) from fix/gpu-dense-smartcode-fable5 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-07-27 16:43:46 +00:00
root 179529de71 fix: swap gpu-dense to SmartCode-Fable-5 (ThinkingCap + Fable 5 CoT)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
gpu-dense on RTX 3090 swapped from qwen3.6-27B-code to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled (UD-Q4_K_XL, ~17.9GB).\n\nImprovements:\n- ~50% fewer thinking tokens via ThinkingCap finetune\n- Fable 5 CoT distillation for improved coding reasoning\n- Same 27B base, fits RTX 3090 at ~73% VRAM\n- Recommended samplers: temp 0.9, top-p 0.95, top-k 60\n- Context: 128K (fleet standard)
2026-07-27 16:43:29 +00:00
mumuni-bot fb45007ace Merge pull request 'fix: remove mumuni from health check — now inside Abiba CT 100' (#32) from fix/update-health-check-remove-mumuni into master 2026-07-26 12:37:35 +00:00
root 0f572ff9f2 fix: remove mumuni from health check — now inside Abiba CT 100 2026-07-26 12:37:07 +00:00
mumuni-bot 3a8e7d9b3a Merge pull request 'fix: remove CT 114 (mumuni) — destroyed, moved inside CT 100' (#31) from fix/remove-mumuni-ct114 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-07-26 12:36:31 +00:00
root 0fb54926f6 fix: remove CT 114 (mumuni) — destroyed, moved inside CT 100
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-07-26 12:36:02 +00:00
mumuni-bot 146abf7f82 Merge pull request 'fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness' (#30) from fix/agent-health-check-v2 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-07-26 12:27:20 +00:00
root c869e75c61 ci: trigger recheck
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Passed
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Passed
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Passed
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Passed
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Passed
test-check Manual override
2026-07-26 12:24:05 +00:00
root 47aa92bee3 fix: verification protocol findings — CT 118 static IP, .110 llama-server restored
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Verification protocol run 2026-07-26:
- CT 118 (jdownloader): DHCP had moved it to .131 post-reboot. Now set to static .20.
- RTX 5070 (.110) llama-server: was stopped (disabled unit). Started and verified — gemma-4-12b responding through LiteLLM.
- All 6 PVE nodes confirmed at correct IPs.
- All 18 CTs confirmed at correct IPs on correct nodes.
- All public endpoints responding (meet, chat, git, litellm, vault, auth).
- Container counts verified: docker-vm 14 ctrs, CT 116 10 ctrs, VPS 5 ctrs.
2026-07-26 12:20:58 +00:00
root de9adb13cf fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Gaps fixed:
- Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
- Agent key lookup uses correct {NAME}_LITELLM_API_KEY format
- New CT liveness check via pct status on PVE nodes
- New config YAML integrity check via yaml.safe_load()
- New wrapper/CLI integrity check (hermes wrapper, infisical path, hermes-real)
- New vault secret non-emptiness check
- Ops escalation: failures produce ALERT lines for cron capture
2026-07-26 12:06:32 +00:00
root c380196fab fix: address review findings
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- Remove kagentz from amdpve node table (belongs on hwepve)
- Fix kagentz pct-run table: hwepve (was amdpve)
- Fix mumuni pct-run table: hwepve (was minipve)
- Fix Zulip recovery command: wrap in single SSH call
2026-07-24 20:11:32 +00:00
root b17c60f997 fix: contract accuracy updates post fleet-wide reboot
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
infrastructure-control.prose.md:
- Add hwepve as 6th Proxmox node
- Fix Gitea IP: .17 (was .110)
- Fix AdGuard IP: .10 on minipve (was .102 on acerpve)
- Fix Abiba placement: hwepve (was amdpve)
- Fix Mumuni placement: hwepve (was minipve)
- Fix Authentik port: add :9000
- Add CT 118 (jdownloader), CT 119 (infisical-vault)
- Add last-verified date (2026-07-24)

infrastructure-update.prose.md:
- Add post-reboot CT sweep procedure
- Add Zulip Docker network recovery steps
- Add WireGuard tunnel verification

scripts/netbird-add-domain.sh:
- New script to register domains in Netbird proxy store.db
2026-07-24 20:01:43 +00:00
abiba-bot 2f0c3c1850 fix: update contracts for hwepve 6th node + Mumuni recovery 2026-07-23 18:47:17 +00:00
jerome cedbdc465d Merge branch 'master' into fm/mumuni-recovery-hwepve 2026-07-23 18:11:06 +00:00
abiba-bot 8b09c78efd Merge: PR #26 conflict resolution — hwepve 6th node + qwen model ref
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-07-23 18:10:36 +00:00
root 767ab25128 merge: resolve PR #26 conflict — keep qwen model ref + add hwepve node
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-07-23 18:10:18 +00:00
abiba-bot 75381f9737 Captain decision batch 2026-07-21: ornith decommission, 256K→128K context, Mumuni Discord disable
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
Three contract updates per captain decision batch. Validated through no-mistakes pipeline (green).
2026-07-23 09:13:06 +00:00
root 9f0e04f22c fix: update contracts for hwepve 6th node + Mumuni migration (lxc/114 on hwepve)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
- proxmox-monitor: 5→6 nodes, add hwepve (.4, Huawei Matebook 16), note Mumuni (lxc/114) placement
- zulip-health: add dual-gateway collision detection to B2, note Mumuni on hwepve
- hermes-agent-baseline: fix Mumuni node minipve→hwepve
- hermes-zulip-restore: add hwepve Proxmox column for Mumuni
- mumuni-delegation: fix CT 118/storepve/.6 → lxc/114/hwepve/.123, 5→6 nodes
- pm2-self-heal: verified — no Mumuni references, no changes needed

Post-migration discoveries:
- Mumuni container migrated from minipve to hwepve (6th node, 192.168.68.4)
- IP preserved (192.168.68.123), hostname 'mumuni' unchanged
- Dual gateway collision (PID 1442 --replace + PID 2924 --force) resolved by
  container restart at 10:07; gateway restarted clean at 13:06 with fresh Zulip
  queue (errors=0, reconnects=0)
2026-07-20 17:08:58 +00:00
16 changed files with 591 additions and 130 deletions
+15 -12
View File
@@ -14,6 +14,10 @@ description: >
Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling. Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling.
For larger context needs → fall back to external providers (deepseek). For larger context needs → fall back to external providers (deepseek).
VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%. VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
UPDATED 2026-07-27: gpu-dense swapped to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled
(UD-Q3_K_XL, ~14.7GB — Q4 was too large for 24GB VRAM with 128K KV cache). ~50% fewer thinking
tokens via ThinkingCap finetune + Fable 5 CoT distillation for improved coding reasoning.
VRAM ~22.4/24.6GB (91%).
agent: abiba agent: abiba
triggers: triggers:
- on model add/remove - on model add/remove
@@ -86,7 +90,7 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen
| Alias | GPU | Current Model | Will Route To | | Alias | GPU | Current Model | Will Route To |
|-------|-----|---------------|---------------| |-------|-----|---------------|---------------|
| `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo | | `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 | | `gpu-dense` | RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q4_K_XL | Whatever runs on RTX 3090 |
| `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 | | `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work **Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
@@ -96,7 +100,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status | | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
|-------|-----|------|------|-----|----------|----------|-------------|--------| |-------|-----|------|------|-----|----------|----------|-------------|--------|
| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s | | SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 | .8 (llm-gpu) | ~22.4/24.6GB (91%) | **128K** | q4_0 | 1 | 2048/1024 | ✅ healthy |
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy | | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s | | qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |
@@ -106,7 +110,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
| Model | GPU | Weight | RPM Cap | Timeout | | Model | GPU | Weight | RPM Cap | Timeout |
|-------|-----|--------|---------|---------| |-------|-----|--------|---------|---------|
| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** | | SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** | | qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** | | gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
@@ -117,7 +121,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
| Model | RPM Cap | Notes | | Model | RPM Cap | Notes |
|-------|---------|-------| |-------|---------|-------|
| strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload | | strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload |
| qwen3.6-27B-code | 500 | High cap — primary workhorse | | SmartCode-Fable-5-27B-UD-Q3_K_XL | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) |
| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s | | gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
### Stable Aliases (for agent configs — never change) ### Stable Aliases (for agent configs — never change)
@@ -204,7 +208,7 @@ Plaintext keys removed from this contract post-vault-migration.
| Agent | CT | IP | LiteLLM Alias | Key Source | Access | | Agent | CT | IP | LiteLLM Alias | Key Source | Access |
|-------|-----|-----|---------------|------------|--------| |-------|-----|-----|---------------|------------|--------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome | | Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root | | Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | SSH root |
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) | | Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM | | Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH | | Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
@@ -247,13 +251,12 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster(). Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround. - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB. - **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2). - **RTX 3090 (2026-07-27)**: Swapped to SmartCode-Fable-5-27B-UD-Q3_K_XL (14.7GB). Q4 was too large for 24GB VRAM with 128K context + KV cache overhead. Q3 fits at ~22.4GB (91%). Uses standard llama.cpp build b9190 (turboquant b9150 incompatible with qwen3_5 arch). Config: `-c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on --cont-batching`. Sampler: `--temp 0.9 --top-p 0.95 --top-k 60 --min-p 0.0 --repeat-penalty 1.0`. Service: `/home/llmuser/llama-fable-wrapper.sh`.
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`. - **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. - **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (old Mumuni CT114 — now inside Abiba CT100 at .24) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116. - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback. - **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected. - **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
@@ -265,7 +268,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context | | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
|-----|-------|-----------|--------------|----------|---------| |-----|-------|-----------|--------------|----------|---------|
| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** | | RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q3_K_XL | **TBD** | — | — | **128K** |
| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** | | RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
| Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** | | Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** |
@@ -298,7 +301,7 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
### Mumuni Agent Profile ### Mumuni Agent Profile
Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs: Mumuni (CT100/abiba, 192.168.68.24) is the primary business assistant. This profile is the reference for all agent configs:
| Setting | Value | Notes | | Setting | Value | Notes |
|---------|-------|-------| |---------|-------|-------|
@@ -323,7 +326,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
| Agent | Host | Status | | Agent | Host | Status |
|-------|------|--------| |-------|------|--------|
| **Mumuni** | CT114 (.123) | ✅ Updated to stable aliases | | **Mumuni** | CT100 (.24) | ✅ Updated to stable aliases |
| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases | | **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM | | **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM | | **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
+2 -2
View File
@@ -25,7 +25,7 @@ done
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform | | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|---------------|------------|----------| |-------|-----|------|-----|---------------|------------|----------|
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes | | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes | | Mumuni | 100 (abiba) | hwepve | .24 | `mumuni` | Infisical vault | Hermes (Pi gateway) |
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** | | Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes | | Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
@@ -253,7 +253,7 @@ EnvironmentFile=/etc/environment # sources LITELLM_API_KEY
Run the consolidated health check: Run the consolidated health check:
```bash ```bash
python3 /root/scripts/agent-health-check.py python3 /root/scripts/agent-health-check.py # v2: now checks all 5 agents including Koby/Koonimo SSH, CT liveness, config YAML, wrapper integrity, vault non-emptiness
``` ```
This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes), This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
verifies gateway liveness, confirms Zulip streaming (`edit_message` present), verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
+1 -1
View File
@@ -35,7 +35,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Agent | Key Alias | Host | SSH | Sub-Agents | | Agent | Key Alias | Host | SSH | Sub-Agents |
|-------|-----------|------|-----|-----------| |-------|-----------|------|-----|-----------|
| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — | | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
| Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ | | Mumuni | `mumuni` | 192.168.68.24 (CT100 abiba) | root@.24 | 6 profiles ✱ |
| Abiba | `abiba-pi` | 192.168.68.24 | local | — | | Abiba | `abiba-pi` | 192.168.68.24 | local | — |
| Koby | `koby` | CT 111 (tdunna) | Zulip | — | | Koby | `koby` | CT 111 (tdunna) | Zulip | — |
| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — | | Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
+1 -1
View File
@@ -189,7 +189,7 @@ litellm_settings:
| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified | | Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
|-------|-----|-----|---------------|------------|--------|-----------------|---------------| |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 | | Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 | | Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 | | Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 | | Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 | | Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
+1 -1
View File
@@ -51,7 +51,7 @@ gateway restart, and connection validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User | | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------| |------|-----|---------|-------------|-------------|------|
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root | | Mumuni | CT100 (abiba) | hwepve | 192.168.68.24 | /root/.hermes | root |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
+61 -37
View File
@@ -3,7 +3,7 @@ kind: pattern
name: infrastructure-control name: infrastructure-control
description: > description: >
Full infrastructure monitoring and control pattern covering the Full infrastructure monitoring and control pattern covering the
5-node Proxmox cluster, 3 Docker ecosystems (22 containers), 6-node Proxmox cluster, 3 Docker ecosystems (22 containers),
NFS storage, and network services. Defines monitors, remediations, NFS storage, and network services. Defines monitors, remediations,
and the access matrix for all environments. and the access matrix for all environments.
@@ -12,6 +12,11 @@ description: >
never mutate infrastructure based on them without first confirming never mutate infrastructure based on them without first confirming
against the live system. Policy fields are authoritative. See the against the live system. Policy fields are authoritative. See the
`verify-before-mutate` skill. `verify-before-mutate` skill.
**Last verified:** 2026-07-27 — gpu-dense swapped to SmartCode-Fable-5
(Qwen3.6-27B distilled, ~50% fewer thinking tokens, improved coding reasoning).
Model pricing reduced 100x across all models ($0.15/$0.60 per 1M tokens).
All LiteLLM models set to 128K max_model_tokens. See data/learnings.md.
--- ---
# Infrastructure Control Pattern # Infrastructure Control Pattern
@@ -29,7 +34,7 @@ description: >
┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ Abiba │ │ Tanko │ │ Mumuni │ │ Abiba │ │ Tanko │ │ Mumuni │
│ (pi) │ │ (Hermes) │ │ (Hermes) │ │ (pi) │ │ (Hermes) │ │ (Hermes) │
│ CT 100 │ │ CT 112 │ │ CT 114 │ CT 100 │ │ CT 112 │ │ CT 100
└──────┬──────┘ └──────┬───────┘ └──────┬───────┘ └──────┬──────┘ └──────┬───────┘ └──────┬───────┘
│ │ │ │ │ │
└──────────────────┼────────────────────┘ └──────────────────┼────────────────────┘
@@ -42,8 +47,8 @@ description: >
│ │ │ │ │ │ │ │ │ │
┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
minipve amdpve storepve acerpve ocupve minipve amdpve storepve acerpve ocupve hwepve
(.12) (.15) (.6) (.9) (.5) (.12) (.15) (.6) (.9) (.5) (.4)
┌─────────────────────────────────────────────┐ ┌─────────────────────────────────────────────┐
@@ -67,7 +72,7 @@ description: >
| docker-vm (.7) | SSH root | SSH key | ✅ | | docker-vm (.7) | SSH root | SSH key | ✅ |
| CT 116 (syslog-api) | SSH root | SSH key | ✅ | | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ | | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ | | Mumuni (inside CT100 abiba .24) | SSH root | id_ed25519 | ✅ |
| Baggy CT (113) | SSH jerome | ❌ no key access | | Baggy CT (113) | SSH jerome | ❌ no key access |
| Netbird (.17) | SSH root | SSH key | ✅ | | Netbird (.17) | SSH root | SSH key | ✅ |
| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ | | Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
@@ -85,25 +90,31 @@ description: >
### Reachability Matrix ### Reachability Matrix
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) | | From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.24) | Baggy (.114) |
|-----------|---------|----------------|--------|-------------|---------------|----------------| |-----------|---------|----------------|--------|-------------|---------------|----------------|
| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH | | **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ | | **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
| **Mumuni** (.123) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | | **Mumuni** (.24) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
**Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba. **Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.
## Section 2: Proxmox Cluster — Monitoring ## Section 2: Proxmox Cluster — Monitoring
### Nodes (5) ### Nodes (6)
| Node | IP | CPU | RAM | VMs/CTs | Role | | Node | IP | CPU | RAM | VMs/CTs | Role |
|------|----|-----|-----|---------|------| |------|----|-----|-----|---------|------|
| minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging | | minipve | .12 | 16C | 30GB | authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute | | amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute |
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat | | storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat |
| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs | | acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs | | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
| hwepve | .4 | ? | ? | abiba, (mumuni CT 114 stopped) | Agents (new node) |
> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve and now runs Mumuni
> Zulip gateway internally. CT 114 (old mumuni container) destroyed 2026-07-26.
> Mumuni now runs inside Abiba CT100 (.24). Old CT114 destroyed 2026-07-26.
### Checks (every 5 min) ### Checks (every 5 min)
@@ -355,15 +366,17 @@ fine. Services that resolve directly to a LAN IP are NetBird-independent.
| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ | | Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ | | LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
| Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ | | Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
| Gitea | git.sysloggh.net:443 | 192.168.68.110 | CNAME → netbird | **Yes** | ⚠️ | | Gitea | git.sysloggh.net:443 | 192.168.68.17:3000 | CNAME → netbird | **Yes** | ⚠️ |
| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE | | Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ | | Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
| DNS UI | dns.sysloggh.net:443 | 192.168.68.102 | CNAME → netbird | **Yes** | ⚠️ | | DNS UI | dns.sysloggh.net:443 | 192.168.68.10:80 | CNAME → netbird | **Yes** | ⚠️ |
| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ | | SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ | | Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
**Verified 2026-07-02:** NetBird VPS rebooted after a hang; all CNAME'd **Verified 2026-07-24:** NetBird VPS rebooted after a hang; all CNAME'd
services recovered. LAN-IP-direct paths stayed up throughout the outage. services recovered. LAN-IP-direct paths stayed up throughout the outage.
Also added `dns.sysloggh.net` route (was missing entirely).
See `scripts/netbird-add-domain.sh` for adding new proxy routes.
### 5.2 Checks (every 2 min) ### 5.2 Checks (every 2 min)
@@ -511,8 +524,8 @@ enforced by the `routing-regression.config_url_violations` check in Section
| LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` | | LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
| LiteLLM (nginx) | `http://192.168.68.116` | — | | LiteLLM (nginx) | `http://192.168.68.116` | — |
| Grafana | `http://192.168.68.116:3001` | — | | Grafana | `http://192.168.68.116:3001` | — |
| Authentik | `https://192.168.68.11` | `https://auth.sysloggh.net` | | Authentik | `https://192.168.68.11:9000` | `https://auth.sysloggh.net` |
| Gitea | `http://192.168.68.110:3000` | `https://git.sysloggh.net` | | Gitea | `http://192.168.68.17:3000` | `https://git.sysloggh.net` |
| Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` | | Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` |
| SearXNG | `http://192.168.68.7:8888` | — | | SearXNG | `http://192.168.68.7:8888` | — |
| Firecrawl | `http://192.168.68.7:3002` | — | | Firecrawl | `http://192.168.68.7:3002` | — |
@@ -575,25 +588,27 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| CT | Name | Node | IP | Role | Agent | | CT | Name | Node | IP | Role | Agent |
|----|------|------|----|------|-------| |----|------|------|----|------|-------|
| 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi | | CT | Name | Node | IP | Role | Agent |
|----|------|------|----|------|-------|
| 100 | abiba | **hwepve** | .24 | Pi agent | ✅ pi |
| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ | | 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
| 102 | adguard | acerpve | | DNS | ❌ | | 102 | adguard | **minipve** | **.10** | DNS | ❌ |
| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ | | 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
| 104 | authentik | minipve | .11 | OIDC | ❌ | | 104 | authentik | minipve | .11 | OIDC | ❌ |
| 105 | kagentz | amdpve | — | Agent Zero | ✅ | | 105 | kagentz | **hwepve** | — | Agent Zero | ✅ |
| 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP | | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
| 107 | pbs | storepve | — | Backups | ❌ | | 107 | pbs | storepve | — | Backups | ❌ |
| 108 | media | storepve | — | Media | ❌ | | 108 | media | storepve | — | Media | ❌ |
| 109 | docker-vm | storepve | .7 | Docker host | ❌ | | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
| 110 | gitea | minipve | | Git | ❌ | | 110 | gitea | minipve | **.17** | Git | ❌ |
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ | | 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ | | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
| 113 | baggy | amdpve | ? | Hermes agent | ✅ | | 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
| 114 | mumuni | minipve | .123 | Hermes agent | | | 115 | scottdenya | amdpve | .75 | Denya OneCare | |
| 115 | scottdenya | amdpve | — | ? | ❌ |
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ | | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
| 117 | zulip | storepve | | Chat | ❌ | | 117 | zulip | storepve | .19 | Chat | ❌ |
| 118 | jitsi | minipve | — | Video | ❌ | | 118 | jdownloader | storepve | — | JDownloader container | ❌ |
| 119 | infisical-vault | minipve | — | Vault | ❌ |
## Appendix C: Docker Compose Files Location ## Appendix C: Docker Compose Files Location
@@ -613,47 +628,56 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
| CT | Name | Node | pct-run | | CT | Name | Node | pct-run |
|-----|------|------|---------| |-----|------|------|---------|
| 100 | abiba | amdpve | `pct-run 100` | | 100 | abiba | hwepve | `pct-run 100` |
| 105 | kagentz | amdpve | `pct-run 105` | | 105 | kagentz | **hwepve** | `pct-run 105` |
| 111 | tdunna | amdpve | `pct-run 111` | | 111 | tdunna | amdpve | `pct-run 111` |
| 112 | tanko | amdpve | `pct-run 112` | | 112 | tanko | amdpve | `pct-run 112` |
| 113 | baggy | amdpve | `pct-run 113` | | 113 | baggy | amdpve | `pct-run 113` |
| 115 | scottdenya | amdpve | `pct-run 115` | | 115 | scottdenya | amdpve | `pct-run 115` |
| 104 | authentik | minipve | `pct-run 104` | | 104 | authentik | minipve | `pct-run 104` |
| 110 | gitea | minipve | `pct-run 110` | | 110 | gitea | minipve | `pct-run 110` |
| 114 | mumuni | minipve | `pct-run 114` |
| 116 | syslog-api | minipve | `pct-run 116` | | 116 | syslog-api | minipve | `pct-run 116` |
| 106 | ra-h-os | storepve | `pct-run 106` | | 106 | ra-h-os | storepve | `pct-run 106` |
| 107 | proxmox-backup | storepve | `pct-run 107` | | 107 | proxmox-backup | storepve | `pct-run 107` |
| 108 | media | storepve | `pct-run 108` | | 108 | media | storepve | `pct-run 108` |
| 117 | zulip | storepve | `pct-run 117` | | 117 | zulip | storepve | `pct-run 117` |
| 102 | adguard | acerpve | `pct-run 102` | | 102 | adguard | **minipve** | `pct-run 102` |
GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly: GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
```bash ```bash
ssh root@192.168.68.8 # RTX 3090 ssh root@192.168.68.8 # RTX 3090
ssh root@192.168.68.110 # RTX 5070 ssh root@192.168.68.110 # RTX 5070
ssh root@192.168.68.15 # Strix Halo ssh root@192.168.68.15 # Strix Halo
ssh root@192.168.68.4 # hwepve (abiba, kagentz) — Mumuni runs inside CT 100
``` ```
## Section 7: Agent Health Check (consolidated — 2026-07-05) ## Section 7: Agent Health Check v2 (2026-07-26)
Replaces 7 scattered Zulip health scripts with a single non-disruptive check Single non-disruptive check running every 10 minutes via cron
running every 10 minutes via cron (`/root/scripts/agent-health-check.py`). (`/root/scripts/agent-health-check.py`). v2 fixes critical gaps:
- Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
- Agent-specific vault keys (`{NAME}_LITELLM_API_KEY`) not shared master key
- CT liveness check via `pct status` on PVE nodes
- Config YAML integrity check
- Wrapper/CLI integrity check
- Vault secret non-emptiness check
The script is **read-only** — it never restarts, kills, or modifies anything. The script is **read-only** — it never restarts, kills, or modifies anything.
Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh,
which was kill+nohup outside systemd) are banned by policy.
### Checks Performed ### Checks Performed
| Check | Frequency | What It Detects | | Check | Frequency | What It Detects |
|-------|-----------|-----------------| |-------|-----------|-----------------|
| LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models | | LiteLLM key validation | 10 min | All 5 agent-specific keys authenticate (not shared master key) |
| GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) | | GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
| Gateway liveness | 10 min | Gateway process running, state file readable | | Gateway liveness | 10 min | Gateway process running, state file readable (all 5 agents) |
| Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) | | Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
| Recent errors | 10 min | Error count in journald for last 10 min | | Recent errors | 10 min | Error count in journald for last 10 min |
| CT liveness | 10 min | `pct status` on PVE nodes — catches stopped CTs |
| Config YAML integrity | 10 min | Python `yaml.safe_load()` — catches syntax errors |
| Wrapper/CLI integrity | 10 min | hermes wrapper exists, infisical path correct, hermes-real reachable |
| Vault secrets | 10 min | Agent-specific vault keys are non-empty and start with `sk-` |
### Disabled Scripts ### Disabled Scripts
+37 -2
View File
@@ -91,17 +91,52 @@ Before ANY update wave:
- Firecrawl test: `curl :3002/` - Firecrawl test: `curl :3002/`
- SearXNG test: `curl :8888` - SearXNG test: `curl :8888`
## Wave 4: Proxmox Kernel Reboot (if needed) ## Wave 4: Proxmox Kernel Reboot
Only if `[ -f /var/run/reboot-required ]` on any node. Only if `[ -f /var/run/reboot-required ]` on any node.
| Target | Action | | Target | Action |
|--------|--------| |--------|--------|
| Affected PVE node | Verify all CTs/VMs migrated or stopped | | Affected PVE node | Verify all CTs/VMs migrated or stopped |
| | `reboot` via PVE API | | | `reboot` via PVE API (or `systemctl reboot -f` if dbus fails) |
| | Wait 120s for node to come back | | | Wait 120s for node to come back |
| | Start any stopped CTs | | | Start any stopped CTs |
### Post-reboot sweep (known gaps)
After every node reboot, run these checks:
1. **CT auto-start sweep** — LXC containers sometimes don't start despite
`onboot: 1`. Check every CT on the rebooted node and start any left stopped:
```bash
pct list | awk '/stopped/{print $1}' | xargs -I{} pct start {}
```
Known cases: scottdenya (CT 115 on amdpve), authentik (CT 104 on minipve).
2. **Zulip recovery** — When docker-vm or storepve reboots, the Zulip main
container loses its Docker network assignment (SIGKILL during storage
outage detaches it from `zulip_default` network). Run:
```bash
ssh root@192.168.68.19 'docker rm -f zulip-zulip-1 && cd /opt/zulip && docker compose up -d'
```
The compose restart recreates the container on the correct network.
3. **docker-vm Docker daemon** — After reboot, Docker can take 3-4 minutes
to become `active`. The docker-proxy for Pulse (port 7655) starts early,
so Pulse is accessible before `docker ps` reports ready. Wait for Docker
before checking other stacks.
### VPS ↔ docker-vm tunnel
After any VPS or docker-vm reboot, verify the dedicated WireGuard tunnel:
```bash
ssh root@72.61.0.17 'wg show wg1' | grep "latest handshake"
# If no handshake in >60s:
ssh root@72.61.0.17 'wg-quick up wg1'
```
The tunnel uses PersistentKeepalive=25 and is systemd-enabled, but should
be verified after a reboot.
## Rollback Protocol ## Rollback Protocol
If ANY verification fails: If ANY verification fails:
+1 -1
View File
@@ -178,7 +178,7 @@ through its agent wrapper.
| Agent | Host | Pattern | Keys | Status | | Agent | Host | Pattern | Keys | Status |
|-------|------|---------|------|--------| |-------|------|---------|------|--------|
| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed | | abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
| mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback | | mumuni | .24 (CT100 abiba) | Pi Hermes gateway (no systemd) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback | | tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed | | koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed | | koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
+1 -1
View File
@@ -102,7 +102,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`). - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault). - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): reads each agent's **live** `LITELLM_API_KEY` from its gateway process env via SSH — never hardcodes keys (hardcoded keys rot on rotation and caused 9×401/30min). Fleet roster: abiba, tanko, mumuni, koby, koonimo (legacy `tdunna`/`baggy` removed — never existed). - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM. - **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
## Maintains ## Maintains
+3 -3
View File
@@ -6,7 +6,7 @@ description: >
delegation, verification, and delivery. Defines when to delegate, which delegation, verification, and delivery. Defines when to delegate, which
worker to use for what, how to handle failures, and the kanban board worker to use for what, how to handle failures, and the kanban board
protocol. Enforces context-window discipline and separation of concerns. protocol. Enforces context-window discipline and separation of concerns.
Runs on Mumuni (CT 118, storepve, .6) via Hermes agent. Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Hermes Zulip gateway).
version: 1.0.0 version: 1.0.0
--- ---
@@ -19,8 +19,8 @@ version: 1.0.0
## Topology ## Topology
**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve) **Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent **Manager:** Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
This contract is infrastructure-agnostic in terms of which nodes are used. This contract is infrastructure-agnostic in terms of which nodes are used.
+7 -6
View File
@@ -5,7 +5,7 @@ description: >
Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx). the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
agent: abiba agent: abiba
@@ -33,8 +33,8 @@ agent: abiba
| Exporter | Host:Port | Scope | Notes | | Exporter | Host:Port | Scope | Notes |
|----------|-----------|-------|-------| |----------|-----------|-------|-------|
| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). | | prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. | | node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. |
| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. | | docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
## PVE API Token ## PVE API Token
@@ -48,7 +48,7 @@ agent: abiba
| UID | Title | Panels | Source | | UID | Title | Panels | Source |
|-----|-------|--------|--------| |-----|-------|--------|--------|
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries | | proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) | | proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) | | docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) | | gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
@@ -77,9 +77,9 @@ agent: abiba
| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator | | `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions | | `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning | | `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config | | `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config |
## Cluster "Tabiri" — 5 Nodes ## Cluster "Tabiri" — 6 Nodes
| Node | IP | Role | | Node | IP | Role |
|------|----|----| |------|----|----|
@@ -88,6 +88,7 @@ agent: abiba
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) | | acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
| minipve | 192.168.68.12 | PVE | | minipve | 192.168.68.12 | PVE |
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) | | amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) |
| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 |
## Operations ## Operations
+300 -58
View File
@@ -1,10 +1,10 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
""" """
/root/scripts/agent-health-check.py — Consolidated Agent Health Verification /root/scripts/agent-health-check.py — Consolidated Agent Health Verification v2
Single non-disruptive health check replacing 7 scattered scripts. Verifies: LiteLLM keys (agent-specific), GPU port conflicts, agent Zulip streaming,
Verifies: LiteLLM keys, GPU port conflicts, agent Zulip streaming, gateway liveness, gateway log health, CT liveness, config YAML integrity,
gateway liveness, and gateway log health. NEVER restarts anything. wrapper/CLI integrity, vault secret non-emptiness. NEVER restarts anything.
Usage: Usage:
python3 /root/scripts/agent-health-check.py # Full check python3 /root/scripts/agent-health-check.py # Full check
@@ -12,59 +12,39 @@ Usage:
python3 /root/scripts/agent-health-check.py --quiet # Only output on failure python3 /root/scripts/agent-health-check.py --quiet # Only output on failure
Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet
Changelog:
v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114),
abiba (.24).
""" """
import subprocess, json, sys, os, time import subprocess, json, sys, os, time
from datetime import datetime from datetime import datetime
LITELLM = "http://192.168.68.116:80" LITELLM = "http://192.168.68.116:80"
INFISICAL_PROJECT = "322fceab-39da-4854-a55a-568e76c0f13f"
INFISICAL_ENV = "prod"
def _get_agent_key(agent_name): # PVE node IPs for CT liveness checks
"""Retrieve agent key from Infisical vault.""" PVE_NODES = {
try: "hwepve": "192.168.68.4",
result = subprocess.run( "amdpve": "192.168.68.15",
["infisical", "secrets", "get", "LITELLM_API_KEY", "minipve": "192.168.68.12",
"--project=agents", "--env=production", "--plain"], "storepve": "192.168.68.6",
capture_output=True, text=True, timeout=10 "acerpve": "192.168.68.9",
) "ocupve": "192.168.68.5",
if result.returncode == 0:
return result.stdout.strip()
except Exception:
pass
# Fallback: try exporting all secrets
try:
result = subprocess.run(
["infisical", "export", "--project=agents", "--env=production",
"--format=dotenv"],
capture_output=True, text=True, timeout=10
)
if result.returncode == 0:
for line in result.stdout.splitlines():
if line.startswith(f"LITELLM_API_KEY_{agent_name.upper()}") or \
(line.startswith("LITELLM_API_KEY=") and agent_name == os.uname().nodename):
return line.split("=", 1)[1].strip().strip('"').strip("'")
except Exception:
pass
return None
# Agent keys are pulled from Infisical vault at runtime.
# The 'key' field is populated dynamically below.
AGENTS = {
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome"},
"mumuni": {"ct": 114, "host": "192.168.68.123", "user": "root"},
"koby": {"ct": 111, "host": None, "user": None},
"koonimo": {"ct": 113, "host": None, "user": None},
} }
# Inject keys from vault # Agent definitions: ct, host, user, pve_node, vault_key_name
for agent_name in AGENTS: AGENTS = {
key = _get_agent_key(agent_name) "tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve", "vault_key": "TANKO_LITELLM_API_KEY"},
if key: "abiba": {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "hwepve", "vault_key": None}, # Pi agent + Mumuni Zulip, no vault key
AGENTS[agent_name]["key"] = key "koby": {"ct": 111, "host": "192.168.68.129", "user": "root", "pve": "amdpve", "vault_key": "KOBY_LITELLM_API_KEY"},
else: "koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root", "pve": "amdpve", "vault_key": "KOONIMO_LITELLM_API_KEY"},
AGENTS[agent_name]["key"] = None }
GPU_HOSTS = { GPU_HOSTS = {
"gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-server"}, "gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-server"},
@@ -74,6 +54,11 @@ GPU_HOSTS = {
FAIL = [] FAIL = []
INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
# ── Helpers ──────────────────────────────────────────────────────────
def ssh(host, cmd, user="root"): def ssh(host, cmd, user="root"):
"""Execute a command on a remote host, return stdout or None.""" """Execute a command on a remote host, return stdout or None."""
try: try:
@@ -112,15 +97,81 @@ def http_json(url, headers=None, timeout=5):
except: except:
return None return None
def run_infisical(args, quiet=True):
"""Run infisical CLI with env-based auth, return stdout or None."""
env = os.environ.copy()
env["INFISICAL_API_URL"] = INFISICAL_API_URL
if INFISICAL_TOKEN:
env["INFISICAL_TOKEN"] = INFISICAL_TOKEN
try:
result = subprocess.run(
["/usr/bin/infisical"] + args,
capture_output=True, text=True, timeout=15, env=env
)
return result.stdout.strip() if result.returncode == 0 else None
except:
return None
# ── KEY LOOKUP FIX ───────────────────────────────────────────────────
def _get_agent_key(agent_name, vault_key_name):
"""Retrieve agent-specific key from Infisical vault.
Uses {NAME}_LITELLM_API_KEY format (e.g., TANKO_LITELLM_API_KEY,
KOONIMO_LITELLM_API_KEY) which matches actual vault key names.
"""
if not vault_key_name:
return None
# Primary: get the agent-specific key by name
key = run_infisical([
"secrets", "get", vault_key_name,
"--projectId=" + INFISICAL_PROJECT,
"--env=" + INFISICAL_ENV,
"--plain",
])
if key and key.startswith("sk-"):
return key
# Fallback: export all and search for the key name
try:
export = run_infisical([
"export",
"--projectId=" + INFISICAL_PROJECT,
"--env=" + INFISICAL_ENV,
"--format=dotenv",
])
if export:
for line in export.splitlines():
if line.startswith(vault_key_name + "="):
value = line.split("=", 1)[1].strip().strip('"').strip("'")
if value.startswith("sk-"):
return value
except:
pass
return None
# Inject keys from vault for each agent
for agent_name in AGENTS:
info = AGENTS[agent_name]
key = _get_agent_key(agent_name, info.get("vault_key"))
AGENTS[agent_name]["key"] = key
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
# CHECK 1: LiteLLM Key Validation # CHECK 1: LiteLLM Key Validation (agent-specific keys)
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
def check_keys(): def check_keys():
for name, agent in AGENTS.items(): for name, agent in AGENTS.items():
key = agent.get("key")
if not key:
print(f"{name}: NO KEY FOUND (vault empty or unreachable)")
FAIL.append(f"key:{name}:no-key")
continue
data = http_json(f"{LITELLM}/v1/models", data = http_json(f"{LITELLM}/v1/models",
headers={"Authorization": f"Bearer {agent['key']}"}) headers={"Authorization": f"Bearer {key}"})
if data and data.get("data"): if data and data.get("data"):
model = data["data"][0].get("id", "?") model = data["data"][0].get("id", "?")
print(f"{name}: key valid → {model}") print(f"{name}: key valid → {model}")
@@ -130,7 +181,7 @@ def check_keys():
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
# CHECK 2: GPU Port Conflict Detection # CHECK 2: GPU Port Conflict Detection (unchanged)
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
def check_gpu_ports(): def check_gpu_ports():
@@ -158,7 +209,6 @@ def check_gpu_ports():
else: else:
print(f" ⚠️ {label}: svc={svc_status}, port owned by {port_owner}") print(f" ⚠️ {label}: svc={svc_status}, port owned by {port_owner}")
else: else:
# Verify health endpoint
health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health") health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
if health and '"status":"ok"' in health: if health and '"status":"ok"' in health:
print(f"{label}: healthy (pid={port_owner})") print(f"{label}: healthy (pid={port_owner})")
@@ -171,7 +221,7 @@ def check_gpu_ports():
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
# CHECK 3: Agent Gateway Liveness + Streaming # CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents)
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
def check_agents(): def check_agents():
@@ -184,8 +234,11 @@ def check_agents():
print(f"{name} (CT {ct}): cannot SSH — skip liveness check") print(f"{name} (CT {ct}): cannot SSH — skip liveness check")
continue continue
# Gateway process (exclude the infisical bash wrapper that contains the same string) # Gateway process
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user) pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
if not pid:
# Try alternate binary name
pid = ssh(host, "pgrep -f 'hermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
if not pid: if not pid:
print(f"{name}: GATEWAY NOT RUNNING") print(f"{name}: GATEWAY NOT RUNNING")
FAIL.append(f"gateway-down:{name}") FAIL.append(f"gateway-down:{name}")
@@ -203,7 +256,7 @@ def check_agents():
else: else:
gw_state, zulip = "no-state-file", "?" gw_state, zulip = "no-state-file", "?"
# Zulip streaming: does adapter have edit_message? # Zulip streaming check
adapter_paths = [ adapter_paths = [
"~/.hermes/plugins/zulip-platform/adapter.py", "~/.hermes/plugins/zulip-platform/adapter.py",
"~/.hermes/plugins/platforms/zulip/adapter.py", "~/.hermes/plugins/platforms/zulip/adapter.py",
@@ -220,13 +273,183 @@ def check_agents():
r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null " r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0", r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
user=user) user=user)
recent_errors = (recent_errors or "0").strip().split("\n")[-1] # take last line recent_errors = (recent_errors or "0").strip().split("\n")[-1]
print(f" {'' if gw_state == 'running' and zulip == 'connected' else '⚠️'} " print(f" {'' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} " f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
f"errors_10m={recent_errors.strip() or '0'} pid={pid}") f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 4: CT Liveness (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_ct_liveness():
"""Check that all agent CTs are running on their PVE nodes."""
for name, agent in AGENTS.items():
ct = agent["ct"]
pve_node = agent.get("pve")
if not pve_node:
print(f"{name} (CT {ct}): no PVE node mapped — skip")
continue
pve_ip = PVE_NODES.get(pve_node)
if not pve_ip:
print(f"{name}: unknown PVE node '{pve_node}' — skip")
continue
status = ssh(pve_ip, f"pct status {ct} 2>/dev/null", user="root")
if not status:
print(f"{name} (CT {ct} on {pve_node}): PVE UNREACHABLE")
FAIL.append(f"ct-unreachable:{name}:{pve_ip}")
elif "running" in status:
print(f"{name} (CT {ct} on {pve_node}): running")
elif "stopped" in status:
print(f"{name} (CT {ct} on {pve_node}): STOPPED")
FAIL.append(f"ct-stopped:{name}")
else:
print(f" ⚠️ {name} (CT {ct} on {pve_node}): {status.strip()}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 5: Config YAML Integrity (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_config_integrity():
"""Verify agent config.yaml parses as valid YAML."""
for name, agent in AGENTS.items():
host = agent.get("host")
user = agent.get("user")
if not host or not user:
print(f"{name}: cannot SSH — skip config check")
continue
# Check YAML parses
yaml_ok = ssh(host,
"python3 -c "
'"import yaml; yaml.safe_load(open(\'/root/.hermes/config.yaml\')); print(\'OK\')" '
"2>&1 || echo 'FAIL'",
user=user)
if not yaml_ok:
print(f"{name}: SSH UNREACHABLE (config check skipped)")
FAIL.append(f"config-unreachable:{name}")
elif "OK" in yaml_ok:
print(f"{name}: config.yaml valid YAML")
else:
print(f"{name}: config.yaml YAML ERROR — {yaml_ok[:120]}")
FAIL.append(f"config-yaml-error:{name}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 6: Wrapper/CLI Integrity (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_wrapper_integrity():
"""Verify the hermes CLI wrapper exists and can reach hermes-real."""
for name, agent in AGENTS.items():
host = agent.get("host")
user = agent.get("user")
if not host or not user:
print(f"{name}: cannot SSH — skip wrapper check")
continue
# Check wrapper exists
wrapper = ssh(host, "ls -la /root/.local/bin/hermes 2>/dev/null", user=user)
if not wrapper:
# Check alternate wrapper locations
wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
if not wrapper:
print(f"{name}: NO HERMES CLI WRAPPER FOUND")
FAIL.append(f"wrapper-missing:{name}")
continue
else:
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
# Check wrapper has correct infisical path
infisical_path_valid = ssh(host,
"head -20 /root/.local/bin/hermes 2>/dev/null | grep -q '/usr/bin/infisical' && echo OK || echo MISS",
user=user)
if infisical_path_valid == "MISS":
# Check if infisical exists on path
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
if not inf_actual:
print(f"{name}: INFISICAL NOT INSTALLED (wrapper broken)")
FAIL.append(f"wrapper-no-infisical:{name}")
else:
print(f" ⚠️ {name}: wrapper infisical path may be wrong (infisical at {inf_actual})")
FAIL.append(f"wrapper-infisical-path:{name}")
# Check hermes-real exists
hermes_real = ssh(host,
"ls -la /root/.local/bin/hermes-real 2>/dev/null || echo MISS",
user=user)
if not hermes_real or hermes_real.strip() == "MISS":
# Check venv path
hermes_real = ssh(host,
"ls -la /usr/local/lib/hermes-agent/venv/bin/hermes 2>/dev/null || echo MISS",
user=user)
if not hermes_real or hermes_real.strip() == "MISS":
print(f"{name}: hermes-real NOT FOUND (wrapper broken)")
FAIL.append(f"wrapper-no-hermes-real:{name}")
else:
print(f"{name}: hermes-real at alt path")
# Check the .env file has the key
env_has_key = ssh(host,
"grep -c 'LITELLM_API_KEY' /root/.hermes/.env 2>/dev/null || echo 0",
user=user)
if env_has_key and env_has_key.strip() not in ("", "0"):
print(f"{name}: wrapper + .env key present")
else:
print(f" ⚠️ {name}: .env may be missing LITELLM_API_KEY entry")
# ═══════════════════════════════════════════════════════════════════
# CHECK 7: Vault Secret Non-Emptiness (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_vault_secrets():
"""Verify agent-specific vault secrets are non-empty and start with sk-."""
for name, agent in AGENTS.items():
vault_key_name = agent.get("vault_key")
if not vault_key_name:
continue
key = agent.get("key")
if not key:
print(f"{name}: vault secret {vault_key_name} MISSING or EMPTY")
FAIL.append(f"vault-empty:{name}:{vault_key_name}")
elif not key.startswith("sk-"):
print(f"{name}: vault secret {vault_key_name} WRONG FORMAT (starts '{key[:8]}...')")
FAIL.append(f"vault-bad-format:{name}:{vault_key_name}")
else:
print(f"{name}: vault {vault_key_name}=sk-...{key[-4:]}")
# ═══════════════════════════════════════════════════════════════════
# DEPLOY: copy updated script to /root/scripts/ on local host
# ═══════════════════════════════════════════════════════════════════
def deploy_self():
"""Copy this script to /root/scripts/agent-health-check.py if out of date."""
dest = "/root/scripts/agent-health-check.py"
try:
with open(__file__, "r") as f:
current = f.read()
if os.path.isfile(dest):
with open(dest, "r") as f:
existing = f.read()
if current == existing:
return # Already deployed
# Write new version
with open(dest, "w") as f:
f.write(current)
os.chmod(dest, 0o755)
print(f" 📦 Deployed updated script to {dest}")
except:
pass # Not fatal if deploy fails
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
# MAIN # MAIN
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
@@ -235,8 +458,12 @@ def main():
quiet = "--quiet" in sys.argv quiet = "--quiet" in sys.argv
as_json = "--json" in sys.argv as_json = "--json" in sys.argv
# Self-deploy to canonical location
if not quiet and "--no-deploy" not in sys.argv:
deploy_self()
if not quiet: if not quiet:
print(f"🏥 Agent Health Check — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}") print(f"🏥 Agent Health Check v2 {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
print() print()
print("🔑 LiteLLM Keys:") print("🔑 LiteLLM Keys:")
@@ -249,11 +476,26 @@ def main():
print("🤖 Agent Gateways:") print("🤖 Agent Gateways:")
check_agents() check_agents()
print()
print("🖥️ CT Liveness:")
check_ct_liveness()
print()
print("📝 Config Integrity:")
check_config_integrity()
print()
print("🔌 Wrapper/CLI Integrity:")
check_wrapper_integrity()
print()
print("🔐 Vault Secrets:")
check_vault_secrets()
if FAIL: if FAIL:
print(f"\n{len(FAIL)} FAILURE(S): {' | '.join(FAIL)}") print(f"\n{len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
if quiet: if quiet:
# In quiet mode, only print failures as a single alert line
print(f"ALERT agent-health:{','.join(FAIL)}") print(f"ALERT agent-health:{','.join(FAIL)}")
elif not quiet: elif not quiet:
print("\n✅ All checks passed") print("\n✅ All checks passed")
+65
View File
@@ -0,0 +1,65 @@
#!/bin/bash
# Netbird Reverse Proxy — Add a new domain route
#
# Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]
#
# Example:
# netbird-add-domain.sh dns.sysloggh.net 192.168.68.10 80
#
# This script adds a domain to the Netbird proxy by inserting records
# directly into the management server's SQLite database, then restarting
# the proxy stack.
#
# Prerequisites: SSH root access to 72.61.0.17
# sqlite3 available on VPS
#
# Requires: The domain must already have a DNS CNAME to netbird.sysloggh.net
# pointing to 72.61.0.17.
set -euo pipefail
DOMAIN="${1:?Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]}"
BACKEND_IP="${2:?Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]}"
PORT="${3:-80}"
PROTOCOL="${4:-http}"
VPS="root@72.61.0.17"
DB_VOLUME="/var/lib/docker/volumes/root_netbird_data/_data"
DB="$DB_VOLUME/store.db"
echo "=== Adding Netbird proxy route ==="
echo "Domain: $DOMAIN"
echo "Backend: $BACKEND_IP:$PORT ($PROTOCOL)"
echo ""
ssh "$VPS" bash << REMOTESCRIPT
set -euo pipefail
# Generate unique ID using timestamp hash (Netbird format)
ID_SUFFIX=\$(date +%s | md5sum | head -c 16)
SVC_ID="d9\${ID_SUFFIX}ptsnc73\$(date +%s | md5sum | head -c 10)"
TGT_ID=\$(sqlite3 "$DB" "SELECT COALESCE(MAX(id), 100) + 1 FROM targets;")
ACCOUNT_ID="d88av3aptsnc73clmogg"
ZONE_ID="d8adqjaptsnc73fro5g0"
echo "Service ID: \$SVC_ID"
echo "Target ID: \$TGT_ID"
# Insert service
sqlite3 "$DB" "INSERT INTO services (id, account_id, name, domain, proxy_cluster, enabled, terminated, pass_host_header, rewrite_redirects, mode, source, port_auto_assigned, private) VALUES (\"\$SVC_ID\", \"\$ACCOUNT_ID\", \"$DOMAIN\", \"$DOMAIN\", \"netbird.sysloggh.net\", 1, 0, 1, 0, \"http\", \"permanent\", 0, 0);"
echo "Service: OK"
# Insert target
sqlite3 "$DB" "INSERT INTO targets (id, account_id, service_id, host, port, protocol, target_id, target_type, enabled, skip_tls_verify, request_timeout, session_idle_timeout, agent_network, disable_access_log) VALUES (\$TGT_ID, \"\$ACCOUNT_ID\", \"\$SVC_ID\", \"$BACKEND_IP\", $PORT, \"$PROTOCOL\", \"\$ZONE_ID\", \"subnet\", 1, 0, 0, 0, 0, 0);"
echo "Target: OK"
# Verify
sqlite3 -column "$DB" "SELECT s.name, t.host, t.port, t.protocol FROM services s JOIN targets t ON s.id=t.service_id WHERE s.name=\"$DOMAIN\";"
echo ""
echo "Restarting proxy stack..."
cd /root && docker compose restart netbird-server 2>/dev/null
sleep 15
docker compose restart proxy 2>/dev/null
echo "Done. Verify with: curl -sI https://$DOMAIN"
REMOTESCRIPT
+91
View File
@@ -0,0 +1,91 @@
#!/bin/bash
# swap-gpu-dense-model.sh — Swap RTX 3090 from qwen3.6-27B-code to SmartCode-Fable-5
# Run when download completes: ssh root@192.168.68.8 'bash -s' < this script
#
# Usage: bash swap-gpu-dense-model.sh
# Requires: new model at /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf
set -e
MODEL_PATH="/home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf"
OLD_WRAPPER="/home/llmuser/llama-wrapper.sh"
echo "═══ Swapping gpu-dense to SmartCode-Fable-5 ═══"
# 1. Verify model file
if [ ! -f "$MODEL_PATH" ]; then
echo "❌ Model not found at $MODEL_PATH"
echo " Download: curl -L -o $MODEL_PATH <huggingface-url>"
exit 1
fi
MODEL_SIZE=$(ls -lh "$MODEL_PATH" | awk '{print $5}')
echo "✅ Model found: $MODEL_SIZE"
# 2. Create new wrapper script for SmartCode-Fable-5
cat > /home/llmuser/llama-fable-wrapper.sh << 'WRAPPER'
#!/bin/bash
# SmartCode-Fable-5 llama-server wrapper for RTX 3090
# Sampler settings from model card: temp 0.9, top-p 0.95, top-k 60, repeat-penalty off
PORT=8080
GHOST_PID=$(ss -tlnp 2>/dev/null | grep -Po ":${PORT}\s+.*pid=\K[0-9]+" | head -1)
if [ -n "$GHOST_PID" ] && [ "$GHOST_PID" != "$$" ]; then
echo "[wrapper] Port $PORT occupied by ghost pid $GHOST_PID — cleaning up" >&2
kill -9 "$GHOST_PID" 2>/dev/null
sleep 2
fi
exec /usr/local/bin/llama-server \
--model /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf \
--ctx-size 131072 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--flash-attn 1 \
--cont-batching \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 1024 \
--n-gpu-layers 99 \
--temp 0.9 \
--top-p 0.95 \
--top-k 60 \
--min-p 0.0 \
--repeat-penalty 1.0 \
--api-key not-needed \
--port 8080 \
--host 0.0.0.0
WRAPPER
chmod 755 /home/llmuser/llama-fable-wrapper.sh
echo "✅ Created /home/llmuser/llama-fable-wrapper.sh"
# 3. Update systemd service to use new wrapper
echo "📝 Updating systemd service..."
sed -i 's|ExecStart=/home/llmuser/llama-wrapper.sh|ExecStart=/home/llmuser/llama-fable-wrapper.sh|' /etc/systemd/system/llama-server.service
systemctl daemon-reload
# 4. Stop old server, start new
echo "🔄 Restarting llama-server..."
systemctl stop llama-server
sleep 3
systemctl start llama-server
sleep 8
# 5. Verify
echo ""
echo "═══ Verification ═══"
systemctl is-active llama-server
echo ""
echo "Port 8080:"
ss -tlnp 2>/dev/null | grep ":8080" | head -1
echo ""
echo "GPU VRAM:"
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader 2>/dev/null
echo ""
echo "=== Health check ==="
curl -s --max-time 5 http://localhost:8080/health 2>/dev/null
echo ""
echo ""
echo "✅ Swap complete. Test via LiteLLM:"
echo " curl -s http://192.168.68.116/v1/chat/completions -H 'Authorization: Bearer <key>' -H 'Content-Type: application/json' -d '{\"model\":\"gpu-dense\",\"messages\":[{\"role\":\"user\",\"content\":\"write hello world in python\"}],\"max_tokens\":100}'"
+4 -4
View File
@@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
## Requires ## Requires
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY` - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14) - **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.24, inside Abiba CT100 on hwepve), and Agent Zero Docker host (192.168.68.14)
- **PM2** on localhost for pi process management - **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `localhost:9200` - **Network access** to `chat.sysloggh.net`, `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce` - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -182,13 +182,13 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
| `last_error` set | Log and monitor | | `last_error` set | Log and monitor |
| Crash loop >10/h | Alert user | | Crash loop >10/h | Alert user |
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123) ### Step 3: Platform B — Hermes (Tanko .122, Mumuni .24)
**B1: Gateway State** **B1: Gateway State**
```bash ```bash
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.24 "cat ~/.hermes/gateway_state.json" # Mumuni inside Abiba CT100
``` ```
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed. Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
@@ -199,7 +199,7 @@ Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error`
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep" ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
``` ```
Gateway PID should exist with uptime > 60s. Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (`pm2 restart mumuni-zulip`). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
**B3: Heartbeat Verification** **B3: Heartbeat Verification**
+1 -1
View File
@@ -65,7 +65,7 @@ triggers:
|------|----|------|---------| |------|----|------|---------|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` | | Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` | | Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
| Mumuni | 192.168.68.123 | root | `hermes gateway restart` | | Mumuni | 192.168.68.24 (CT100 abiba) | root | `hermes gateway restart` |
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` | | Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
## Debounce ## Debounce