Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 root
|
e0f5e4f74e |
+12
-15
@@ -14,10 +14,6 @@ description: >
|
||||
Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling.
|
||||
For larger context needs → fall back to external providers (deepseek).
|
||||
VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
|
||||
UPDATED 2026-07-27: gpu-dense swapped to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled
|
||||
(UD-Q3_K_XL, ~14.7GB — Q4 was too large for 24GB VRAM with 128K KV cache). ~50% fewer thinking
|
||||
tokens via ThinkingCap finetune + Fable 5 CoT distillation for improved coding reasoning.
|
||||
VRAM ~22.4/24.6GB (91%).
|
||||
agent: abiba
|
||||
triggers:
|
||||
- on model add/remove
|
||||
@@ -90,7 +86,7 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen
|
||||
| Alias | GPU | Current Model | Will Route To |
|
||||
|-------|-----|---------------|---------------|
|
||||
| `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
|
||||
| `gpu-dense` | RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q4_K_XL | Whatever runs on RTX 3090 |
|
||||
| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 |
|
||||
| `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
|
||||
|
||||
**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
|
||||
@@ -100,7 +96,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
|
||||
|
||||
| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
|
||||
|-------|-----|------|------|-----|----------|----------|-------------|--------|
|
||||
| SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 | .8 (llm-gpu) | ~22.4/24.6GB (91%) | **128K** | q4_0 | 1 | 2048/1024 | ✅ healthy |
|
||||
| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s |
|
||||
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
|
||||
| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |
|
||||
|
||||
@@ -110,7 +106,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
|
||||
|
||||
| Model | GPU | Weight | RPM Cap | Timeout |
|
||||
|-------|-----|--------|---------|---------|
|
||||
| SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
|
||||
| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
|
||||
| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
|
||||
| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
|
||||
|
||||
@@ -121,7 +117,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
|
||||
| Model | RPM Cap | Notes |
|
||||
|-------|---------|-------|
|
||||
| strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload |
|
||||
| SmartCode-Fable-5-27B-UD-Q3_K_XL | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) |
|
||||
| qwen3.6-27B-code | 500 | High cap — primary workhorse |
|
||||
| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
|
||||
|
||||
### Stable Aliases (for agent configs — never change)
|
||||
@@ -208,7 +204,7 @@ Plaintext keys removed from this contract post-vault-migration.
|
||||
| Agent | CT | IP | LiteLLM Alias | Key Source | Access |
|
||||
|-------|-----|-----|---------------|------------|--------|
|
||||
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
|
||||
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | SSH root |
|
||||
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
|
||||
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
|
||||
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
|
||||
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
|
||||
@@ -251,12 +247,13 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
|
||||
Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
|
||||
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
|
||||
- **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
|
||||
- **RTX 3090 (2026-07-27)**: Swapped to SmartCode-Fable-5-27B-UD-Q3_K_XL (14.7GB). Q4 was too large for 24GB VRAM with 128K context + KV cache overhead. Q3 fits at ~22.4GB (91%). Uses standard llama.cpp build b9190 (turboquant b9150 incompatible with qwen3_5 arch). Config: `-c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on --cont-batching`. Sampler: `--temp 0.9 --top-p 0.95 --top-k 60 --min-p 0.0 --repeat-penalty 1.0`. Service: `/home/llmuser/llama-fable-wrapper.sh`.
|
||||
- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
|
||||
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`.
|
||||
- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
|
||||
- **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
|
||||
- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
|
||||
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
|
||||
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
|
||||
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (old Mumuni CT114 — now inside Abiba CT100 at .24) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
|
||||
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
|
||||
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
|
||||
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
|
||||
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
|
||||
@@ -268,7 +265,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
|
||||
|
||||
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
|
||||
|-----|-------|-----------|--------------|----------|---------|
|
||||
| RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q3_K_XL | **TBD** | — | — | **128K** |
|
||||
| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** |
|
||||
| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
|
||||
| Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** |
|
||||
|
||||
@@ -301,7 +298,7 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
|
||||
|
||||
### Mumuni Agent Profile
|
||||
|
||||
Mumuni (CT100/abiba, 192.168.68.24) is the primary business assistant. This profile is the reference for all agent configs:
|
||||
Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs:
|
||||
|
||||
| Setting | Value | Notes |
|
||||
|---------|-------|-------|
|
||||
@@ -326,7 +323,7 @@ Mumuni (CT100/abiba, 192.168.68.24) is the primary business assistant. This prof
|
||||
|
||||
| Agent | Host | Status |
|
||||
|-------|------|--------|
|
||||
| **Mumuni** | CT100 (.24) | ✅ Updated to stable aliases |
|
||||
| **Mumuni** | CT114 (.123) | ✅ Updated to stable aliases |
|
||||
| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
|
||||
| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
|
||||
| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
|
||||
|
||||
@@ -25,7 +25,7 @@ done
|
||||
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|
||||
|-------|-----|------|-----|---------------|------------|----------|
|
||||
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
||||
| Mumuni | 100 (abiba) | hwepve | .24 | `mumuni` | Infisical vault | Hermes (Pi gateway) |
|
||||
| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
|
||||
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
||||
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
||||
|
||||
@@ -35,7 +35,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
|
||||
| Agent | Key Alias | Host | SSH | Sub-Agents |
|
||||
|-------|-----------|------|-----|-----------|
|
||||
| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
|
||||
| Mumuni | `mumuni` | 192.168.68.24 (CT100 abiba) | root@.24 | 6 profiles ✱ |
|
||||
| Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
|
||||
| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
|
||||
| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
|
||||
| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
|
||||
|
||||
@@ -189,7 +189,7 @@ litellm_settings:
|
||||
| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
|
||||
|-------|-----|-----|---------------|------------|--------|-----------------|---------------|
|
||||
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
|
||||
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
|
||||
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
|
||||
| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
|
||||
| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
|
||||
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
|
||||
|
||||
@@ -51,7 +51,7 @@ gateway restart, and connection validation.
|
||||
|
||||
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
||||
|------|-----|---------|-------------|-------------|------|
|
||||
| Mumuni | CT100 (abiba) | hwepve | 192.168.68.24 | /root/.hermes | root |
|
||||
| Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root |
|
||||
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
|
||||
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
|
||||
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
|
||||
|
||||
@@ -13,10 +13,11 @@ description: >
|
||||
against the live system. Policy fields are authoritative. See the
|
||||
`verify-before-mutate` skill.
|
||||
|
||||
**Last verified:** 2026-07-27 — gpu-dense swapped to SmartCode-Fable-5
|
||||
(Qwen3.6-27B distilled, ~50% fewer thinking tokens, improved coding reasoning).
|
||||
Model pricing reduced 100x across all models ($0.15/$0.60 per 1M tokens).
|
||||
All LiteLLM models set to 128K max_model_tokens. See data/learnings.md.
|
||||
**Last verified:** 2026-07-26 — verification protocol run. CT 118 set to
|
||||
static IP .20 (was DHCP .131). llama-server on RTX 5070 (.110) was down,
|
||||
restarted and verified gemma-4-12b responding through LiteLLM. All 6 PVE
|
||||
nodes confirmed. All CT hostnames/IPs confirmed except jdownloader (now
|
||||
static). All public endpoints responding. See data/learnings.md.
|
||||
---
|
||||
|
||||
# Infrastructure Control Pattern
|
||||
@@ -72,7 +73,7 @@ description: >
|
||||
| docker-vm (.7) | SSH root | SSH key | ✅ |
|
||||
| CT 116 (syslog-api) | SSH root | SSH key | ✅ |
|
||||
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
|
||||
| Mumuni (inside CT100 abiba .24) | SSH root | id_ed25519 | ✅ |
|
||||
| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
|
||||
| Baggy CT (113) | SSH jerome | ❌ no key access |
|
||||
| Netbird (.17) | SSH root | SSH key | ✅ |
|
||||
| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
|
||||
@@ -90,11 +91,11 @@ description: >
|
||||
|
||||
### Reachability Matrix
|
||||
|
||||
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.24) | Baggy (.114) |
|
||||
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
|
||||
|-----------|---------|----------------|--------|-------------|---------------|----------------|
|
||||
| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
|
||||
| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
|
||||
| **Mumuni** (.24) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||
| **Mumuni** (.123) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||
|
||||
**Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.
|
||||
|
||||
@@ -112,9 +113,9 @@ description: >
|
||||
| hwepve | .4 | ? | ? | abiba, (mumuni CT 114 stopped) | Agents (new node) |
|
||||
|
||||
> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
|
||||
> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve and now runs Mumuni
|
||||
> Zulip gateway internally. CT 114 (old mumuni container) destroyed 2026-07-26.
|
||||
> Mumuni now runs inside Abiba CT100 (.24). Old CT114 destroyed 2026-07-26.
|
||||
> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve, not amdpve. Mumuni
|
||||
> (CT 114) is on hwepve (currently stopped), not minipve. Mumuni also has a
|
||||
> second instance on minipve at .123 — distinguish by CT ID, not hostname.
|
||||
|
||||
### Checks (every 5 min)
|
||||
|
||||
@@ -604,6 +605,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
|
||||
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
|
||||
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
|
||||
| 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
|
||||
| 114 | mumuni | **hwepve** | .123 | Hermes agent (stopped) | ✅ |
|
||||
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
|
||||
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
||||
| 117 | zulip | storepve | .19 | Chat | ❌ |
|
||||
@@ -636,7 +638,7 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
|
||||
| 115 | scottdenya | amdpve | `pct-run 115` |
|
||||
| 104 | authentik | minipve | `pct-run 104` |
|
||||
| 110 | gitea | minipve | `pct-run 110` |
|
||||
|
||||
| 114 | mumuni | **hwepve** | `pct-run 114` |
|
||||
| 116 | syslog-api | minipve | `pct-run 116` |
|
||||
| 106 | ra-h-os | storepve | `pct-run 106` |
|
||||
| 107 | proxmox-backup | storepve | `pct-run 107` |
|
||||
@@ -649,7 +651,7 @@ GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use S
|
||||
ssh root@192.168.68.8 # RTX 3090
|
||||
ssh root@192.168.68.110 # RTX 5070
|
||||
ssh root@192.168.68.15 # Strix Halo
|
||||
ssh root@192.168.68.4 # hwepve (abiba, kagentz) — Mumuni runs inside CT 100
|
||||
ssh root@192.168.68.4 # hwepve (abiba, kagentz, mumuni)
|
||||
```
|
||||
|
||||
## Section 7: Agent Health Check v2 (2026-07-26)
|
||||
|
||||
@@ -178,7 +178,7 @@ through its agent wrapper.
|
||||
| Agent | Host | Pattern | Keys | Status |
|
||||
|-------|------|---------|------|--------|
|
||||
| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
|
||||
| mumuni | .24 (CT100 abiba) | Pi Hermes gateway (no systemd) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
|
||||
| mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
|
||||
| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
|
||||
| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
|
||||
| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
|
||||
|
||||
@@ -102,7 +102,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
|
||||
|
||||
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
|
||||
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
|
||||
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
|
||||
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.123), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
|
||||
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
|
||||
|
||||
## Maintains
|
||||
|
||||
@@ -6,7 +6,7 @@ description: >
|
||||
delegation, verification, and delivery. Defines when to delegate, which
|
||||
worker to use for what, how to handle failures, and the kanban board
|
||||
protocol. Enforces context-window discipline and separation of concerns.
|
||||
Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Pi + Hermes Zulip gateway).
|
||||
Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent.
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
@@ -20,7 +20,7 @@ version: 1.0.0
|
||||
## Topology
|
||||
|
||||
**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
|
||||
**Manager:** Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent
|
||||
**Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent
|
||||
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
|
||||
|
||||
This contract is infrastructure-agnostic in terms of which nodes are used.
|
||||
|
||||
@@ -17,7 +17,7 @@ Changelog:
|
||||
v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
|
||||
vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
|
||||
name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
|
||||
Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114),
|
||||
Fleet roster: tanko (.122), mumuni (.123), koby (.129), koonimo (.114),
|
||||
abiba (.24).
|
||||
"""
|
||||
|
||||
@@ -41,9 +41,10 @@ PVE_NODES = {
|
||||
# Agent definitions: ct, host, user, pve_node, vault_key_name
|
||||
AGENTS = {
|
||||
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve", "vault_key": "TANKO_LITELLM_API_KEY"},
|
||||
"abiba": {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "hwepve", "vault_key": None}, # Pi agent + Mumuni Zulip, no vault key
|
||||
"mumuni": {"ct": 114, "host": "192.168.68.123", "user": "root", "pve": "hwepve", "vault_key": "MUMUNI_LITELLM_API_KEY"},
|
||||
"koby": {"ct": 111, "host": "192.168.68.129", "user": "root", "pve": "amdpve", "vault_key": "KOBY_LITELLM_API_KEY"},
|
||||
"koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root", "pve": "amdpve", "vault_key": "KOONIMO_LITELLM_API_KEY"},
|
||||
"abiba": {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "hwepve", "vault_key": None}, # Pi agent, no vault key
|
||||
}
|
||||
|
||||
GPU_HOSTS = {
|
||||
|
||||
@@ -1,91 +0,0 @@
|
||||
#!/bin/bash
|
||||
# swap-gpu-dense-model.sh — Swap RTX 3090 from qwen3.6-27B-code to SmartCode-Fable-5
|
||||
# Run when download completes: ssh root@192.168.68.8 'bash -s' < this script
|
||||
#
|
||||
# Usage: bash swap-gpu-dense-model.sh
|
||||
# Requires: new model at /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf
|
||||
|
||||
set -e
|
||||
|
||||
MODEL_PATH="/home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf"
|
||||
OLD_WRAPPER="/home/llmuser/llama-wrapper.sh"
|
||||
|
||||
echo "═══ Swapping gpu-dense to SmartCode-Fable-5 ═══"
|
||||
|
||||
# 1. Verify model file
|
||||
if [ ! -f "$MODEL_PATH" ]; then
|
||||
echo "❌ Model not found at $MODEL_PATH"
|
||||
echo " Download: curl -L -o $MODEL_PATH <huggingface-url>"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
MODEL_SIZE=$(ls -lh "$MODEL_PATH" | awk '{print $5}')
|
||||
echo "✅ Model found: $MODEL_SIZE"
|
||||
|
||||
# 2. Create new wrapper script for SmartCode-Fable-5
|
||||
cat > /home/llmuser/llama-fable-wrapper.sh << 'WRAPPER'
|
||||
#!/bin/bash
|
||||
# SmartCode-Fable-5 llama-server wrapper for RTX 3090
|
||||
# Sampler settings from model card: temp 0.9, top-p 0.95, top-k 60, repeat-penalty off
|
||||
|
||||
PORT=8080
|
||||
GHOST_PID=$(ss -tlnp 2>/dev/null | grep -Po ":${PORT}\s+.*pid=\K[0-9]+" | head -1)
|
||||
if [ -n "$GHOST_PID" ] && [ "$GHOST_PID" != "$$" ]; then
|
||||
echo "[wrapper] Port $PORT occupied by ghost pid $GHOST_PID — cleaning up" >&2
|
||||
kill -9 "$GHOST_PID" 2>/dev/null
|
||||
sleep 2
|
||||
fi
|
||||
|
||||
exec /usr/local/bin/llama-server \
|
||||
--model /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf \
|
||||
--ctx-size 131072 \
|
||||
--cache-type-k q4_0 \
|
||||
--cache-type-v q4_0 \
|
||||
--flash-attn 1 \
|
||||
--cont-batching \
|
||||
--parallel 1 \
|
||||
--batch-size 2048 \
|
||||
--ubatch-size 1024 \
|
||||
--n-gpu-layers 99 \
|
||||
--temp 0.9 \
|
||||
--top-p 0.95 \
|
||||
--top-k 60 \
|
||||
--min-p 0.0 \
|
||||
--repeat-penalty 1.0 \
|
||||
--api-key not-needed \
|
||||
--port 8080 \
|
||||
--host 0.0.0.0
|
||||
WRAPPER
|
||||
|
||||
chmod 755 /home/llmuser/llama-fable-wrapper.sh
|
||||
echo "✅ Created /home/llmuser/llama-fable-wrapper.sh"
|
||||
|
||||
# 3. Update systemd service to use new wrapper
|
||||
echo "📝 Updating systemd service..."
|
||||
sed -i 's|ExecStart=/home/llmuser/llama-wrapper.sh|ExecStart=/home/llmuser/llama-fable-wrapper.sh|' /etc/systemd/system/llama-server.service
|
||||
systemctl daemon-reload
|
||||
|
||||
# 4. Stop old server, start new
|
||||
echo "🔄 Restarting llama-server..."
|
||||
systemctl stop llama-server
|
||||
sleep 3
|
||||
systemctl start llama-server
|
||||
sleep 8
|
||||
|
||||
# 5. Verify
|
||||
echo ""
|
||||
echo "═══ Verification ═══"
|
||||
systemctl is-active llama-server
|
||||
echo ""
|
||||
echo "Port 8080:"
|
||||
ss -tlnp 2>/dev/null | grep ":8080" | head -1
|
||||
echo ""
|
||||
echo "GPU VRAM:"
|
||||
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader 2>/dev/null
|
||||
echo ""
|
||||
echo "=== Health check ==="
|
||||
curl -s --max-time 5 http://localhost:8080/health 2>/dev/null
|
||||
echo ""
|
||||
echo ""
|
||||
echo "✅ Swap complete. Test via LiteLLM:"
|
||||
echo " curl -s http://192.168.68.116/v1/chat/completions -H 'Authorization: Bearer <key>' -H 'Content-Type: application/json' -d '{\"model\":\"gpu-dense\",\"messages\":[{\"role\":\"user\",\"content\":\"write hello world in python\"}],\"max_tokens\":100}'"
|
||||
@@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
|
||||
## Requires
|
||||
|
||||
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
||||
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.24, inside Abiba CT100 on hwepve), and Agent Zero Docker host (192.168.68.14)
|
||||
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
|
||||
- **PM2** on localhost for pi process management
|
||||
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
||||
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
||||
@@ -182,13 +182,13 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
|
||||
| `last_error` set | Log and monitor |
|
||||
| Crash loop >10/h | Alert user |
|
||||
|
||||
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .24)
|
||||
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
|
||||
|
||||
**B1: Gateway State**
|
||||
|
||||
```bash
|
||||
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
|
||||
ssh root@192.168.68.24 "cat ~/.hermes/gateway_state.json" # Mumuni inside Abiba CT100
|
||||
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
|
||||
```
|
||||
|
||||
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
|
||||
|
||||
@@ -65,7 +65,7 @@ triggers:
|
||||
|------|----|------|---------|
|
||||
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
|
||||
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
|
||||
| Mumuni | 192.168.68.24 (CT100 abiba) | root | `hermes gateway restart` |
|
||||
| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
|
||||
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
|
||||
|
||||
## Debounce
|
||||
|
||||
Reference in New Issue
Block a user