Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e052a069cb | ||
|
|
c9359e1808 | ||
|
|
6a0af728e9 | ||
|
|
3b6cf44a30 | ||
|
|
835647e241 |
+4
-4
@@ -208,7 +208,7 @@ Plaintext keys removed from this contract post-vault-migration.
|
|||||||
| Agent | CT | IP | LiteLLM Alias | Key Source | Access |
|
| Agent | CT | IP | LiteLLM Alias | Key Source | Access |
|
||||||
|-------|-----|-----|---------------|------------|--------|
|
|-------|-----|-----|---------------|------------|--------|
|
||||||
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
|
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
|
||||||
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
|
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | SSH root |
|
||||||
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
|
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
|
||||||
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
|
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
|
||||||
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
|
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
|
||||||
@@ -256,7 +256,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
|
|||||||
- **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
|
- **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
|
||||||
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
|
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
|
||||||
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
|
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
|
||||||
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
|
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (old Mumuni CT114 — now inside Abiba CT100 at .24) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
|
||||||
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
|
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
|
||||||
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
|
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
|
||||||
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
|
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
|
||||||
@@ -301,7 +301,7 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
|
|||||||
|
|
||||||
### Mumuni Agent Profile
|
### Mumuni Agent Profile
|
||||||
|
|
||||||
Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs:
|
Mumuni (CT100/abiba, 192.168.68.24) is the primary business assistant. This profile is the reference for all agent configs:
|
||||||
|
|
||||||
| Setting | Value | Notes |
|
| Setting | Value | Notes |
|
||||||
|---------|-------|-------|
|
|---------|-------|-------|
|
||||||
@@ -326,7 +326,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
|
|||||||
|
|
||||||
| Agent | Host | Status |
|
| Agent | Host | Status |
|
||||||
|-------|------|--------|
|
|-------|------|--------|
|
||||||
| **Mumuni** | CT114 (.123) | ✅ Updated to stable aliases |
|
| **Mumuni** | CT100 (.24) | ✅ Updated to stable aliases |
|
||||||
| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
|
| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
|
||||||
| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
|
| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
|
||||||
| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
|
| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
|
||||||
|
|||||||
@@ -251,8 +251,8 @@ call update-gpu-health
|
|||||||
|
|
||||||
## Reporting
|
## Reporting
|
||||||
|
|
||||||
### 1. Knowledge Graph
|
### 1. Gitea Log (not knowledge graph — hard rule)
|
||||||
Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
|
Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchable, not in graph.
|
||||||
|
|
||||||
### 2. Zulip Alerts (#agent-hub → alerts-gpu)
|
### 2. Zulip Alerts (#agent-hub → alerts-gpu)
|
||||||
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
|
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ done
|
|||||||
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|
||||||
|-------|-----|------|-----|---------------|------------|----------|
|
|-------|-----|------|-----|---------------|------------|----------|
|
||||||
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
||||||
| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
|
| Mumuni | 100 (abiba) | hwepve | .24 | `mumuni` | Infisical vault | Hermes (Pi gateway) |
|
||||||
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||||
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
||||||
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
||||||
|
|||||||
@@ -35,7 +35,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
|
|||||||
| Agent | Key Alias | Host | SSH | Sub-Agents |
|
| Agent | Key Alias | Host | SSH | Sub-Agents |
|
||||||
|-------|-----------|------|-----|-----------|
|
|-------|-----------|------|-----|-----------|
|
||||||
| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
|
| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
|
||||||
| Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
|
| Mumuni | `mumuni` | 192.168.68.24 (CT100 abiba) | root@.24 | 6 profiles ✱ |
|
||||||
| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
|
| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
|
||||||
| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
|
| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
|
||||||
| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
|
| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
|
||||||
|
|||||||
@@ -189,7 +189,7 @@ litellm_settings:
|
|||||||
| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
|
| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
|
||||||
|-------|-----|-----|---------------|------------|--------|-----------------|---------------|
|
|-------|-----|-----|---------------|------------|--------|-----------------|---------------|
|
||||||
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
|
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
|
||||||
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
|
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
|
||||||
| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
|
| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
|
||||||
| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
|
| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
|
||||||
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
|
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ gateway restart, and connection validation.
|
|||||||
|
|
||||||
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
||||||
|------|-----|---------|-------------|-------------|------|
|
|------|-----|---------|-------------|-------------|------|
|
||||||
| Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root |
|
| Mumuni | CT100 (abiba) | hwepve | 192.168.68.24 | /root/.hermes | root |
|
||||||
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
|
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
|
||||||
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
|
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
|
||||||
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
|
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
|
||||||
|
|||||||
@@ -72,7 +72,7 @@ description: >
|
|||||||
| docker-vm (.7) | SSH root | SSH key | ✅ |
|
| docker-vm (.7) | SSH root | SSH key | ✅ |
|
||||||
| CT 116 (syslog-api) | SSH root | SSH key | ✅ |
|
| CT 116 (syslog-api) | SSH root | SSH key | ✅ |
|
||||||
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
|
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
|
||||||
| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
|
| Mumuni (inside CT100 abiba .24) | SSH root | id_ed25519 | ✅ |
|
||||||
| Baggy CT (113) | SSH jerome | ❌ no key access |
|
| Baggy CT (113) | SSH jerome | ❌ no key access |
|
||||||
| Netbird (.17) | SSH root | SSH key | ✅ |
|
| Netbird (.17) | SSH root | SSH key | ✅ |
|
||||||
| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
|
| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
|
||||||
@@ -90,11 +90,11 @@ description: >
|
|||||||
|
|
||||||
### Reachability Matrix
|
### Reachability Matrix
|
||||||
|
|
||||||
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
|
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.24) | Baggy (.114) |
|
||||||
|-----------|---------|----------------|--------|-------------|---------------|----------------|
|
|-----------|---------|----------------|--------|-------------|---------------|----------------|
|
||||||
| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
|
| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
|
||||||
| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
|
| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
|
||||||
| **Mumuni** (.123) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
| **Mumuni** (.24) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||||
|
|
||||||
**Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.
|
**Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.
|
||||||
|
|
||||||
@@ -114,7 +114,7 @@ description: >
|
|||||||
> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
|
> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
|
||||||
> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve and now runs Mumuni
|
> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve and now runs Mumuni
|
||||||
> Zulip gateway internally. CT 114 (old mumuni container) destroyed 2026-07-26.
|
> Zulip gateway internally. CT 114 (old mumuni container) destroyed 2026-07-26.
|
||||||
> Mumuni also has a second instance on minipve at .123 — distinguish by CT ID.
|
> Mumuni now runs inside Abiba CT100 (.24). Old CT114 destroyed 2026-07-26.
|
||||||
|
|
||||||
### Checks (every 5 min)
|
### Checks (every 5 min)
|
||||||
|
|
||||||
|
|||||||
@@ -178,7 +178,7 @@ through its agent wrapper.
|
|||||||
| Agent | Host | Pattern | Keys | Status |
|
| Agent | Host | Pattern | Keys | Status |
|
||||||
|-------|------|---------|------|--------|
|
|-------|------|---------|------|--------|
|
||||||
| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
|
| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
|
||||||
| mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
|
| mumuni | .24 (CT100 abiba) | Pi Hermes gateway (no systemd) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
|
||||||
| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
|
| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
|
||||||
| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
|
| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
|
||||||
| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
|
| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ note: >
|
|||||||
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
|
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
|
||||||
now reimplemented as `litellm-health-check.sh` on CT 116.
|
now reimplemented as `litellm-health-check.sh` on CT 116.
|
||||||
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
|
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
|
||||||
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
|
Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs).
|
||||||
GPU monitoring integrated from gpu-monitor on .24:9100.
|
GPU monitoring integrated from gpu-monitor on .24:9100.
|
||||||
|
|
||||||
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
|
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
|
||||||
@@ -20,7 +20,7 @@ description: >
|
|||||||
LiteLLM inference stack health monitoring + self-healing. Verifies the full
|
LiteLLM inference stack health monitoring + self-healing. Verifies the full
|
||||||
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
|
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
|
||||||
inference, and agent keys. Applies remediation rules for common failures.
|
inference, and agent keys. Applies remediation rules for common failures.
|
||||||
Reports every action via Zulip DM and RA-H OS knowledge graph.
|
Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
|
||||||
---
|
---
|
||||||
|
|
||||||
# LiteLLM Operations — Health Check + Self-Heal
|
# LiteLLM Operations — Health Check + Self-Heal
|
||||||
@@ -102,7 +102,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
|
|||||||
|
|
||||||
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
|
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
|
||||||
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
|
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
|
||||||
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.123), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
|
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
|
||||||
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
|
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
|
||||||
|
|
||||||
## Maintains
|
## Maintains
|
||||||
@@ -211,8 +211,8 @@ for reference but inactive. If Redis issues occur, check harness-redis container
|
|||||||
|
|
||||||
Every remediation cycle produces a structured report:
|
Every remediation cycle produces a structured report:
|
||||||
|
|
||||||
### 1. RA-H OS Knowledge Graph Node
|
### 1. Gitea Log Entry
|
||||||
Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
|
Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph.
|
||||||
|
|
||||||
### 2. Zulip DM to Owner
|
### 2. Zulip DM to Owner
|
||||||
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
|
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
|
||||||
@@ -252,9 +252,11 @@ call report-generator
|
|||||||
health: health
|
health: health
|
||||||
actions: actions
|
actions: actions
|
||||||
|
|
||||||
-- Phase 4: Log to knowledge graph
|
-- Phase 4: Log to Gitea (not knowledge graph — hard rule)
|
||||||
call kg-logger
|
call gitea-logger
|
||||||
run_id: run_id
|
run_id: run_id
|
||||||
|
repo: SyslogSolution/health-logs
|
||||||
|
path: litellm/{run_id}.json
|
||||||
health: health
|
health: health
|
||||||
actions: actions
|
actions: actions
|
||||||
|
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ description: >
|
|||||||
delegation, verification, and delivery. Defines when to delegate, which
|
delegation, verification, and delivery. Defines when to delegate, which
|
||||||
worker to use for what, how to handle failures, and the kanban board
|
worker to use for what, how to handle failures, and the kanban board
|
||||||
protocol. Enforces context-window discipline and separation of concerns.
|
protocol. Enforces context-window discipline and separation of concerns.
|
||||||
Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent.
|
Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Pi + Hermes Zulip gateway).
|
||||||
version: 1.0.0
|
version: 1.0.0
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -20,7 +20,7 @@ version: 1.0.0
|
|||||||
## Topology
|
## Topology
|
||||||
|
|
||||||
**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
|
**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
|
||||||
**Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent
|
**Manager:** Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent
|
||||||
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
|
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
|
||||||
|
|
||||||
This contract is infrastructure-agnostic in terms of which nodes are used.
|
This contract is infrastructure-agnostic in terms of which nodes are used.
|
||||||
|
|||||||
@@ -52,7 +52,7 @@ description: >
|
|||||||
- If status is "online" → pass, log restarts count
|
- If status is "online" → pass, log restarts count
|
||||||
- If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
|
- If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
|
||||||
- If restarts > 5 in last hour → alert owner with full diagnostics
|
- If restarts > 5 in last hour → alert owner with full diagnostics
|
||||||
4. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken
|
4. **Log results** — Append to `SyslogSolution/health-logs/pm2/{timestamp}.md` in Gitea (not knowledge graph — hard rule)
|
||||||
5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
|
5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
|
||||||
6. **Wait 5 min** → repeat from step 1
|
6. **Wait 5 min** → repeat from step 1
|
||||||
|
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ Changelog:
|
|||||||
v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
|
v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
|
||||||
vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
|
vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
|
||||||
name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
|
name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
|
||||||
Fleet roster: tanko (.122), mumuni (.123), koby (.129), koonimo (.114),
|
Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114),
|
||||||
abiba (.24).
|
abiba (.24).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|||||||
Executable
+91
@@ -0,0 +1,91 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# swap-gpu-dense-model.sh — Swap RTX 3090 from qwen3.6-27B-code to SmartCode-Fable-5
|
||||||
|
# Run when download completes: ssh root@192.168.68.8 'bash -s' < this script
|
||||||
|
#
|
||||||
|
# Usage: bash swap-gpu-dense-model.sh
|
||||||
|
# Requires: new model at /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
MODEL_PATH="/home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf"
|
||||||
|
OLD_WRAPPER="/home/llmuser/llama-wrapper.sh"
|
||||||
|
|
||||||
|
echo "═══ Swapping gpu-dense to SmartCode-Fable-5 ═══"
|
||||||
|
|
||||||
|
# 1. Verify model file
|
||||||
|
if [ ! -f "$MODEL_PATH" ]; then
|
||||||
|
echo "❌ Model not found at $MODEL_PATH"
|
||||||
|
echo " Download: curl -L -o $MODEL_PATH <huggingface-url>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
MODEL_SIZE=$(ls -lh "$MODEL_PATH" | awk '{print $5}')
|
||||||
|
echo "✅ Model found: $MODEL_SIZE"
|
||||||
|
|
||||||
|
# 2. Create new wrapper script for SmartCode-Fable-5
|
||||||
|
cat > /home/llmuser/llama-fable-wrapper.sh << 'WRAPPER'
|
||||||
|
#!/bin/bash
|
||||||
|
# SmartCode-Fable-5 llama-server wrapper for RTX 3090
|
||||||
|
# Sampler settings from model card: temp 0.9, top-p 0.95, top-k 60, repeat-penalty off
|
||||||
|
|
||||||
|
PORT=8080
|
||||||
|
GHOST_PID=$(ss -tlnp 2>/dev/null | grep -Po ":${PORT}\s+.*pid=\K[0-9]+" | head -1)
|
||||||
|
if [ -n "$GHOST_PID" ] && [ "$GHOST_PID" != "$$" ]; then
|
||||||
|
echo "[wrapper] Port $PORT occupied by ghost pid $GHOST_PID — cleaning up" >&2
|
||||||
|
kill -9 "$GHOST_PID" 2>/dev/null
|
||||||
|
sleep 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
exec /usr/local/bin/llama-server \
|
||||||
|
--model /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf \
|
||||||
|
--ctx-size 131072 \
|
||||||
|
--cache-type-k q4_0 \
|
||||||
|
--cache-type-v q4_0 \
|
||||||
|
--flash-attn 1 \
|
||||||
|
--cont-batching \
|
||||||
|
--parallel 1 \
|
||||||
|
--batch-size 2048 \
|
||||||
|
--ubatch-size 1024 \
|
||||||
|
--n-gpu-layers 99 \
|
||||||
|
--temp 0.9 \
|
||||||
|
--top-p 0.95 \
|
||||||
|
--top-k 60 \
|
||||||
|
--min-p 0.0 \
|
||||||
|
--repeat-penalty 1.0 \
|
||||||
|
--api-key not-needed \
|
||||||
|
--port 8080 \
|
||||||
|
--host 0.0.0.0
|
||||||
|
WRAPPER
|
||||||
|
|
||||||
|
chmod 755 /home/llmuser/llama-fable-wrapper.sh
|
||||||
|
echo "✅ Created /home/llmuser/llama-fable-wrapper.sh"
|
||||||
|
|
||||||
|
# 3. Update systemd service to use new wrapper
|
||||||
|
echo "📝 Updating systemd service..."
|
||||||
|
sed -i 's|ExecStart=/home/llmuser/llama-wrapper.sh|ExecStart=/home/llmuser/llama-fable-wrapper.sh|' /etc/systemd/system/llama-server.service
|
||||||
|
systemctl daemon-reload
|
||||||
|
|
||||||
|
# 4. Stop old server, start new
|
||||||
|
echo "🔄 Restarting llama-server..."
|
||||||
|
systemctl stop llama-server
|
||||||
|
sleep 3
|
||||||
|
systemctl start llama-server
|
||||||
|
sleep 8
|
||||||
|
|
||||||
|
# 5. Verify
|
||||||
|
echo ""
|
||||||
|
echo "═══ Verification ═══"
|
||||||
|
systemctl is-active llama-server
|
||||||
|
echo ""
|
||||||
|
echo "Port 8080:"
|
||||||
|
ss -tlnp 2>/dev/null | grep ":8080" | head -1
|
||||||
|
echo ""
|
||||||
|
echo "GPU VRAM:"
|
||||||
|
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader 2>/dev/null
|
||||||
|
echo ""
|
||||||
|
echo "=== Health check ==="
|
||||||
|
curl -s --max-time 5 http://localhost:8080/health 2>/dev/null
|
||||||
|
echo ""
|
||||||
|
echo ""
|
||||||
|
echo "✅ Swap complete. Test via LiteLLM:"
|
||||||
|
echo " curl -s http://192.168.68.116/v1/chat/completions -H 'Authorization: Bearer <key>' -H 'Content-Type: application/json' -d '{\"model\":\"gpu-dense\",\"messages\":[{\"role\":\"user\",\"content\":\"write hello world in python\"}],\"max_tokens\":100}'"
|
||||||
@@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
|
|||||||
## Requires
|
## Requires
|
||||||
|
|
||||||
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
||||||
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
|
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.24, inside Abiba CT100 on hwepve), and Agent Zero Docker host (192.168.68.14)
|
||||||
- **PM2** on localhost for pi process management
|
- **PM2** on localhost for pi process management
|
||||||
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
||||||
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
||||||
@@ -182,13 +182,13 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
|
|||||||
| `last_error` set | Log and monitor |
|
| `last_error` set | Log and monitor |
|
||||||
| Crash loop >10/h | Alert user |
|
| Crash loop >10/h | Alert user |
|
||||||
|
|
||||||
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
|
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .24)
|
||||||
|
|
||||||
**B1: Gateway State**
|
**B1: Gateway State**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
|
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
|
||||||
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
|
ssh root@192.168.68.24 "cat ~/.hermes/gateway_state.json" # Mumuni inside Abiba CT100
|
||||||
```
|
```
|
||||||
|
|
||||||
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
|
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
|
||||||
|
|||||||
@@ -65,7 +65,7 @@ triggers:
|
|||||||
|------|----|------|---------|
|
|------|----|------|---------|
|
||||||
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
|
||||||
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
|
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
|
||||||
| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
|
| Mumuni | 192.168.68.24 (CT100 abiba) | root | `hermes gateway restart` |
|
||||||
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
|
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
|
||||||
|
|
||||||
## Debounce
|
## Debounce
|
||||||
|
|||||||
Reference in New Issue
Block a user