Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b799d46596 | ||
|
|
1d4f6c8ebb | ||
|
|
38a32f8b32 | ||
|
|
96b0caa0b3 | ||
|
|
c13a15acad |
@@ -96,7 +96,7 @@ work immediately after restart.
|
|||||||
model:
|
model:
|
||||||
default: <agent_model> # e.g., strix-moe, qwen3.6-27B-code, syslog-auto
|
default: <agent_model> # e.g., strix-moe, qwen3.6-27B-code, syslog-auto
|
||||||
provider: harness
|
provider: harness
|
||||||
base_url: http://192.168.68.116/v1
|
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated path; /v1 also OK
|
||||||
api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper
|
api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper
|
||||||
max_tokens: 4096 # ⚠️ CRITICAL: Prevents unbounded generation
|
max_tokens: 4096 # ⚠️ CRITICAL: Prevents unbounded generation
|
||||||
context_length: 131072 # For syslog-auto (all GPUs at 128K for stability).
|
context_length: 131072 # For syslog-auto (all GPUs at 128K for stability).
|
||||||
@@ -146,7 +146,7 @@ compression:
|
|||||||
# ─── Auxiliary Tasks (CONSISTENCY RULE) ───
|
# ─── Auxiliary Tasks (CONSISTENCY RULE) ───
|
||||||
# All auxiliary services MUST use identical model, base_url, and api_key_env:
|
# All auxiliary services MUST use identical model, base_url, and api_key_env:
|
||||||
# model: gpu-light # stable alias (NOT raw "gemma-4-12b")
|
# model: gpu-light # stable alias (NOT raw "gemma-4-12b")
|
||||||
# base_url: http://192.168.68.116/v1
|
# base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
|
||||||
# api_key_env: LITELLM_API_KEY
|
# api_key_env: LITELLM_API_KEY
|
||||||
# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
|
# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
|
||||||
# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
|
# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
|
||||||
@@ -157,20 +157,20 @@ auxiliary:
|
|||||||
vision:
|
vision:
|
||||||
provider: harness
|
provider: harness
|
||||||
model: gpu-light # stable alias for RTX 5070 (was raw gemma-4-12b)
|
model: gpu-light # stable alias for RTX 5070 (was raw gemma-4-12b)
|
||||||
base_url: http://192.168.68.116/v1
|
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
|
||||||
api_key_env: LITELLM_API_KEY
|
api_key_env: LITELLM_API_KEY
|
||||||
timeout: 60
|
timeout: 60
|
||||||
download_timeout: 30
|
download_timeout: 30
|
||||||
web_extract:
|
web_extract:
|
||||||
provider: harness
|
provider: harness
|
||||||
model: gpu-light # stable alias for RTX 5070
|
model: gpu-light # stable alias for RTX 5070
|
||||||
base_url: http://192.168.68.116/v1
|
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
|
||||||
api_key_env: LITELLM_API_KEY
|
api_key_env: LITELLM_API_KEY
|
||||||
timeout: 30
|
timeout: 30
|
||||||
compression:
|
compression:
|
||||||
provider: harness
|
provider: harness
|
||||||
model: syslog-auto # MUST match compression.model above. Stable alias for Strix Halo (weighted pool).
|
model: syslog-auto # MUST match compression.model above. Stable alias for Strix Halo (weighted pool).
|
||||||
base_url: http://192.168.68.116/v1 # Rule 5: /v1 NOT /litellm/v1
|
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated path; /v1 also OK
|
||||||
api_key_env: LITELLM_API_KEY
|
api_key_env: LITELLM_API_KEY
|
||||||
timeout: 300 # gpu-fleet: 300s for large-history summarization (was 60)
|
timeout: 300 # gpu-fleet: 300s for large-history summarization (was 60)
|
||||||
|
|
||||||
@@ -180,14 +180,14 @@ auxiliary:
|
|||||||
delegation:
|
delegation:
|
||||||
model: gpu-dense # stable alias for RTX 3090 (was raw qwen3.6-27B-code)
|
model: gpu-dense # stable alias for RTX 3090 (was raw qwen3.6-27B-code)
|
||||||
provider: harness
|
provider: harness
|
||||||
base_url: http://192.168.68.116/v1
|
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
|
||||||
api_key_env: LITELLM_API_KEY
|
api_key_env: LITELLM_API_KEY
|
||||||
|
|
||||||
# ─── Custom Provider ───
|
# ─── Custom Provider ───
|
||||||
custom_providers:
|
custom_providers:
|
||||||
- name: harness
|
- name: harness
|
||||||
model: syslog-auto # weighted pool (default)
|
model: syslog-auto # weighted pool (default)
|
||||||
base_url: http://192.168.68.116/v1
|
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
|
||||||
api_key_env: LITELLM_API_KEY
|
api_key_env: LITELLM_API_KEY
|
||||||
api_mode: chat_completions
|
api_mode: chat_completions
|
||||||
```
|
```
|
||||||
@@ -237,10 +237,13 @@ The following MUST be identical across ALL profiles:
|
|||||||
- When main config uses `api_key_env`, sub-agents automatically use it
|
- When main config uses `api_key_env`, sub-agents automatically use it
|
||||||
- This means key rotation only touches ONE vault secret (`LITELLM_API_KEY`)
|
- This means key rotation only touches ONE vault secret (`LITELLM_API_KEY`)
|
||||||
|
|
||||||
### Rule 5: Main Config Base URL
|
### Rule 5: Main Config Base URL (UPDATED 2026-08-09)
|
||||||
|- Use direct IP: `http://192.168.68.116/v1`
|
|- Use the authenticated LiteLLM path: `http://192.168.68.116/litellm/v1` (canonical, captain-approved migration)
|
||||||
|
|- Legacy `http://192.68.68.116/v1` also works — nginx fronts BOTH paths with key auth
|
||||||
|
(verified 2026-08-09: 401 without key, 200 with key, on both /v1 and /litellm/v1)
|
||||||
|
|- Both locations have `proxy_read_timeout 600s` (verified in harness-nginx nginx.conf) —
|
||||||
|
the old "60s timeout on /litellm/" claim was stale and is retracted
|
||||||
|- NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down
|
|- NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down
|
||||||
|- NOT the old path (`/litellm/v1`) — nginx now routes `/v1` directly
|
|
||||||
|
|
||||||
### Rule 6: max_tokens Is Required (Thermal Safety)
|
### Rule 6: max_tokens Is Required (Thermal Safety)
|
||||||
- **Every Hermes config MUST set `model.max_tokens: 4096`** — this is non-negotiable
|
- **Every Hermes config MUST set `model.max_tokens: 4096`** — this is non-negotiable
|
||||||
@@ -261,7 +264,7 @@ The following MUST be identical across ALL profiles:
|
|||||||
fall back to other GPUs if Strix gets hot. Both `compression.model` and `auxiliary.compression.model`
|
fall back to other GPUs if Strix gets hot. Both `compression.model` and `auxiliary.compression.model`
|
||||||
MUST be `syslog-auto`.
|
MUST be `syslog-auto`.
|
||||||
- All auxiliary services MUST use identical routing:
|
- All auxiliary services MUST use identical routing:
|
||||||
- `base_url: http://192.168.68.116/v1` (Rule 5: `/v1`, NOT `/litellm/v1`)
|
- `base_url: http://192.168.68.116/litellm/v1` (Rule 5, 2026-08-09: canonical authenticated; `/v1` also OK)
|
||||||
- `api_key_env: LITELLM_API_KEY`
|
- `api_key_env: LITELLM_API_KEY`
|
||||||
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
|
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
|
||||||
- **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
|
- **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
|
||||||
@@ -321,10 +324,13 @@ verify ALL FOUR of these against the live config. They are the only root causes
|
|||||||
1. **max_context_window correct?** — BOTH `compression.max_context_window` AND `context.max_context_window`
|
1. **max_context_window correct?** — BOTH `compression.max_context_window` AND `context.max_context_window`
|
||||||
MUST be `131072` (all GPUs are 128K). A value of `262144` causes instability near 100K and must NOT be used.
|
MUST be `131072` (all GPUs are 128K). A value of `262144` causes instability near 100K and must NOT be used.
|
||||||
~83K instead of ~170K. Check: `grep -n max_context_window ~/.hermes/config.yaml`
|
~83K instead of ~170K. Check: `grep -n max_context_window ~/.hermes/config.yaml`
|
||||||
2. **base_url uses /v1 NOT /litellm/v1?** — `custom_providers[0].base_url`, `delegation.base_url`,
|
2. **base_url uses authenticated path?** — `custom_providers[0].base_url`, `delegation.base_url`,
|
||||||
and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/v1` (Rule 5). nginx `/litellm/`
|
and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/litellm/v1` (Rule 5, canonical)
|
||||||
has a 60s default timeout → 504 on any inference >60s; `/v1/` has 600s.
|
or `http://192.168.68.116/v1` (legacy, still authenticated via nginx). BOTH verified 200 with
|
||||||
Check: `grep -n 'litellm/v1' ~/.hermes/config.yaml` (must return NOTHING)
|
key + 600s proxy_read_timeout on 2026-08-09. Never bare `:4000` direct.
|
||||||
|
Check: `grep -nE 'base_url: http://192.168.68.116(:4000)?/v1' ~/.hermes/config.yaml` — the
|
||||||
|
ONLY paths allowed are `/v1` or `/litellm/v1` (both via nginx :80).
|
||||||
|
`:4000` or missing `litellm/v1`/`v1` prefix = violation.
|
||||||
3. **LITELLM_API_KEY valid?** — The key must be a real LiteLLM key (`sk-` + 64 hex, 67 chars).
|
3. **LITELLM_API_KEY valid?** — The key must be a real LiteLLM key (`sk-` + 64 hex, 67 chars).
|
||||||
Malformed values (e.g. `sk-_SWAl_Vu_…`, 47 chars) return 401 → DeepSeek fallback.
|
Malformed values (e.g. `sk-_SWAl_Vu_…`, 47 chars) return 401 → DeepSeek fallback.
|
||||||
Verify: `curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $LITELLM_API_KEY" http://192.168.68.116/v1/models` (must be 200)
|
Verify: `curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $LITELLM_API_KEY" http://192.168.68.116/v1/models` (must be 200)
|
||||||
|
|||||||
@@ -34,11 +34,12 @@ Syslog is migrating away from **unauthenticated direct access** to the shared in
|
|||||||
|
|
||||||
| Path | Auth | Status |
|
| Path | Auth | Status |
|
||||||
|------|------|--------|
|
|------|------|--------|
|
||||||
| `http://192.168.68.116/v1` | None (direct) | ❌ **DEPRECATED** — being phased out |
|
| `http://192.168.68.116/v1` | Bearer `sk-*` key (nginx-fronted) | ✅ **VALID** — authenticated via nginx :80 (verified 2026-08-09: 401 without key, 200 with) |
|
||||||
| `http://192.168.68.116/litellm/v1/responses` | Bearer `sk-*` key | ✅ **CURRENT** — authenticated LiteLLM proxy |
|
| `http://192.168.68.116/litellm/v1` | Bearer `sk-*` key (nginx-fronted) | ✅ **CURRENT / CANONICAL** — captain-approved migration target; 600s proxy_read_timeout (verified) |
|
||||||
|
| `http://192.168.68.116:4000/v1` | Bearer `sk-*` key (direct container) | ❌ **FORBIDDEN** — bypasses nginx; port 4000 direct is not a config path |
|
||||||
|
|
||||||
All harness/litellm providers MUST use the authenticated `/litellm/v1/responses` path.
|
All harness/litellm providers MUST use an authenticated nginx-fronted path (`/litellm/v1` canonical, `/v1` legacy-valid).
|
||||||
Any `base_url` pointing to bare `/v1` on 192.168.68.116 is a **migration violation**.
|
Any `base_url` pointing at `:4000` or a bare IP without nginx is a **migration violation**.
|
||||||
|
|
||||||
### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
|
### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
|
||||||
|
|
||||||
@@ -191,7 +192,7 @@ litellm_settings:
|
|||||||
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
|
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
|
||||||
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
|
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
|
||||||
| Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
|
| Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
|
||||||
| Koonimo | 113 | .113 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
|
| Koonimo | 113 | .114 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-08-09 |
|
||||||
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
|
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
|
||||||
| Kagenz0 | 105 | .14 | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
|
| Kagenz0 | 105 | .14 | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
|
||||||
|
|
||||||
@@ -200,12 +201,12 @@ litellm_settings:
|
|||||||
|
|
||||||
### Migration Status: Authenticated Path
|
### Migration Status: Authenticated Path
|
||||||
|
|
||||||
| Agent | `/litellm/v1/responses` | Deprecated `/v1` | Status |
|
| Agent | `/litellm/v1` | Legacy `/v1` | Status |
|
||||||
|-------|--------------------------|--------------------|--------|
|
|-------|--------------|-------------|--------|
|
||||||
| Mumuni | ✅ 5 sections | 0 | ✅ Authenticated |
|
| Mumuni | ✅ harness provider | ✅ auxiliary on /v1 (valid) | ✅ Authenticated (verified 2026-08-09) |
|
||||||
| Tanko | ⚠️ No SSH access | — | Needs check |
|
| Tanko | ✅ 5 sections | 0 | ✅ Migrated 2026-08-08, keys 200 |
|
||||||
| Koby | ⚠️ No route to host | — | Needs check |
|
| Koby | ✅ custom provider (harness name) | — | ✅ External DeepSeek primary (intentional) |
|
||||||
| Koonimo | ⚠️ Connection timed out | — | Needs check |
|
| Koonimo | ✅ .114 (baggy) | — | ✅ 128K context applied 2026-08-09 |
|
||||||
|
|
||||||
### Systemd Service Pattern (2026-07-11 — vault migration)
|
### Systemd Service Pattern (2026-07-11 — vault migration)
|
||||||
|
|
||||||
|
|||||||
+16
-6
@@ -2,21 +2,31 @@
|
|||||||
kind: responsibility
|
kind: responsibility
|
||||||
name: pm2-self-heal
|
name: pm2-self-heal
|
||||||
description: >
|
description: >
|
||||||
Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and
|
Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner,
|
||||||
auto-restarts any that are stopped or errored. Logs every action to
|
spoton-service, zulip-watchdog) and auto-restarts any that are stopped or
|
||||||
the knowledge graph and alerts the owner via Zulip DM on failures.
|
errored. Logs every action to Gitea (SyslogSolution/health-logs — not
|
||||||
|
knowledge graph, hard rule) and alerts the owner via
|
||||||
|
Zulip DM on failures.
|
||||||
CRITICAL: Never restart abiba-zulip — it runs this contract.
|
CRITICAL: Never restart abiba-zulip — it runs this contract.
|
||||||
|
AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative):
|
||||||
|
gpu-monitor is systemd-managed (gpu-monitor.service) — NOT PM2;
|
||||||
|
gpu-watchdog decommissioned (function folded into gpu-monitor.service);
|
||||||
|
gitea-runner KEPT (online in PM2); abiba-zulip KEPT (online 4d+, the
|
||||||
|
2026-07-04 'removed/decommissioned' note was stale and is removed).
|
||||||
---
|
---
|
||||||
|
|
||||||
## Maintains
|
## Maintains
|
||||||
|
|
||||||
- abiba-telegram: { status: "online", uptime: string, restarts: number }
|
- abiba-telegram: { status: "online", uptime: string, restarts: number }
|
||||||
- gpu-watchdog: { status: "online", uptime: string, restarts: number }
|
- abiba-zulip: { status: "online", uptime: string, restarts: number }
|
||||||
- gpu-monitor: { status: "online", uptime: string, restarts: number }
|
|
||||||
- gitea-runner: { status: "online", uptime: string, restarts: number }
|
- gitea-runner: { status: "online", uptime: string, restarts: number }
|
||||||
|
- spoton-service: { status: "online", uptime: string, restarts: number }
|
||||||
|
- zulip-watchdog: { status: "online", uptime: string, restarts: number }
|
||||||
- last_check: timestamp
|
- last_check: timestamp
|
||||||
|
|
||||||
> **Note (2026-07-04):** `abiba-zulip` removed — Zulip extension decommissioned.
|
> **Note (2026-07-04, SUPERSEDED 2026-08-09):** `abiba-zulip` remains ONLINE and
|
||||||
|
> is monitored — the decommission note was stale (process re-added; do not treat
|
||||||
|
> it as removed).
|
||||||
|
|
||||||
## Continuity
|
## Continuity
|
||||||
|
|
||||||
|
|||||||
@@ -199,7 +199,14 @@ Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error`
|
|||||||
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
||||||
```
|
```
|
||||||
|
|
||||||
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (`pm2 restart mumuni-zulip`). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
|
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway per-agent (parameterized 2026-08-09, captain ruling):
|
||||||
|
|
||||||
|
| Agent | Restart command | Notes |
|
||||||
|
|-------|-----------------|-------|
|
||||||
|
| Mumuni (.24) | `pm2 restart abiba-zulip` | Hermes gateway runs under PM2 as `abiba-zulip` |
|
||||||
|
| Tanko (.122) | `bash /opt/hermes-zulip-plugin/run.sh` (or the agent's systemd/user unit) | Tanko does NOT use PM2 — never run `pm2 restart mumuni-zulip` for Tanko (process does not exist) |
|
||||||
|
|
||||||
|
Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
|
||||||
|
|
||||||
**B3: Heartbeat Verification**
|
**B3: Heartbeat Verification**
|
||||||
|
|
||||||
|
|||||||
@@ -75,9 +75,8 @@ Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
|
|||||||
|
|
||||||
## Reporting
|
## Reporting
|
||||||
|
|
||||||
Every cycle produces a knowledge graph node:
|
This contract is RETIRED — health-check logs are NOT knowledge graph content.
|
||||||
- Title: `[LEARN] zulip-self-heal: <timestamp>`
|
No graph nodes are created. Logs go to Gitea (SyslogSolution/health-logs).
|
||||||
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
|
|
||||||
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
|
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
|
||||||
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
|
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user