Compare commits

..
Author SHA1 Message Date
root c243fcddc3 no-mistakes(document): docs: fix stale SearXNG endpoint in monitoring checks
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-08-01 15:19:15 +00:00
root 50114f32c0 fix: correct SearXNG endpoint from dead storepve hostname to live VM109 IP (192.168.68.7:8888)
storepve (192.168.68.6) has no service on :8888. SearXNG runs on VM 109
(docker-vm) at 192.168.68.7:8888 (verified HTTP 200). Aligns with every other
contract reference. Missed in PR #39.
2026-08-01 14:56:45 +00:00
jerome 4dc59633b3 Merge pull request 'fix(agent-health): vault token fallback + strix-server service name' (#39) from fix/agent-health-check into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
Reviewed-on: #39
2026-08-01 14:20:02 +00:00
root b3193e5e1b fix(agent-health): 14 stale-reference fixes (Mumuni CT100/.24, Koby .129, Koonimo CT113, wrapper paths)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-01 14:10:52 +00:00
root f1490b656e no-mistakes(document): docs: fix stale ornith refs to strix-server/strix-moe
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-08-01 13:35:21 +00:00
root b56501a1cf no-mistakes(review): Guard infisical-token fallback read against OSError crashes 2026-08-01 13:27:41 +00:00
root 1921937bee fix(agent-health): vault token fallback + strix-server service name 2026-08-01 13:24:38 +00:00
14 changed files with 31 additions and 21 deletions
+1 -1
View File
@@ -584,7 +584,7 @@ contracts:
verify: curl -sf https://git.sysloggh.net/api/v1/version
expect: 200 OK
- check: SearXNG reachable
verify: curl -sf http://192.168.68.17:8080
verify: curl -sf http://192.168.68.7:8888
expect: 200 OK
artifact: infrastructure health report
receipt:
+1 -1
View File
@@ -356,7 +356,7 @@ Postconditions to verify:
},
{
"check": "SearXNG reachable",
"verify": "curl -sf http://192.168.68.17:8080",
"verify": "curl -sf http://192.168.68.7:8888",
"expect": "200 OK"
}
]
+1 -1
View File
@@ -13,7 +13,7 @@ the description:
1. **What system does this contract touch?** Name the hosts, CTs, containers,
and services explicitly. "The inference fleet" is vague. "GPU .8 (RTX 3090,
qwen), .110 (RTX 5070, gemma), .15 (Strix Halo, ornith), and LiteLLM on CT
qwen), .110 (RTX 5070, gemma), .15 (Strix Halo, strix-moe), and LiteLLM on CT
116" is specific.
2. **Who runs this contract, and when?** State the agent, the trigger (cron,
+2 -2
View File
@@ -25,8 +25,8 @@ done
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|---------------|------------|----------|
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
| Mumuni | 100 | hwepve | .24 | `mumuni` | Infisical vault | Hermes |
| Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
+3 -3
View File
@@ -32,10 +32,10 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Agent | Key Alias | Host | SSH | Sub-Agents |
|-------|-----------|------|-----|-----------|
| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
| Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
| Mumuni | `mumuni` | 192.168.68.24 | root@.24 | 6 profiles ✱ |
| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
| Koonimo | `koonimo` | CT 113 (baggy) | SSH root | — |
| Kagenz0 | `kagenz0-*` | ? | Zulip | — |
> CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
@@ -48,7 +48,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Component | Endpoint | Purpose |
|---|---|---|
| Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
| SearXNG | `http://storepve:8888` | Privacy-respecting web search |
| SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search |
| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
| RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
+1 -1
View File
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------|
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
| Mumuni | CT100 | — | 192.168.68.24 | /root/.hermes | root |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
+1 -1
View File
@@ -2,7 +2,7 @@
kind: function
name: hermes-zulip-restore
description: >
Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
Restores Zulip connectivity for any Hermes agent (Mumuni CT100, Tanko CT112,
Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
path, verifies env credentials, restarts the gateway, and confirms Zulip
connects. Run this whenever a Hermes agent stops responding on Zulip or after
+1 -1
View File
@@ -87,7 +87,7 @@ call apply-liteLLM-routing
call apply-agent-compression
agent: mumuni
host: 192.168.68.123
host: 192.168.68.24
config_path: /root/.hermes/config.yaml
-- Phase 4: Enable llama.cpp prompt caching on GPU hosts
+1 -1
View File
@@ -90,7 +90,7 @@ description: >
### Reachability Matrix
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.24) | Baggy (.114) |
|-----------|---------|----------------|--------|-------------|---------------|----------------|
| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
+1 -1
View File
@@ -140,7 +140,7 @@ After ALL updates (apt + images + restarts), verify every critical service is ba
| Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
| Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
| PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
| Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
| Hermes gateways | SSH to Mumuni CT 100, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
+2 -2
View File
@@ -60,7 +60,7 @@ Before ANY update wave:
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 114 (mumuni, hwepve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| CT 100 (mumuni/abiba, hwepve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
@@ -160,7 +160,7 @@ Before Wave 1, snapshot these files:
/opt/home_stack/docker-compose.yml (VM 109 .7)
/opt/audiobookshelf/docker-compose.yml (VM 109 .7)
/root/.pi/agent/extensions/config.yaml (CT 100 .24)
/etc/systemd/system/ornith-server.service (amdpve .15 — strix-moe)
/etc/systemd/system/strix-server.service (amdpve .15 — strix-moe)
/etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110)
# Hermes agent configs (key enforcement — 2026-07-10)
/root/.hermes/config.yaml (Mumuni CT 114, Tanko CT 112, etc.)
+11 -1
View File
@@ -49,7 +49,7 @@ AGENTS = {
GPU_HOSTS = {
"gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-server"},
"gpu-rtx5070 (.110)": {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
"gpu-strixhalo (.15)": {"host": "192.168.68.15", "port": 8080, "service": "ornith-server"},
"gpu-strixhalo (.15)": {"host": "192.168.68.15", "port": 8080, "service": "strix-server"},
}
FAIL = []
@@ -57,6 +57,16 @@ FAIL = []
INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
# Fallback: if no env token, read the shared vault token file
if not INFISICAL_TOKEN:
_token_path = os.path.expanduser("~/.infisical-token")
if os.path.isfile(_token_path):
try:
with open(_token_path) as _f:
INFISICAL_TOKEN = _f.read().strip()
except (OSError, UnicodeDecodeError):
pass
# ── Helpers ──────────────────────────────────────────────────────────
def ssh(host, cmd, user="root"):
+4 -4
View File
@@ -313,8 +313,8 @@ def collect():
"updated_at": tanko_data.get("updated_at"),
}
# Mumuni (CT 114, IP 192.168.68.123)
mumuni_state = ssh("192.168.68.123", "cat ~/.hermes/gateway_state.json 2>/dev/null")
# Mumuni (CT 100, IP 192.168.68.24)
mumuni_state = ssh("192.168.68.24", "cat ~/.hermes/gateway_state.json 2>/dev/null")
mumuni_data = {}
try:
mumuni_data = json.loads(mumuni_state) if mumuni_state else {}
@@ -322,7 +322,7 @@ def collect():
mumuni_data = {}
mumuni_platforms = mumuni_data.get("platforms", {})
report["agents"]["mumuni"] = {
"platform": "hermes", "ct": 114, "ip": "192.168.68.123",
"platform": "hermes", "ct": 114, "ip": "192.168.68.24",
"gateway_state": mumuni_data.get("gateway_state", "unknown"),
"telegram_state": mumuni_platforms.get("telegram", {}).get("state", "unknown"),
"zulip_state": mumuni_platforms.get("zulip", {}).get("state", "not_installed"),
@@ -330,7 +330,7 @@ def collect():
"hermes_version": "",
}
# Get Hermes version
ver = ssh("192.168.68.123", "hermes --version 2>/dev/null | head -1")
ver = ssh("192.168.68.24", "hermes --version 2>/dev/null | head -1")
if ver:
report["agents"]["mumuni"]["hermes_version"] = ver.split("·")[0].replace("Hermes Agent ","").strip()
+1 -1
View File
@@ -100,7 +100,7 @@ else
fi
# ── Platform B: Hermes (Mumuni) ──
MUMUNI_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.123 \
MUMUNI_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.24 \
"cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}")
MUMUNI_ZULIP=$(echo "$MUMUNI_STATE" | python3 -c "
import sys,json