Compare commits

..
Author SHA1 Message Date
root 3e4d765497 V16: Rules 22-25 (trust calibration, error-feedback, attribution gate, backtest integrity) + land prior uncommitted Rule 15 two-phase exit rewrite 2026-07-24 16:46:15 -04:00
root 661c95e39d alpaca: add P-012 — internal metric integrity self-validates, self-heals, fail-closed
Detects snapshot poisoning by positive re-derivation (pre-inception, raw-balance
leak, adaptive sanity bound); quarantines (never deletes); tiered fail-closed
health (HEALTHY commits, DEGRADED report-only, CRITICAL holds) with age-adaptive
MIN_CLEAN; self-heals to HEALTHY via natural clean-snapshot refill. Backs the
2026-07-02 audit finding (36/37 snapshots poisoned, OSS steering live params off
a phantom drawdown).

Impl: scripts/metric_integrity.py, scripts/migrate_p010_metric_integrity.py,
compute_oss.py clean-filter+tier, apply_adaptations.py tier gate.
Tests: tests/test_p010_metric_integrity.py (10).
2026-07-24 16:46:15 -04:00
Hermes Agent ca4f18e539 alpaca: Full remediation (Gates 0-4 complete) — drawdown-first objective, P-001 through P-011 invariants 2026-07-24 16:46:15 -04:00
50 changed files with 2189 additions and 2093 deletions
-1
View File
@@ -1 +0,0 @@
__pycache__/
+2 -2
View File
@@ -67,10 +67,10 @@ Two incidents taught us this:
| Contract | Sensitivity | Who can change |
|----------|------------|----------------|
| `infrastructure-control.prose.md` | **CRITICAL** — topology source of truth | Abiba, Tanko (Tanko maintains its own CT row) |
| `infrastructure-control.prose.md` | **CRITICAL** — topology source of truth | Abiba only (after live verification) |
| `proxmox-monitor.prose.md` | **CRITICAL** — deployed monitoring | Abiba only |
| `hermes-config-template.prose.md` | **HIGH** — all agent configs | Abiba, Mumuni, Tanko |
| `zulip-health.prose.md` | **HIGH** — agent communication | Abiba, Mumuni, Tanko |
| `zulip-health.prose.md` | **HIGH** — agent communication | Abiba, Mumuni |
| Other contracts | Normal | Any registered agent |
| `scripts/*.sh` | **HIGH** — runtime scripts | Abiba only |
+1 -2
View File
@@ -20,8 +20,7 @@ An agent ran `pct set` without checking `pct config` first and changed the IP
to the wrong value, breaking Zulip. The staleness was harmless until acted on.
**Layer 3 enforcement — the `safe-mutate` wrapper** (deployed on all 5 agents:
Abiba, Tanko, Mumuni, Koby, Koonimo — Tanko runs on DSH/DeepSeek Harness since
2026-08-27 but safe-mutate enforcement still applies). ALL infrastructure mutations MUST go
Abiba, Tanko, Mumuni, Koby, Koonimo). ALL infrastructure mutations MUST go
through `safe-mutate`. Raw `sed -i`, `pct set`, `docker compose up
--force-recreate`, `kill`, `rm` on infrastructure outside `safe-mutate` is an
auditable protocol violation. The wrapper runs a verify command, optionally
-4
View File
@@ -1,6 +1,4 @@
---
report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
kind: function
name: abiba-zulip-restore
description: >
@@ -13,7 +11,6 @@ version: 1.0.0
status: active
runtime_contract: 2
---
---
# Abiba Zulip Restore — Resume pi Zulip Communication
@@ -307,7 +304,6 @@ module.exports = {
};
```
---
---
**Last verified good state**: 2026-07-13 — Extension v2 running via `pi --mode rpc`, health endpoint :9200 returning `{status:"ok",connected:true}`, queue a669f21e.
-186
View File
@@ -1,186 +0,0 @@
# Agent Zero Issue Fix Summary
**Date**: 2026-09-01
**Agent**: Agent Zero (Docker container on kagentz CT105)
**Issue**: AuthenticationError + Telegram conflicts
**Status**: ✅ RESOLVED
---
## Problems Identified
### 1. OpenRouter Authentication Error (CRITICAL)
```
litellm.exceptions.AuthenticationError: OpenrouterException -
{"error":{"message":"User not found.","code":401}}
```
**Root Cause**: The OpenRouter API key in `/a0/usr/.env` belonged to a different OpenRouter user.
**Old Key**: `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`
**New Key**: `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`
**New User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
### 2. Telegram Bot Conflict (CRITICAL)
```
TelegramConflictError: Conflict: terminated by other getUpdates request
```
**Root Cause**: Two Telegram bot instances were competing for the same token:
1. Agent Zero's built-in Telegram plugin (`/a0/usr/plugins/_telegram_integration/config.json`)
2. Standalone Telegram poller scripts (`/a0/usr/projects/telegram/telegram_bot.py`)
Both were using token `8476855065:***` in polling mode.
**Fix**: Disabled the built-in Telegram plugin by setting `"enabled": false` in the config.
### 3. MCP Service Connectivity Issues (SEVERE)
```
McpError: Timed out while waiting for response to ClientRequest. Waited 10.0 seconds.
```
**Root Cause**: The OpenRouter 401 errors caused the agent to fail, which in turn caused MCP services to timeout.
**Status**: ✅ RESOLVED with OpenRouter key fix.
---
## Fixes Applied
### Fix 1: Update OpenRouter Key
```bash
# Container .env update
sudo docker exec agent-zero bash -c '
sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=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|" /a0/usr/.env
'
```
**Verification**:
```bash
curl -s https://openrouter.ai/api/v1/auth/key \
-H "Authorization: Bearer sk-or-v1-0af3f3..." | python3 -m json.tool
```
Result: HTTP 200, user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`, not free tier.
### Fix 2: Disable Telegram Plugin
```bash
sudo docker exec agent-zero bash -c '
python3 << "PYEOF"
import json
config_path = "/a0/usr/plugins/_telegram_integration/config.json"
with open(config_path) as f:
config = json.load(f)
config["bots"][0]["enabled"] = False
with open(config_path, "w") as f:
json.dump(config, f, indent=2)
print("✓ Disabled telegram plugin @kagentz_bot")
PYEOF
'
```
### Fix 3: Restart Agent Zero UI
```bash
sudo docker exec agent-zero supervisorctl restart run_ui
```
**Result**: Process restarted (PID 3320), services running.
### Fix 4: Full Container Restart (Required)
```bash
sudo docker restart agent-zero
```
**Why needed**: The `run_ui` process was caching the old API key in memory. A full container restart was required to force Agent Zero to reload the `.env` file with the new OpenRouter key.
**Result**: All services restarted cleanly, no more 401 errors.
### Fix 5: Update Stale `.env.clobbered-by-new-image` (Critical)
**Root cause**: Agent Zero was loading the key from `/a0/usr/.env.clobbered-by-new-image` (line 28) instead of the main `/a0/usr/.env` (line 72). The clobbered file still had the old, stale key.
**Fix**:
```bash
KEY=$(grep "^API_KEY_OPENROUTER=" /a0/usr/.env | cut -d"=" -f2-)
sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=$KEY|" /a0/usr/.env.clobbered-by-new-image
```
**Lesson**: When updating Agent Zero's `.env`, check BOTH files:
- `/a0/usr/.env` (main)
- `/a0/usr/.env.clobbered-by-new-image` (backup, but loaded by Agent Zero)
The clobbered file is the one Agent Zero actually uses for LLM calls.
---
## Infrastructure Documentation
### New Contract Created
**File**: `/home/hermes/syslog/prose-contracts/agent-zero-openrouter-key.prose.md`
Contains:
- Key management procedures
- Rotation instructions
- Verification steps
- Current key inventory
- Related contracts
### Updated Contract
**File**: `/home/home/syslog/prose-contracts/litellm-api-keys.prose.md`
Added section:
- Agent Zero OpenRouter integration
- Key storage locations
- Model configuration
- Why not LiteLLM proxy
- Rotation procedure
---
## Current State
| Component | Status | Details |
|-----------|--------|---------|
| **OpenRouter Key** | ✅ Valid | `sk-or-v1-0af3f3…`, user verified |
| **Telegram Bot** | ✅ Resolved | Plugin disabled, conflicts cleared |
| **MCP Services** | ✅ Working | No timeouts after key fix |
| **Container** | ✅ Running | PID 3320, uptime 16+ hours |
| **Services** | ✅ All UP | run_ui, run_tunnel_api, run_searxng, run_cron, the_listener |
---
## Related Files
| Path | Purpose |
|------|---------|
| `/a0/usr/.env` | Container key storage |
| `/a0/usr/plugins/_telegram_integration/config.json` | Telegram plugin config |
| `/a0/usr/plugins/_model_config/presets.yaml` | Model selection (moonshotai/kimi-k3) |
| `/home/hermes/syslog/prose-contracts/agent-zero-openrouter-key.prose.md` | Key management contract |
| `/home/hermes/syslog/prose-contracts/litellm-api-keys.prose.md` | Fleet key inventory |
---
## Next Steps
1. **Sync key to Infisical vault** (optional, currently .env fallback only)
2. **Monitor usage** — Check OpenRouter dashboard for daily/weekly spend
3. **Consider LiteLLM migration** — Long-term: convert Agent Zero to use LiteLLM proxy for fleet-standard key management
4. **Set up vault sync** — Create machine identity in Infisical for automated key rotation
---
## Prevention
To prevent similar issues:
1. **Always verify API keys** against their providers before using
2. **Keep fleet-wide key inventory** updated in prose contracts
3. **Rotate keys on schedule** (quarterly hygiene, not on-demand only)
4. **Test key changes** in staging before production rollout
5. **Document key locations** in both code and prose contracts
---
**Verified by**: Mumuni 🦅
**Last updated**: 2026-09-01
**Session**: 1
-129
View File
@@ -1,129 +0,0 @@
---
kind: function
name: agent-zero-openrouter-key
description: >
Manages the OpenRouter API key for Agent Zero (Docker container on kagentz .14).
Agent Zero uses OpenRouter as its primary LLM provider for the moonshotai/kimi-k3
model. The key is stored in Infisical vault (project=agents, env=production) and
referenced from /a0/usr/.env in the container. Key must be rotated when the
OpenRouter user account changes or on quarterly hygiene. Last verified: 2026-09-01.
---
## Parameters
- action: "verify" | "rotate" | "update" | "list" — What to do (default: "verify")
- container_name: string — Docker container name (default: "agent-zero")
- host: string — Proxmox host running the container (default: "kagentz" at 192.168.68.14)
- env_path: string — Path to .env file in container (default: "/a0/usr/.env")
- vault_project: string — Infisical project slug (default: "agents")
- vault_env: string — Infisical environment (default: "production")
## Returns
- action: string — What was done
- key_status: string — "valid" | "invalid" | "not_found"
- key_prefix: string — First 10 chars of the key (for identification)
- user_id: string — OpenRouter user ID associated with the key
- vault_synced: boolean — Whether the key is in the Infisical vault
- container_updated: boolean — Whether the container's .env was updated
- verification: { status: string, detail: string } — Health check result
## Execution
### 1. Verify the key
1. **Extract key from container**
```bash
sudo docker exec agent-zero grep '^API_KEY_OPENROUTER' /a0/usr/.env | cut -d'=' -f2-
```
2. **Test against OpenRouter API**
```bash
curl -s https://openrouter.ai/api/v1/auth/key \
-H "Authorization: Bearer <key>" | python3 -m json.tool
```
Expected: HTTP 200, JSON with `data.label` and `data.is_free_tier`
3. **Check vault sync**
```bash
infisical secrets get OPENROUTER_API_KEY \
--token=$(cat ~/.infisical-token) \
--projectId=agents \
--env=production \
--domain=https://vault.sysloggh.net
```
4. **Return status**
- If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-0af", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
- If OpenRouter returns 401: `{ key_status: "invalid", detail: "User not found" }`
- If vault secret is missing: `{ vault_synced: false }`
### 2. Rotate the key
1. **Generate new key** in OpenRouter UI or via API
2. **Update container .env**
```bash
sudo docker exec agent-zero sed -i 's/^API_KEY_OPENROUTER=.*/API_KEY_OPENROUTER=<new_key>/' /a0/usr/.env
```
3. **Update Infisical vault**
```bash
infisical secrets set OPENROUTER_API_KEY=<new_key> \
--token=$(cat ~/.infisical-token) \
--projectId=agents \
--env=production \
--domain=https://vault.sysloggh.net
```
4. **Restart Agent Zero UI**
```bash
sudo docker exec agent-zero supervisorctl restart run_ui
```
5. **Verify** — Run "verify" action again
### 3. Update (key changed but no rotation)
1. **Update container .env** (same as rotate step 2)
2. **Sync vault** (same as rotate step 3)
3. **Restart run_ui** (same as rotate step 4)
## Current Key Inventory
| Field | Value |
|-------|-------|
| **Key Prefix** | `sk-or-v1-0af3f3` |
| **Full Key** | `«redacted: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»` (in vault + /a0/usr/.env) |
| **OpenRouter User** | `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT` |
| **Free Tier** | No |
| **Monthly Usage** | 0 (as of 2026-09-01) |
| **Last Verified** | 2026-09-01 |
| **Vault Sync** | ⏳ Pending (service token not on kagentz) |
## Key Rotation Log
| Date | Action | Notes |
|------|--------|-------|
| 2026-09-01 | fix-401 | Old key `sk-or-v1-036e5ca5…` returned 401 "User not found". Replaced with new key `sk-or-v1-0af3f3…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |
## Infrastructure References
- **Docker container**: `agent-zero` (image: `agent0ai/agent-zero:latest`)
- **Host**: kagentz (192.168.68.14, Proxmox LXC CT105)
- **Volume**: `/var/lib/docker/volumes/agent_zero/_data` → `/a0/usr`
- **Config path**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=…`)
- **Model preset**: "Cost Efficient" (uses `openrouter/moonshotai/kimi-k3`)
- **Model config**: `/a0/usr/plugins/_model_config/config.json`
## Verification Before Acting
**Key is a lead, not a fact.** Live OpenRouter accounts can change (user deletion,
plan change, key revocation). Before acting on this contract:
1. Verify the key against OpenRouter's `/auth/key` endpoint
2. Check the user ID matches the expected account
3. Confirm the model `moonshotai/kimi-k3` is available on that account's plan
4. Only then update the vault and container
## Related Contracts
- `litellm-api-keys.prose.md` — LiteLLM key management (Agent Zero does NOT use LiteLLM for OpenRouter)
- `infrastructure-control.prose.md` — Proxmox topology, container locations
- `gpu-fleet.prose.md` — Fleet-wide agent key inventory (add Agent Zero here)
-230
View File
@@ -1,230 +0,0 @@
#!/usr/bin/env python3
"""
Hermes Config Audit — validates a live config.yaml against the prose contract rules.
Usage:
python3 audit-hermes-config.py <config.yaml>
python3 audit-hermes-config.py /root/.hermes/config.yaml
Exit codes:
0 = all checks pass
1 = one or more contract violations found
This script encodes every rule from hermes-config-template.prose.md so config
changes can be verified before and after application. It is the single automated
enforcement layer for the prose contract.
Contract: /root/prose-contracts/hermes-config-template.prose.md
"""
import sys
import yaml
VIOLATIONS = []
WARNINGS = []
PASSES = []
def check(condition, rule, message):
if condition:
PASSES.append(f"[{rule}] {message}")
else:
VIOLATIONS.append(f"[{rule}] {message}")
def warn(rule, message):
WARNINGS.append(f"[{rule}] {message}")
def audit(path):
with open(path) as f:
cfg = yaml.safe_load(f)
model = cfg.get("model", {})
fb = cfg.get("fallback_providers", {})
comp = cfg.get("compression", {})
aux = cfg.get("auxiliary", {})
deleg = cfg.get("delegation", {})
cps = cfg.get("custom_providers", [])
cp = cps[0] if cps else {}
# --- Rule 3: API Keys via Environment ---
check(
model.get("api_key") in ("", None),
"Rule 3",
f"model.api_key must be empty (got {model.get('api_key')!r}) — keys via env var, not hardcoded",
)
check(
model.get("api_key_env") == "LITELLM_API_KEY",
"Rule 3",
f"model.api_key_env must be LITELLM_API_KEY (got {model.get('api_key_env')!r})",
)
# --- Rule 5: Main Config Base URL ---
expected_base = "http://192.168.68.116/v1"
check(
model.get("base_url") == expected_base,
"Rule 5",
f"model.base_url must be {expected_base} (got {model.get('base_url')!r}) — /v1 not /litellm/v1",
)
# --- Rule 6: max_tokens Is Required ---
check(
isinstance(model.get("max_tokens"), int) and model.get("max_tokens") <= 8192,
"Rule 6",
f"model.max_tokens must be set and <= 8192 (got {model.get('max_tokens')!r}) — thermal safety",
)
# --- Rule 7: Auxiliary Model Consistency ---
check(
comp.get("model") == "syslog-auto",
"Rule 7",
f"compression.model must be syslog-auto (got {comp.get('model')!r}) — auto-routing to prevent Strix Halo overload",
)
aux_comp = aux.get("compression", {})
check(
aux_comp.get("model") == "syslog-auto",
"Rule 7",
f"auxiliary.compression.model must be syslog-auto (got {aux_comp.get('model')!r}) — must match compression.model",
)
# --- Rule 8: GPU Workload Distribution ---
check(
aux.get("vision", {}).get("model") == "gpu-light",
"Rule 8",
f"auxiliary.vision.model must be gpu-light (got {aux.get('vision', {}).get('model')!r}) — RTX 5070 stable alias",
)
check(
aux.get("web_extract", {}).get("model") == "gpu-light",
"Rule 8",
f"auxiliary.web_extract.model must be gpu-light (got {aux.get('web_extract', {}).get('model')!r}) — RTX 5070 stable alias",
)
# --- Rule 9: Compression Threshold ---
check(
comp.get("threshold") == 0.65,
"Rule 9",
f"compression.threshold must be 0.65 for 128K models (got {comp.get('threshold')!r})",
)
check(
comp.get("max_context_window") == 131072,
"Rule 9",
f"compression.max_context_window must be 131072 (got {comp.get('max_context_window')!r}) — matches 128K GPU capacity",
)
# --- Rule 10: Default Model Must Be syslog-auto ---
check(
model.get("default") == "syslog-auto",
"Rule 10",
f"model.default must be syslog-auto (got {model.get('default')!r}) — auto-routing default",
)
# --- Rule 14: Provider Name Must Match custom_providers Name ---
check(
model.get("provider") == "harness",
"Rule 14",
f"model.provider must be 'harness' (got {model.get('provider')!r}) — NOT 'custom'. "
f"provider: custom causes generic resolution path that ignores key_env → 'no-key-required' → 401",
)
check(
comp.get("provider") == "harness",
"Rule 14",
f"compression.provider must be 'harness' (got {comp.get('provider')!r})",
)
for aux_name in ("vision", "web_extract", "compression"):
aux_provider = aux.get(aux_name, {}).get("provider")
check(
aux_provider == "harness",
"Rule 14",
f"auxiliary.{aux_name}.provider must be 'harness' (got {aux_provider!r})",
)
check(
deleg.get("provider") == "harness",
"Rule 14",
f"delegation.provider must be 'harness' (got {deleg.get('provider')!r})",
)
check(
fb.get("provider") == "deepseek",
"Rule 14",
f"fallback_providers.provider must be 'deepseek' (got {fb.get('provider')!r}) — "
f"true fallback diversity, not same endpoint as primary",
)
check(
fb.get("model") == "deepseek-v4-flash",
"Rule 14",
f"fallback_providers.model must be 'deepseek-v4-flash' (got {fb.get('model')!r})",
)
check(
fb.get("api_key_env") == "DEEPSEEK_API_KEY",
"Rule 14",
f"fallback_providers.api_key_env must be DEEPSEEK_API_KEY (got {fb.get('api_key_env')!r})",
)
# --- custom_providers sanity ---
check(
cp.get("name") == "harness",
"custom_providers",
f"custom_providers[0].name must be 'harness' (got {cp.get('name')!r})",
)
check(
cp.get("key_env") == "LITELLM_API_KEY" or cp.get("api_key_env") == "LITELLM_API_KEY",
"custom_providers",
f"custom_providers[0] must have key_env or api_key_env = LITELLM_API_KEY "
f"(got key_env={cp.get('key_env')!r}, api_key_env={cp.get('api_key_env')!r})",
)
check(
cp.get("base_url", "").endswith("/v1"),
"custom_providers",
f"custom_providers[0].base_url must end with /v1 (got {cp.get('base_url')!r})",
)
# --- No raw model names (Rule 7/8 spirit) ---
raw_names = {"gemma-4-12b", "qwen3.6-27B-code", "qwen3.6-35B-udq4", "ornith-1.0-35b"}
for section_path, section_dict in [
("model", model), ("compression", comp),
("auxiliary.vision", aux.get("vision", {})),
("auxiliary.web_extract", aux.get("web_extract", {})),
("auxiliary.compression", aux.get("compression", {})),
("delegation", deleg),
]:
m = section_dict.get("model", "")
if m in raw_names:
warn(
"Rule 7/8",
f"{section_path}.model = {m!r} — raw model name, use stable alias instead "
f"(gpu-light, gpu-dense, strix-moe, syslog-auto)",
)
# --- Report ---
print(f"{'=' * 60}")
print(f"Hermes Config Audit: {path}")
print(f"{'=' * 60}")
print(f"\n✅ PASSED ({len(PASSES)}):")
for p in PASSES:
print(f" ✅ {p}")
if WARNINGS:
print(f"\n⚠️ WARNINGS ({len(WARNINGS)}):")
for w in WARNINGS:
print(f" ⚠️ {w}")
if VIOLATIONS:
print(f"\n❌ VIOLATIONS ({len(VIOLATIONS)}):")
for v in VIOLATIONS:
print(f" ❌ {v}")
print(f"\n{'=' * 60}")
print(f"RESULT: FAIL — {len(VIOLATIONS)} violation(s) must be fixed")
print(f"{'=' * 60}")
return 1
else:
print(f"\n{'=' * 60}")
print(f"RESULT: PASS — all contract rules satisfied")
print(f"{'=' * 60}")
return 0
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python3 audit-hermes-config.py <config.yaml>")
sys.exit(2)
sys.exit(audit(sys.argv[1]))
+2 -3
View File
@@ -45,10 +45,9 @@ description: >
## Status
**Active** — for Hermes agents only. This plugin is NOT retired. It remains in service
for Hermes agents that connect to Zulip (Mumuni, Koby, Koonimo). The pi
for any Hermes agent that connects to Zulip (Mumuni, Tanko, Koby, Koonimo). The pi
Zulip extension was decommissioned 2026-07-04 but this contract targets the Hermes
plugin system, which is unaffected. **Tanko is excluded — it runs on DSH (DeepSeek
Harness) since 2026-08-27 and no longer uses the Hermes Zulip plugin.**
plugin system, which is unaffected.
## Parameters
+1 -71
View File
@@ -584,7 +584,7 @@ contracts:
verify: curl -sf https://git.sysloggh.net/api/v1/version
expect: 200 OK
- check: SearXNG reachable
verify: curl -sf http://192.168.68.7:8888
verify: curl -sf http://192.168.68.17:8080
expect: 200 OK
artifact: infrastructure health report
receipt:
@@ -1867,73 +1867,3 @@ contracts:
last_run: null
last_status: null
drift_alerts: []
# Koby Report-Only Registry (2026-08-17 — Captain)
# ⛔ KOBY IS NEVER REPAIRED — detect + report, never fix on .129
koby_report_only: true
koby_host: "CT 111 (tdunna)"
koby_ip: ".129"
koby_user: "Theo"
# Contracts that should be Koby-aware (detect only, no heal path)
koby_aware_contracts:
- name: pm2-self-heal
path: pm2-self-heal.prose.md
koby_action: skip_heal
koby_note: "Koby PM2 processes reported to Zulip, never auto-restarted on .129"
- name: zulip-health
path: zulip-health.prose.md
koby_action: skip_heal
koby_note: "Koby Zulip bridge issues reported to Zulip, never repaired on .129"
- name: hermes-zulip-restore
path: hermes-zulip-restore.prose.md
koby_action: skip_heal
koby_note: "Koby Zulip restoration skipped, only diagnostic alerts"
- name: abiba-zulip-restore
path: abiba-zulip-restore.prose.md
koby_action: skip_heal
koby_note: "Abiba-Zulip restoration not applicable to Koby"
- name: litellm-self-heal
path: litellm-self-heal.prose.md
koby_action: skip_heal
koby_note: "Koby LiteLLM issues reported, never fixed on .129"
- name: disk-gc-threat-response
path: disk-gc-threat-response.prose.md
koby_action: skip_heal
koby_note: "Koby disk GC threats reported, never executed on .129"
- name: memory-fixer
path: memory-fixer.prose.md
koby_action: skip_heal
koby_note: "Koby memory issues reported, never fixed on .129"
- name: memory-audit-maintenance
path: memory-audit-maintenance.prose.md
koby_action: skip_heal
koby_note: "Koby memory audits reported, never performed on .129"
- name: gpu-self-heal
path: gpu-self-heal.prose.md
koby_action: skip_heal
koby_note: "Koby GPU issues reported, never fixed on .129"
- name: gpu-monitor
path: gpu-monitor.prose.md
koby_action: skip_heal
koby_note: "Koby GPU monitoring reports only, never repairs on .129"
- name: agent-health-check
path: agent-health-check.prose.md
koby_action: skip_heal
koby_note: "Koby agent health checks reported, never repairs on .129"
# Scripts that should skip Koby
koby_aware_scripts:
- name: agent-health-check.py
path: scripts/agent-health-check.py
koby_action: skip_heal
koby_note: "Script should only run diagnostics on Koby, not repairs"
@@ -0,0 +1,391 @@
# CHECKPOINT — Alpaca Trading System Remediation Plan
**Status:** GATE 0 COMPLETE · **Created:** 2026-07-01 · **Resumed:** 2026-07-01T14:35Z
**Pick-up instruction for a new session:** Read this file top-to-bottom. Gate 0 is done; resume at Gate 1.
---
## ✅ GATE 0 — COMPLETE (2026-07-01T14:46Z)
### OSS Before/After
| Metric | Before (broken) | After (fixed) | Delta |
|--------|-----------------|---------------|-------|
| `oss` | 1.094 (above target!) | **-0.605** (below floor) | −1.699 |
| `oss_trend` | "improving" | **"degrading"** | ✓ fixed |
| `calmar` | 2.174 (sign destroyed) | **-2.174** | sign restored |
| `spy_30d_return` | 0.58 (hardcoded) | **-1.44** (live) | real benchmark |
| `rolling_alpha` | -1.01 (wrong) | **1.01** (portfolio beat SPY) | flipped |
| `portfolio_beta` | -0.74 (wrong) | **0.30** (correct) | sign fixed |
| `win_rate_adaptations` | 0.5 (dead code) | **removed** | dead code excised|
| `benchmark_error` | — | **null** | fetch succeeded |
### Changes Made
| Gate | File | Changes |
|------|------|---------|
| 0.1 | `scripts/compute_oss.py:65-66` | `abs(cagr)` → `cagr`; floor `0.0` → `-5.0` |
| 0.2 | `scripts/compute_oss.py:344-362` | Removed dead `_compute_adapt_win_rate()` + call site |
| 0.3 | `scripts/compute_oss.py:45-61` | Added `_fetch_spy_30d_return()` (yfinance via ^GSPC) |
| 0.3 | `scripts/compute_oss.py:293-301` | Replaced hardcoded 0.58 with live fetch; alpha=None on failure |
| 0.3 | `scripts/compute_oss.py:334` | `_compute_beta` now accepts live spy_return param |
| 0.3 | `scripts/compute_oss.py:328-329` | JSON output now includes `spy_30d_return` + `benchmark_error` |
| 0.4 | `tests/test_oss_monotonicity.py` | NEW — 3 property tests (Calmar sign, monotonicity, known values) |
| 0.4 | `tests/test_benchmark_integrity.py` | NEW — 3 tests (no 0.58 literal, fetch signature, alpha sign mock) |
| 0.4 | `~/.hermes/scripts/contract_invariant_watchdog.py` | Added 2 new test files to TEST_FILES list (now 7 files) |
### Verification
- `compute_oss.py` runs clean: OSS=-0.605, trend=degrading, live SPY=-1.44%
- `tests/`: 36 passed (30 original + 6 new)
- Watchdog: "✅ All contract invariants hold (7 test files passed)"
### Next: Gate 1 — Make the optimization loop load-bearing
---
## ✅ GATE 1 — COMPLETE (2026-07-01T15:00Z)
### Changes Made
| Sub-gate | File | Changes |
|----------|------|---------|
| 1.1 | `scripts/migrate_gate1_schema.py` | NEW — creates `adaptations` + `parameter_state` tables, seeds 5 params |
| 1.1 | `trade_journal.db` | `parameter_state` (5 params) + `adaptations` (0 rows, clean) |
| 1.2 | `scripts/apply_adaptations.py` | NEW — reads OSS+regime, proposes bounded changes, writes DB |
| 1.3 | `scripts/apply_adaptations.py:auto_unwind()` | Auto-reverts params after 3 consecutive non-positive forward_oss_delta |
| 1.4 | `scripts/compute_oss.py:W_ADAPT=0.10` | Re-weighted: W_CALMAR=0.35, W_WIN_RATE=0.10, W_ADAPT=0.10 |
| 1.4 | `scripts/compute_oss.py:_compute_adapt_win_rate()` | Queries `adaptations.result='improved'` (not dead `analysis_runs`) |
| 1.5 | `tests/test_adaptation_persistence.py` | NEW — 3 tests (seeded params, write+unwind, migration idempotent) |
| 1.5 | `~/.hermes/scripts/contract_invariant_watchdog.py` | Added test_adaptation_persistence.py (now 8 test files) |
### Adaptation Logic
| OSS Range | Trend | Regime | Action |
|-----------|-------|--------|--------|
| < 0.5 | degrading | bearish | **Tighten**: lower STOP_LOSS_PCT, raise CONFIDENCE_FLOOR, lower HEDGE_CAP |
| > 0.8 | improving | bullish | **Loosen**: raise STOP_LOSS_PCT, lower CONFIDENCE_FLOOR, raise HEDGE_CAP |
| Any | — | neutral | **Hold** — no change needed |
- Step size: 10% of (max−min) range
- Bounds enforced from `parameter_state`
- Auto-unwind after 3 consecutive non-positive forward_oss_delta
### Verification
- `apply_adaptations.py` runs clean: proposes STOP_LOSS_PCT adaptation in bearish regime
- Auto-unwind test: 3 failed adaptations → reversion to original value
- `compute_oss.py` shows `AdaptWR=0.5` (neutral default, wired to real data)
- **39 tests pass** (30 original + 6 Gate 0 + 3 Gate 1)
- **Watchdog: "✅ All contract invariants hold (8 test files passed)"**
- LLM cron `2ca03134ea62` is now reporter, not actor — calls `apply_adaptations.py`
### Next: Gate 2 — Make hedging capable
---
## 0. ORIENTATION (read first)
**Repo:** `/opt/alpaca-trading-agent` · **venv:** `.venv/bin/python`
**Contract (source of truth):** `/opt/openprose-contracts/cron-contracts/alpaca-trading-system.prose.md`
**Full audit narrative:** `/opt/openprose-contracts/cron-contracts/alpaca-adversarial-audit-2026-07-01.md`
**Watchdog:** `~/.hermes/scripts/contract_invariant_watchdog.py` (cron job `85522a081e55`, daily 08:00, no_agent, silent on pass)
**Adaptation cron (LLM-mode):** job `2ca03134ea62` (DeepSeek-v4-pro, pinned)
**DBs:**
- `trade_journal.db` — perf, orders, trades, `analysis_runs`, `config_store`. **This is the DB `compute_oss.py` reads** (`DB_PATH`, line 18).
- `consensus/signals.db` — `signals`, `consensus`.
**Governing design principle (Theo, non-negotiable):** the contract must be **declarative** — declare desired *outcome states* + expose *levers* + define *feedback signals*, IaC-style. NOT prescriptive step lists. Every fix below converts a frozen constant into a tuned lever or adds a missing feedback signal. Wherever you must hardcode, justify it as an irreducible exception.
**[fv] discipline for this work:** internal claims (code lines, DB columns) are verified by direct inspection (stronger than search — it's ground truth). External claims (financial definitions, SOTA findings) require independent `web_search` per claim. Label every assertion CONFIRMED / DISPUTED / UNVERIFIED. Add a property test that would have caught each defect before shipping the fix.
---
## 1. VERIFIED CURRENT STATE (the evidence base)
### Live OSS output (2026-07-01T14:19Z) — the smoking gun
```
oss=1.094 (target 1.0, floor 0.5) oss_trend="improving"
calmar=2.174 profit_factor=0.679 win_rate=0.5
total_return_pct=-0.43% sharpe=-1.38 realized_pnl_30d=-$1.67 over 24 trades
```
A losing, negative-Sharpe, PF<1 portfolio scores ABOVE its aspirational target and reads "improving."
**VERIFY:** `cd /opt/alpaca-trading-agent && .venv/bin/python scripts/compute_oss.py | python3 -m json.tool`
### Confirmed defects (all [fv] CONFIRMED by direct inspection)
| ID | Defect | Exact location | Evidence |
|----|--------|---------------|----------|
| D1 | Calmar uses `abs(cagr)` — sign destroyed | `scripts/compute_oss.py:65` | `compute_calmar(-2.35,-0.43,30)=2.174` vs `(+0.43)=2.28` |
| D1b | Calmar clamp floor `max(calmar,0.0)` erases negatives even if sign fixed | `scripts/compute_oss.py:66` | `return round(min(max(calmar,0.0),5.0),3)` |
| D2 | Benchmark hardcoded | `scripts/compute_oss.py:274` (`portfolio_return - 0.58`), `:319` (`spy_30d_return = 0.58`) | never fetched; alpha+beta are self-referential |
| D3 | Win-rate query references 2 non-existent columns | `scripts/compute_oss.py:352` (`analysis_type`), `:349` (`result`) | `PRAGMA table_info(analysis_runs)` has NEITHER → bare `except:pass` at :360 → returns 0.5 forever |
| D4 | Hedge structurally incapable | `.env.prod` + `consensus/ross/config.py:84-86` | `MAX_HEDGE_POSITIONS=1`, `HEDGE_POSITION_SIZE_PCT=0.01` → max ~1% of equity; contract Rule 8 assumes 25% (`PARKER_HEDGE_CAP_PCT=0.25`, config.py:113) |
| D5 | P-004/005/006 unimplemented | grep entire tree | `hedge_strategy`, `discovery_lens`, `source_provenance` = 0 hits in *.py, 0 DB columns |
| D6 | No adaptation persistence | DB introspection | tables `parameter_state`/`adaptations`/`regime_state` do not exist; `config_store` untouched since 2026-05-25 |
| D7 | Discovery is static | `config_store.tickers` | 7 fixed symbols, last write 2026-05-22; frank=1, janet=1 distinct tickers ever |
| D8 | Position size flat, `POSITION_SIZE_PCT` dead | `consensus/ross/sizing.py:7`, `config.py:138` | `calculate_position_size` returns `cfg.MAX_POSITION_NOTIONAL` which `= MIN_NOTIONAL` ($5); the 0.025 pct is never used |
| D9 | Watchdog covers only P-002/P-003 | `contract_invariant_watchdog.py:19-25` | 5 test files = dust filter + equity source + pipeline; P-004/5/6 never tested |
| D10 | "Consensus" is single-track | `consensus/sophie.py` + signals.db | all executed decisions score=1; universes disjoint by design |
### Confirmed ASSETS (things that already exist — reuse, don't rebuild)
| Asset | Location | Use for |
|-------|----------|---------|
| Live market fetch (VIX/SPX/NDX/BTC via yfinance) | `consensus/track_parker.py:76-101` (`_fetch_market_data`) | D2 fix — reuse `spx.history(period="1mo")` for real SPY 30d return |
| Bounded tunable levers (already documented w/ min-max) | `config.py:99-113`: STOP_LOSS_PCT[-0.03,-0.07], PROFIT_TAKE_PCT[0.03,0.10], STALE_EXIT_DAYS[3,7], CONFIDENCE_FLOOR[0.65,0.85], PARKER_HEDGE_CAP_PCT[0.10,0.30] | D6 — levers EXIST; they need a *writer*, not a redesign |
| Property-test harness (hypothesis, temp-DB, module-reload) | `tests/test_perf_equity_property_invariants.py` | template for all new invariant tests |
| Regime detection (VIX-trend + SPX-vs-SMA20) | `track_parker.py:152+` (`_detect_regime`) | D4 — regime signal exists to drive dynamic hedge sizing |
| slippage_bps / intended_price columns | broker_orders schema | P-008 cost-aware gate (data already captured, unused) |
| Risk-gate framework (Gates 0-7) | `consensus/ross/gates.py:14` (`check_risk_gates`) | insert cost-aware gate here |
---
## 2. EXTERNAL EVIDENCE (July 2026 SOTA) — [fv] CONFIRMED against sources
- **Calmar = signed CAGR ÷ |maxDD|** — negative return MUST yield negative Calmar. (Investopedia, Wikipedia, QuantifiedStrategies — CONFIRMED, ≥3 sources.) → D1/D1b are unambiguous bugs.
- **FINSABER (arXiv:2505.07078v5, KDD'26):** LLM edge deteriorates over long horizons/broad universe; agents "overly conservative in bull, overly aggressive in bear." Prescription: **prioritize trend detection + regime-aware risk controls over framework complexity.** CONFIRMED (abstract).
- **StockBench (arXiv:2510.02209):** most LLM agents don't beat buy&hold over 4mo, but ALL limit drawdown (-11/-14% vs -15.2%). → realistic capturable edge = **drawdown control, not alpha.** CONFIRMED.
- **Agent Market Arena (arXiv:2510.11695):** performance driven by **architecture (memory/risk/sizing), not LLM choice.** CONFIRMED.
- **CLQT (arXiv:2606.29771):** return leaderboards mislead; apparent alpha dissolves once look-ahead controlled → "mostly passive factor exposure." Evaluate capability, not single score. CONFIRMED. → OSS is exactly the single-number leaderboard being warned against.
- **Signal half-life/capacity (OpenAlgo):** turnover set by half-life; every trade pays half-spread+impact; net-of-cost is the only honest test. CONFIRMED.
- **Leveraged inverse decay (StockTitan 2026):** daily-reset -3x (SQQQ/SPXU) decays even when directionally right → prefer 1x inverse (SH/PSQ) for multi-day holds. CONFIRMED. Contract Rule 18 already says this; `MAX_HEDGE_HOLD_DAYS=3` + HEDGE_TICKERS still lists SQQQ/SPXU fight it.
**Synthesis implication:** the honest declared outcome state is **"match benchmark net of costs while bounding drawdown,"** NOT "maximize growth." Re-weight the objective accordingly (P-011).
---
## 3. THE PLAN (dependency-ordered gates; LLM-paced, no human timelines)
> Each task: **DO** (exact edit) → **VERIFY** (command + expected result) → **TEST** (property/regression test to add). No task is "done" until its VERIFY passes and its TEST is committed.
### GATE 0 — Restore metric truth (BLOCKING; everything downstream is meaningless until this lands)
**0.1 — Fix Calmar sign + clamp**
- DO: `scripts/compute_oss.py:65` → `calmar = cagr / (abs(max_drawdown_pct) / 100.0)` (drop `abs(` on cagr).
- DO: `:66` → allow negatives: `return round(min(max(calmar, -5.0), 5.0), 3)` (floor -5.0, not 0.0).
- CONSIDER: the OSS aggregate weight math (`:278-280`) assumes non-negative components — after this, re-derive so a negative Calmar drags OSS below floor. Verify OSS can now go below 0.5 on a losing month.
- VERIFY: `.venv/bin/python -c "import sys;sys.path.insert(0,'scripts');from compute_oss import compute_calmar as c;print(c(-2.35,-0.43,30), c(-2.35,0.43,30))"` → first value must be **negative**, second positive.
- VERIFY: `.venv/bin/python scripts/compute_oss.py` → with current -0.43% return, `oss` must now be **< 0.5** and `oss_trend` must NOT be "improving".
- TEST: add `tests/test_oss_monotonicity.py` (hypothesis): `∀ dd, days, r1<0<r2 ⇒ compute_calmar(dd,r1,days) < compute_calmar(dd,r2,days)` AND `compute_calmar(dd, r<0, days) < 0`.
**0.2 — Fix or remove adaptation win-rate**
- Root cause: query needs `analysis_type` AND `result` columns that don't exist in `analysis_runs`.
- OPTION A (preferred, aligns with Gate 1): create the `adaptations` table (Gate 1.1) and repoint this query at it.
- OPTION B (interim): remove the `W_ADAPT * 0.5` term from OSS and renormalize weights, so OSS isn't diluted by a frozen constant. Document as temporary.
- VERIFY: grep confirms no silent `except: pass` returns a magic 0.5 that feeds OSS.
- TEST: assert `_compute_adapt_win_rate` raises or returns None (not 0.5) when the backing table/columns are absent — no silent defaults.
**0.3 — Live benchmark (kill the 0.58 literal)**
- DO: add `_fetch_spy_30d_return()` reusing Parker's pattern (`import yfinance as yf; yf.Ticker("^GSPC").history(period="1mo")`; 30d pct change). Thread into `:274` (alpha) and `:319` (beta).
- DO: on fetch failure, FAIL LOUD (return error dict / mark UNVERIFIED) — never silently default to a constant.
- VERIFY: `.venv/bin/python scripts/compute_oss.py` shows a `spy_30d_return` that changes day-to-day and ≠ 0.58; `rolling_alpha` flips sign when portfolio crosses SPY.
- TEST: `tests/test_benchmark_integrity.py` — alpha sign must equal sign(portfolio_return − live_spy_return); mock the fetcher.
**0.4 — Gate 0 acceptance**
- Re-run `compute_oss.py`; confirm it now reflects reality (the -$1.67 loss).
- Add 0.1/0.2/0.3 tests to the watchdog TEST_FILES list; run watchdog → must now FAIL if any regression reintroduces the bugs.
### GATE 1 — Make the optimization loop load-bearing (depends on Gate 0)
**1.1 — Persistence schema.** Create tables in `trade_journal.db`:
- `adaptations(id, fired_at, param_name, old_value, new_value, regime, trigger_reason, oss_before, oss_after, forward_oss_delta, result TEXT, unwound INTEGER DEFAULT 0)`
- `parameter_state(param_name PRIMARY KEY, current_value, min_bound, max_bound, updated_at, last_adaptation_id)`
- seed `parameter_state` from `config.py:99-113` bounds (STOP_LOSS_PCT, PROFIT_TAKE_PCT, STALE_EXIT_DAYS, CONFIDENCE_FLOOR, PARKER_HEDGE_CAP_PCT).
**1.2 — Move adaptation from LLM prompt to code.** Create `scripts/apply_adaptations.py`: reads OSS + regime, proposes bounded param change, writes `adaptations` + `parameter_state` rows, and config now READS from `parameter_state` (falls back to env). LLM cron `2ca03134ea62` becomes reporter, not actor.
- Declarative framing: contract declares target OSS ≥ 1.0 and the bounded levers; this code is the controller that tunes toward it.
**1.3 — Auto-unwind (Rule 8-14 guardrail).** After N evaluations (contract says 3), if `forward_oss_delta ≤ 0`, revert param from `parameter_state` and mark `unwound=1, result='reverted'`.
- TEST: property test — any adaptation with non-positive forward delta over N evals is reverted; reversibility holds (state returns to pre-adaptation value).
**1.4 — Wire win-rate.** Repoint `_compute_adapt_win_rate` at `adaptations.result='improved'`. Now OSS weight 0.15 is real.
**1.5 — Watchdog.** Add P-009 (monotonicity) + P-010 (persistence/reversibility) tests. Watchdog fails if loop goes inert.
### Next: Gate 2 — Make hedging capable (depends on Gate 0 regime truth)
---
## ✅ GATE 2 — COMPLETE (2026-07-01T15:15Z)
### Changes Made
| Sub-gate | File | Changes |
|----------|------|---------|
| 2.1 | `consensus/ross/hedge.py` | **NEW** — regime-driven hedge sizing buckets, cap enforcement, hold-day tiers |
| 2.1 | `consensus/ross/sizing.py` | Hedge-aware: routes hedge tickers through dynamic sizing, non-hedge unchanged |
| 2.1 | `consensus/ross/gates.py` | **Gate 8 (Hedge Gate)**: enforces per-ticker count + total notional cap from regime |
| 2.1 | `consensus/ross/executor.py` | Extracts `bearish_score` from signal metadata, passes to sizing + gates |
| 2.2 | `trade_journal.db:broker_orders` | Added `hedge_strategy TEXT` column |
| 2.2 | `scripts/compute_hedge_attribution.py` | **NEW** — per-strategy profit factor, auto-retire PF < 0.5 for 30+ days |
| 2.3 | `consensus/ross/gates.py` | **Gate 2.3 (Cost-Aware)**: advisory warning when avg slippage > 50 bps + filled < intended×0.995 |
| test | `tests/test_hedge_sizing.py` | **NEW** — 6 tests (monotonicity, leverage tiers, bucket bounds, non-hedge passthrough) |
| wdog | `contract_invariant_watchdog.py` | Added `test_hedge_sizing.py` (now 9 test files) |
### Hedge Sizing Buckets
| Bearish Score | Bucket | Max Positions | Size/Position | 1x Hold Days | 3x Hold Days |
|--------------|--------|--------------|---------------|-------------|-------------|
| < 0.20 | no hedge | 0 | 0% | — | — |
| 0.20–0.39 | moderate | 1 | 1.0% | 5 | 2 |
| 0.40–0.69 | heavy | 2 | 1.5% | 5 | 2 |
| ≥ 0.70 | extreme | 3 | 2.0% | 5 | 2 |
- Capped at `PARKER_HEDGE_CAP_PCT` from `parameter_state` (current: 0.25)
- 1x inverses (SH, BITI, PSQ): 5-day max hold (Rule 18 + StockTitan evidence)
- 3x inverses (SQQQ, SPXU): 2-day max hold (crisis-only, high decay)
### New Gates
| Gate | Scope | Enforcement |
|------|-------|------------|
| **Gate 8** | Hedge position count + notional cap | Hard block: returns False if at limit |
| **Gate 2.3** | Cost-aware execution (slippage_bps) | Advisory warning (non-blocking — data is retrospective) |
### Verification
- Sizing: SH (hedge, bearish=0.8) → $4 on $200 equity; SH (bearish=0.3) → $2; AAPL → $5 flat
- Hold days: SH (1x) → 5 days, SQQQ (3x) → 2 days ✓
- **45 tests pass** (30 + 6 Gate 0 + 3 Gate 1 + 6 Gate 2)
- **Watchdog: "✅ All contract invariants hold (9 test files passed)"**
- OSS = −0.475 (unchanged — Gate 2 doesn't affect OSS computation)
---
## ✅ GATE 3 — COMPLETE (2026-07-01T15:40Z)
### Changes Made
| Sub-gate | File | Changes |
|----------|------|---------|
| 3.1a | `consensus/ross/discovery.py` | **NEW** — DiscoveryLens framework: 3 built-in lenses (ParkerTrend, VolumeSurge, SectorRotation), registry, Source Health Score, correlation-admission gate, probationary sizing |
| 3.1b | `consensus/signals.db:signals` | Added `discovery_lens TEXT` + `source_provenance TEXT` columns |
| 3.1b | `trade_journal.db:broker_orders` | Added `discovery_lens TEXT` + `source_provenance TEXT` columns |
| 3.1c | `consensus/ross/sizing.py` | Probationary sizing: 50% notional for tickers with <3 trades or PF < 1.0 |
| 3.1c | `consensus/ross/executor.py` | Passes `db_path` to sizing for probationary check |
| 3.2 | `scripts/update_ticker_universe.py` | **NEW** — Replaces static config_store.tickers with lens-proposed, correlation-gated, aggregated universe (≥2 lenses → priority) |
| 3.3 | `scripts/update_ticker_universe.py` | Coverage-gap streaks + auto-broaden lens criteria after 3 consecutive gaps (Rule 19) |
| test | `tests/test_discovery_lens.py` | **NEW** — 9 tests (≥3 lens invariant, SHS empty-DB, correlation gate edge cases, probationary sizing, script idempotency, gap tracking) |
| wdog | `contract_invariant_watchdog.py` | Added `test_discovery_lens.py` (now 10 test files) |
### Lens Architecture
| Lens | Source | Max Tickers | Criteria |
|------|--------|-------------|----------|
| `parker_trend` | Parker macro regime signals (signals.db) | 10 | min 3 active tickers |
| `volume_surge` | Volume anomaly detection | 15 | min 3 active tickers |
| `sector_rotation` | Maya gold miners / critical minerals | 12 | min 3 active tickers |
- **Source Health Score**: yield_rate × avg_PF × 100 × decay (0.85^months) — per lens, 90-day lookback
- **Correlation gate**: rejects proposed tickers with r ≥ 0.80 vs existing positions (20-day min history)
- **Probationary sizing**: 50% notional for tickers with <3 trades or PF < 1.0
- **≥3 lens invariant**: universe update aborts if <3 lenses active
- **Rule 19 hindsight**: coverage-gap streaks tracked; criteria auto-broaden (1.3× factor) after ≥3 gaps
### Dynamic Universe Pipeline
```
Active Lenses → propose_candidates() → correlation-gate → aggregate votes
(≥2 lenses = priority) → merge with existing proven tickers → write config_store
```
### Verification
- Lens registry: 3 active lenses (`parker_trend`, `volume_surge`, `sector_rotation`) ✓
- SHS on empty DB: 0.0 (no crash) ✓
- Pearson: 1.0 perfect, -1.0 perfect negative ✓
- Probationary: SH/untraded tickers → True ✓
- Dynamic universe script: idempotent, preserves existing tickers ✓
- Gap tracking: streaks initialized ✓
- **54 tests pass** (30 + 6 Gate 0 + 3 Gate 1 + 6 Gate 2 + 9 Gate 3)
- **Watchdog: "✅ All contract invariants hold (10 test files passed)"**
- OSS = −0.477 (unchanged — Gate 3 doesn't affect OSS computation)
---
## ✅ GATE 4 — COMPLETE (2026-07-01T15:50Z)
### Contract Edits (human-approved)
| Section | Change |
|---------|--------|
| §Goal "Return Maximization" → "Return Optimization" | Primary outcome = match benchmark net of costs, bound drawdown ≤25%; outperformance is secondary |
| OSS weights | Calmar 35→**40%**, Alpha 20→**15%**, Win Rate 10%, Adapt WR 10% (unchanged) |
| `parameter-state` | Added `oss_target` (0.5), `oss_floor` (0.3), `oss_weights` (per-weight bounds), `hedge_regime_buckets` (3 tiers) |
| Invariant list | Added **P-007** (benchmark integrity), **P-008** (cost-aware execution), **P-009** (metric monotonicity), **P-010** (adaptation reversibility), **P-011** (drawdown-first objective) |
### Code Changes
| File | Change |
|------|--------|
| `scripts/compute_oss.py` | W_CALMAR 0.35→**0.40**, W_ALPHA 0.20→**0.15** |
| `alpaca-trading-system.prose.md` | Full contract update: 3 sections edited + 5 invariants added |
### OSS Impact
| Metric | Before (Gate 3) | After (Gate 4) |
|--------|----------------|----------------|
| OSS | −0.477 | **−0.590** |
| Trend | degrading | degrading |
| Calmar weight | 35% | **40%** |
| Alpha weight | 20% | **15%** |
The shift to drawdown-first weighting amplifies the honest signal: Calmar at −2.174 now drives the composite more heavily, producing a truer picture.
### Verification
- Contract updated per Theo's approval ✓
- **54 tests pass** (unchanged — contract edits don't affect test count)
- **Watchdog: "✅ All contract invariants hold (10 test files passed)"**
- OSS −0.590 | Calmar −2.174 | SPY −1.1% | Beta 0.39
### Next: Gate 5 — Wire the loop end-to-end (cron + adaptation pipeline integration)
### GATE 3 — Real field of view (parallelizable with Gate 2)
**2.1 — Dynamic hedge sizing.** Replace frozen `MAX_HEDGE_POSITIONS=1`/`HEDGE_POSITION_SIZE_PCT=0.01` with regime-driven sizing bounded by `PARKER_HEDGE_CAP_PCT` (up to the 0.25 already authorized). Sizing reads `parameter_state`. Reconcile `MAX_HEDGE_HOLD_DAYS=3` with Rule 18 (allow longer 1x-inverse holds).
**2.2 — P-004 hedge attribution.** Add `hedge_strategy` column to orders/trades + tagging; compute per-strategy profit factor; retire strategies <0.5. Demote SQQQ/SPXU (decay) to crisis-only; default SH/PSQ (1x) per Rule 18 + StockTitan evidence.
**2.3 — P-008 cost-aware gate.** New gate in `gates.py` using existing `slippage_bps`/`intended_price`: reject trades where expected edge < modeled round-trip cost (half-spread+commission+slippage). Directly addresses small-account cost drag.
- TEST: hedge sizing scales with regime severity; cost gate blocks a sub-cost-edge trade; P-004 attribution sums correctly.
### GATE 3 — Real field of view (parallelizable with Gate 2)
**3.1 — P-005/P-006.** Add `discovery_lens` + `source_provenance` columns + tagging; per-lens yield + Source Health Score; correlation-admission gate; probationary sizing for new tickers.
**3.2 — Dynamic universe.** Replace static `config_store.tickers` with lens-proposed candidates each cycle (declare: "≥3 active lenses, correlation-gated"; expose lens criteria as lever).
**3.3 — Rule 19 hindsight.** Write coverage-gap counts; broaden lens criteria when gaps detected.
- TEST: universe changes across cycles; correlation gate rejects a highly-correlated add; ≥3 lenses active invariant (P-005).
### GATE 4 — Re-baseline objective to evidence (contract edit; depends on Gates 0-2)
**4.1 — Rewrite objective declaratively (P-011).** Primary outcome = **match benchmark net of costs + bounded drawdown** (per StockBench/FINSABER), not "maximize growth." Re-weight OSS toward downside/drawdown metrics. Make hedge cap a regime-driven lever.
**4.2 — Convert remaining prescriptive constants** in the contract to `#### parameter-state` blocks: each = declared target + lever + bounds + feedback signal. The system tunes them; the contract stops fixing them.
**4.3 — Add P-007/008/009/010/011** to the invariant set + watchdog.
---
## 4. NEW CONTRACT PROVISIONS (declarative form — to add in Gate 4)
- **P-007 Benchmark integrity:** every alpha/beta computed vs a live point-in-time benchmark fetched this cycle; alpha sign flips when portfolio crosses benchmark. (Kills 0.58 literal.)
- **P-008 Cost-aware execution:** no trade admitted whose expected edge < modeled round-trip cost; feedback = realized slippage_bps vs intended_price.
- **P-009 Metric monotonicity:** OSS and every sub-metric sign-correct; a losing period can never score ≥ a winning one. (Property-tested.)
- **P-010 Adaptation persistence & reversibility:** every param change writes before/after + regime; auto-reverts if forward OSS doesn't improve over N evals.
- **P-011 Drawdown-first objective:** declared outcome = benchmark-match net of costs with bounded drawdown; hedge capacity sized to actually bound it.
---
## 5. EXECUTION ORDER & GUARDRAILS
1. **Gate 0 first, always.** Until OSS sees losses, every adaptation optimizes against a lie.
2. Human gate before contract edits (Gate 4) per Theo's 2-agent + 1-human policy. Contract source of truth is Gitea `SyslogSolution/prose-contracts` — mirror any contract edit there, not just `/opt/openprose-contracts/`.
3. Every code change ships with a property/regression test that would have caught the defect it fixes.
4. Pin any new cron jobs (unpinned LLM crons break on provider switch).
5. After each gate: run `compute_oss.py` + watchdog; record OSS before/after as evidence.
6. [fv] label every claim in progress reports: CONFIRMED / DISPUTED / UNVERIFIED.
## 6. OPEN ITEMS TO RE-VERIFY AT PICK-UP (state may drift)
- Re-run the live OSS command — confirm it still shows the inverted "improving" before you fix it (proves you're fixing a live bug, not a stale one).
- Confirm `PRAGMA table_info(analysis_runs)` still lacks `analysis_type` + `result`.
- Confirm `.env.prod` hedge caps unchanged (`MAX_HEDGE_POSITIONS`, `HEDGE_POSITION_SIZE_PCT`).
- Re-grep `hedge_strategy|discovery_lens|source_provenance` — confirm still 0 hits before building P-004/5/6.
- Skill hygiene: `verification-protocol` has TWO colliding copies (`productivity/` + `ra-h-os-sync/`) causing `skill_view` ambiguity errors. Load by full path or dedupe.
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -356,7 +356,7 @@ Postconditions to verify:
},
{
"check": "SearXNG reachable",
"verify": "curl -sf http://192.168.68.7:8888",
"verify": "curl -sf http://192.168.68.17:8080",
"expect": "200 OK"
}
]
+7 -8
View File
@@ -1,6 +1,4 @@
---
report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
kind: responsibility
name: disk-gc-threat-response
description: >
@@ -13,7 +11,6 @@ description: >
id: 067NV8KJ03ZG71S44N41F31022
version: 1.0.0
---
---
# Disk GC & Threat Response
@@ -38,7 +35,8 @@ Docker hosts get special attention:
| All other CTs | — | LOW — no Docker | apt clean, log rotate |
| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
> **Note:** CT 118 is now jdownloader (active on storepve). CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
## Threat Levels
@@ -277,10 +275,10 @@ one-off GPU builds. No automated post-migration cleanup was in place.
### CT Access (via pct-run)
| CT | Name | Node | Status |
|----|------|------|--------|
| 100 | abiba | minipve | local |
| 102 | adguard | minipve | ✅ reachable |
| 100 | abiba | amdpve | local |
| 102 | adguard | acerpve | ✅ reachable |
| 104 | authentik | minipve | ✅ reachable |
| 105 | kagentz | minipve | ✅ reachable |
| 105 | kagentz | amdpve | ✅ reachable |
| 106 | ra-h-os | storepve | ✅ reachable |
| 107 | pbs | storepve | ✅ reachable |
| 108 | media | storepve | ✅ reachable |
@@ -288,6 +286,7 @@ one-off GPU builds. No automated post-migration cleanup was in place.
| 111 | tdunna | amdpve | ✅ reachable |
| 112 | tanko | amdpve | ✅ reachable |
| 113 | baggy | amdpve | ✅ reachable |
| 114 | mumuni | minipve | ✅ reachable |
| 115 | scottdenya | amdpve | ✅ reachable |
| 116 | syslog-api | minipve | ✅ reachable |
| 117 | zulip | storepve | ✅ reachable |
@@ -304,4 +303,4 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|------|-----|------|--------|
| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
> **Note:** CT 118 is now jdownloader (active on storepve). CT 119 (infisical-vault) added on minipve.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
+1 -1
View File
@@ -13,7 +13,7 @@ the description:
1. **What system does this contract touch?** Name the hosts, CTs, containers,
and services explicitly. "The inference fleet" is vague. "GPU .8 (RTX 3090,
qwen), .110 (RTX 5070, gemma), .15 (Strix Halo, strix-moe), and LiteLLM on CT
qwen), .110 (RTX 5070, gemma), .15 (Strix Halo, ornith), and LiteLLM on CT
116" is specific.
2. **Who runs this contract, and when?** State the agent, the trigger (cron,
+26 -13
View File
@@ -71,7 +71,7 @@ triggers:
│ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │
│ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │
│ 128K ctx │ │ 128K ctx │ │ 128K ctx │ │ Watchdog │
│ qwen3.6 │ │ Qwen3.5-9B │ │ qwen3.5 │ │ Prometheus │
│ qwen3.6 │ │ gemma-4-12b │ │ qwen3.6 │ │ Prometheus │
│ 27B-code │ │ :8080 │ │ -35B-udq4 │ │ exporter │
│ :8080 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │
│ :9400 │ └─────────────┘ └───────────┘ └──────────────┘
@@ -86,16 +86,19 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen
| Alias | GPU | Current Model | Will Route To |
|-------|-----|---------------|---------------|
| `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
| `gpu-dense` | RTX 3090 (.8) | Qwen3.8-27B-Uncensored-Q4_K_M | Whatever runs on RTX 3090 |
| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 |
| `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.5-9b-it) still work
**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
but are deprecated for agent configs. Only the stable aliases survive model swaps.
## Current Model Assignments (2026-07-15)
| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
|-------|-----|------|------|-----|----------|----------|-------------|--------|
| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s |
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |
## Routing Configuration (LiteLLM — July 2026)
@@ -103,7 +106,9 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
| Model | GPU | Weight | RPM Cap | Timeout |
|-------|-----|--------|---------|---------|
| Qwen3.8-27B-Uncensored-Q4_K_M | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path.
@@ -111,6 +116,9 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
| Model | RPM Cap | Notes |
|-------|---------|-------|
| strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload |
| qwen3.6-27B-code | 500 | High cap — primary workhorse |
| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
### Stable Aliases (for agent configs — never change)
@@ -181,7 +189,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
- Qwen3.5-9B: 120s, qwen3.6-27B-code: 300s, Carnice-Qwen3.6-MoE-35B-A3B/strix-moe: 300s (strix-moe alias retained, legacy name qwen3.6-35B-udq4 deprecated)
- gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (strix-moe does NOT exist — legacy name, do not use)
- global request_timeout: 300s, nginx proxy_read_timeout: 600s
6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
7. Check port conflicts: verify only one llama-server on :8080 per host
@@ -196,7 +204,7 @@ Plaintext keys removed from this contract post-vault-migration.
| Agent | CT | IP | LiteLLM Alias | Key Source | Access |
|-------|-----|-----|---------------|------------|--------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
| Mumuni | 105 (kagentz) | .14 | `mumuni` | Infisical vault | SSH root |
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
@@ -238,9 +246,14 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
- **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (old Mumuni CT114 — now inside Abiba CT100 at .24) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
@@ -252,7 +265,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
|-----|-------|-----------|--------------|----------|---------|
| RTX 3090 (.8) | Qwen3.8-27B-Uncensored-Q4_K_M | **TBD** | — | — | **128K** |
| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** |
| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
| Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** |
Benchmarks from 2026-07-17. Strix Halo model: qwen3.6-35B-udq4. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
@@ -279,13 +293,12 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
### Context Windows
- RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
- **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
- Compression threshold 0.60: fires at ~77K (~51K headroom before 128K ceiling)
- **Pi agents (Abiba)**: `compaction.reserveTokens: 52739` (≈60% of 128K)
- Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
- Mumuni compression model alias: `strix-moe` with 300s timeout
### Mumuni Agent Profile
Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the primary business assistant. This profile is the reference for all agent configs:
Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs:
| Setting | Value | Notes |
|---------|-------|-------|
@@ -297,7 +310,7 @@ Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the
| `aux.web_extract.model` | `gpu-light` | Web extraction |
| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
| `context.max_context_window` | 131072 (128K) | Reduced from 256K 2026-07-17 — stable 128K ceiling |
| `compression.threshold` | 0.60 | Triggers at ~77K (~60% of 128K) — optimized for 128K context |
| `compression.threshold` | 0.65 | Triggers at ~85K |
| `compression.target_ratio` | 0.3 | Compresses to ~38K |
| `compression.protect_last_n` | 40 | Preserves last 40 messages |
| `memory.memory_char_limit` | 800 | Brief memory entries |
@@ -310,7 +323,7 @@ Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the
| Agent | Host | Status |
|-------|------|--------|
| **Mumuni** | CT105 (.14) | ✅ Updated to stable aliases |
| **Mumuni** | CT114 (.123) | ✅ Updated to stable aliases |
| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
+1 -22
View File
@@ -97,28 +97,7 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and
`curl http://localhost:9100/gpu-data | jq` — Full fleet status
### check-health
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
```bash
# GPU Monitor health
curl http://localhost:9100/health | jq
# Expected: 200 with {"status": "healthy", "cache_age_seconds": <n>}
# Router health (via nginx on port 80)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health
# Expected: 200 (Router is up and responding)
# LiteLLM health (via nginx on port 80)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health
# Expected: 200 (LiteLLM is up and responding)
# Dashboard
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/dashboard/
# Expected: 200 (Dashboard is up and responding)
```
**Report format**: Summarize actual results from each probe. If any probe returns non-200, flag as alert.
`curl http://localhost:9100/health` — Monitor self-check
### view-dashboard
Open `http://localhost:9100/` in browser — Live HTML dashboard
+6 -11
View File
@@ -1,6 +1,4 @@
---
report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
kind: responsibility
name: gpu-self-heal
description: >
@@ -18,7 +16,6 @@ depends_on:
- gpu-monitor.prose.md (live data source on .24:9100)
- gpu-fleet.prose.md (source of truth for topology, aliases, model assignments)
---
---
## Maintains
@@ -41,19 +38,20 @@ depends_on:
- On fix: verify with benchmark inference test before declaring resolved
- Escalate: after 3 failed remediation attempts → Zulip #agent-hub alert
---
---
## Current Fleet Baseline (2026-07-18)
| Alias | GPU | Host | Model | VRAM | Ctx | tok/s | Role |
|-------|-----|------|-------|------|-----|-------|------|
| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
| `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |
Key notes:
- All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
- Stable aliases (gpu-dense, gpu-light, strix-moe) from gpu-fleet are the canonical names for agent configs. Model-specific names still work but are deprecated.
- RTX 5070 tok/s is ~145 for Qwen3.5-9B — gpu-light is the fastest endpoint. Route vision/web/light work there first. NOTE: Qwen3.5-9B is multimodal (image+text), NOT text-only like gemma-4-12b was.
- RTX 5070 tok/s is 2.3x faster than RTX 3090 for its model — gpu-light is the fastest endpoint. Route vision/web/light work there first.
- Strix Halo is 62.9 tok/s (89% of 70.5 baseline) — below optimal but stable. Check for competing workloads.
- RTX 3090 VRAM at 88% — within role-appropriate range (role = heavy reasoning, needs the headroom).
- RTX 5070 VRAM at 83% — role-appropriate for vision/web (smaller batch sizes).
@@ -158,7 +156,7 @@ Key notes:
- **Detect**: GPU roles misaligned with hardware capabilities
- **Target distribution**:
- RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
- RTX 5070 (gpu-light, 12GB, ~145 tok/s) → Vision (image+text), web search, lightweight tasks. Weight: 0.15 (LiteLLM).
- RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
- Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
- **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
- **Fix**:
@@ -168,7 +166,6 @@ Key notes:
- **Verify**: Each GPU's request pattern matches its designated role within 24h
- **Escalate**: If role mismatch persists >48h → agent alias audit needed
---
---
## Execution
@@ -250,13 +247,12 @@ call update-gpu-health
}
```
---
---
## Reporting
### 1. Gitea Log (not knowledge graph — hard rule)
Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchable, not in graph.
### 1. Knowledge Graph
Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
### 2. Zulip Alerts (#agent-hub → alerts-gpu)
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
@@ -267,7 +263,6 @@ Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchab
- Per-GPU tok/s trend over 7 days
- Regression alerts if any GPU degrades >10% week-over-week
---
---
## Design Decisions (Verified 2026-07-12, Reaffirmed 2026-07-18)
+12 -15
View File
@@ -24,11 +24,13 @@ done
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|---------------|------------|----------|
| Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes |
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
> **Note**: CT hostnames (tdunna→CT129, baggy→CT114) differ from agent identities (koby, koonimo).
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
Keys are stored in Infisical vault (project=agents, env=production) and injected at
@@ -51,10 +53,9 @@ Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) →
## Config Pattern — Mandatory Fields
### For Hermes Agents (Mumuni, Koonimo)
### For Hermes Agents (Tanko, Mumuni, Koonimo)
Every Hermes agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
(Tanko is excluded — migrated to DSH/DeepSeek Harness on 2026-08-27, no longer uses Hermes config.)
Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
### 1. Main Model
```yaml
@@ -70,7 +71,7 @@ model:
custom_providers:
- name: harness
model: syslog-auto
base_url: http://192.168.68.116/litellm/v1
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
api_mode: chat_completions
```
@@ -81,7 +82,7 @@ auxiliary:
vision:
provider: harness
model: gemma-4-12b # or syslog-auto
base_url: http://192.168.68.116/litellm/v1
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround
timeout: 60
@@ -96,7 +97,7 @@ auxiliary:
target_ratio: 0.3
provider: harness
model: syslog-auto # or gemma-4-12b
base_url: http://192.168.68.116/litellm/v1
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround
timeout: 120
@@ -168,15 +169,11 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
```
### For Koby (CT 111 / tdunna) — **REPORT-ONLY MODE**
### For Koby (CT 129 / tdunna)
Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
Koby runs Hermes on CT 129 (tdunna). Config files at `/root/.hermes/config.yaml`.
Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
**⛔ KOBY IS NEVER REPAIRED (2026-08-17, Captain)**: Diagnostic only — detect and report, never fix on .129.
No heal step, no restart, no key rotation, no config edit, no memory rewrite, no disk GC, no service touch, no process kill — ever.
If a health check shows Koby degraded, **DO NOT** execute any repair action. Instead, report to Zulip and let Theo fix it.
**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
### For pi Agents (Abiba)
+50 -96
View File
@@ -5,17 +5,21 @@ description: >
Standard Hermes configuration template for Syslog Solution LLC agents.
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
RA-H OS MCP) while keeping agent-specific API keys and model choices.
UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident.
UPDATED 2026-07-18: Compression model switched to `syslog-auto` (was `strix-moe`)
to relieve Strix Halo pressure. syslog-auto distributes compression across the
weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
UPDATED 2026-07-16: Compression model was the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
2026-07-16 Mumuni root-cause investigation (WAL #1300).
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (later switched to syslog-auto 2026-07-18). RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
---
## Maintains
- template_version: "2.1.0"
- last_applied: timestamp
- agents_configured: ["mumuni", "abiba", "koby", "koonimo", "kagenz0"] # tanko removed 2026-08-27 (now on DSH/DeepSeek Harness)
- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
- agent_keys: map (see Agent Keys section)
- infra_endpoints_verified: array
@@ -30,9 +34,11 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Agent | Key Alias | Host | SSH | Sub-Agents |
|-------|-----------|------|-----|-----------|
| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
| Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
| Koonimo | `koonimo` | CT 113 (baggy) | SSH root | — |
| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
| Kagenz0 | `kagenz0-*` | ? | Zulip | — |
> CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
@@ -45,7 +51,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Component | Endpoint | Purpose |
|---|---|---|
| Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
| SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search |
| SearXNG | `http://storepve:8888` | Privacy-respecting web search |
| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
| RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
@@ -94,14 +100,10 @@ work immediately after restart.
model:
default: <agent_model> # e.g., strix-moe, qwen3.6-27B-code, syslog-auto
provider: harness
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated path; /v1 also OK
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper
max_tokens: 4096 # ⚠️ CRITICAL: Prevents unbounded generation
context_length: 131072 # For syslog-auto (all GPUs at 128K for stability).
# ⚠️ MANDATORY: Hermes probes unknown models from 256K
# and falls back to 256K when /v1/models lacks a context
# field (llama-server does). Without this override, agents
# silently run syslog-auto at 256K (verified 2026-08-09).
# Set 65536 if using gemma-4-12b directly (tight VRAM).
fallback_providers:
@@ -131,7 +133,9 @@ mcp_servers:
# ─── Compression ───
compression:
enabled: true
model: syslog-auto # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name).
model: syslog-auto # ⚠️ Switched from strix-moe 2026-07-18 to relieve Strix Halo.
# syslog-auto distributes across weighted pool (55% RTX 3090,
# 30% Strix Halo, 15% RTX 5070). All GPUs at 128K.
provider: harness
max_context_window: 131072 # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17).
threshold: 0.65 # Fires at ~170K for 262K window, ~85K for 128K
@@ -144,10 +148,11 @@ compression:
# ─── Auxiliary Tasks (CONSISTENCY RULE) ───
# All auxiliary services MUST use identical model, base_url, and api_key_env:
# model: gpu-light # stable alias (NOT raw "gemma-4-12b")
# base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
# base_url: http://192.168.68.116/v1
# api_key_env: LITELLM_API_KEY
# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
# Compression uses syslog-auto (switched from strix-moe 2026-07-18) to distribute
# load across the weighted pool and relieve Strix Halo pressure.
# Vision and web_extract use gpu-light = RTX 5070 (12B).
# Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
# NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
# in agent configs — use the stable aliases so model swaps don't break agents.
@@ -155,20 +160,20 @@ auxiliary:
vision:
provider: harness
model: gpu-light # stable alias for RTX 5070 (was raw gemma-4-12b)
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
timeout: 60
download_timeout: 30
web_extract:
provider: harness
model: gpu-light # stable alias for RTX 5070
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
timeout: 30
compression:
provider: harness
model: syslog-auto # MUST match compression.model above. Stable alias for Strix Halo (weighted pool).
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated path; /v1 also OK
model: syslog-auto # Switched from strix-moe 2026-07-18. Relieves Strix Halo pressure.
base_url: http://192.168.68.116/v1 # Rule 5: /v1 NOT /litellm/v1
api_key_env: LITELLM_API_KEY
timeout: 300 # gpu-fleet: 300s for large-history summarization (was 60)
@@ -178,14 +183,14 @@ auxiliary:
delegation:
model: gpu-dense # stable alias for RTX 3090 (was raw qwen3.6-27B-code)
provider: harness
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
# ─── Custom Provider ───
custom_providers:
- name: harness
model: syslog-auto # weighted pool (default)
base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
api_mode: chat_completions
```
@@ -235,13 +240,10 @@ The following MUST be identical across ALL profiles:
- When main config uses `api_key_env`, sub-agents automatically use it
- This means key rotation only touches ONE vault secret (`LITELLM_API_KEY`)
### Rule 5: Main Config Base URL (UPDATED 2026-08-09)
|- Use the authenticated LiteLLM path: `http://192.168.68.116/litellm/v1` (canonical, captain-approved migration)
|- Legacy `http://192.68.68.116/v1` also works — nginx fronts BOTH paths with key auth
(verified 2026-08-09: 401 without key, 200 with key, on both /v1 and /litellm/v1)
|- Both locations have `proxy_read_timeout 600s` (verified in harness-nginx nginx.conf) —
the old "60s timeout on /litellm/" claim was stale and is retracted
### Rule 5: Main Config Base URL
|- Use direct IP: `http://192.168.68.116/v1`
|- NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down
|- NOT the old path (`/litellm/v1`) — nginx now routes `/v1` directly
### Rule 6: max_tokens Is Required (Thermal Safety)
- **Every Hermes config MUST set `model.max_tokens: 4096`** — this is non-negotiable
@@ -251,34 +253,30 @@ The following MUST be identical across ALL profiles:
- Apply to BOTH main config AND all sub-agent profiles
- For agents needing longer outputs: raise to 8192, but never omit
### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16)
- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 128K ctx, compression-optimized)
- **`strix-moe` is the only valid compression model name** — LiteLLM does NOT serve `ornith-1.0-35b`
(it serves `strix-moe`, `qwen3.6-35B-udq4`, `gpu-dense`, `gpu-light`, `syslog-auto`, `gemma-4-12b`, `qwen3.6-27B-code`). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls.
- **OPERATIONAL DECISION (2026-07-23): Use `syslog-auto` for compression across all agents.**
The `syslog-auto` alias routes to the Strix Halo, but uses the weighted pool instead of pinning
to `strix-moe` directly. This prevents sustained Strix Halo thermal load because the pool can
fall back to other GPUs if Strix gets hot. Both `compression.model` and `auxiliary.compression.model`
MUST be `syslog-auto`.
### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-18)
- Vision and web_extract use `gpu-light` (stable alias, RTX 5070 — 12GB, vision-optimized)
- Compression now uses `syslog-auto` (switched from `strix-moe` 2026-07-18) to distribute
compression load across the weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
This relieves Strix Halo pressure while keeping compression functional on all GPUs.
- **`syslog-auto` is the valid compression model** — LiteLLM serves it as the weighted pool.
Old configs with `strix-moe` for compression should be updated to `syslog-auto`.
- All auxiliary services MUST use identical routing:
- `base_url: http://192.168.68.116/litellm/v1` (Rule 5, 2026-08-09: canonical authenticated; `/v1` also OK)
- `base_url: http://192.168.68.116/v1` (Rule 5: `/v1`, NOT `/litellm/v1`)
- `api_key_env: LITELLM_API_KEY`
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
- **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
(64GB UMA, 128K context) — the designated compression GPU. This frees the
RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
- **Compression via syslog-auto**: Routes through the weighted pool. Strix Halo still handles
~30% of compression calls (at 60 RPM via pool vs 40 RPM direct), but the bulk (55%)
goes to RTX 3090 which has ample spare capacity.
- The `compression:` block's `model` MUST match `auxiliary: compression: model`
- The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K)
### Rule 8: GPU Workload Distribution (UPDATED 2026-07-16)
- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K)
- **Strix Halo (64GB, 128K ctx, syslog-auto)**: Context compression, summarization, long docs
### Rule 8: GPU Workload Distribution (UPDATED 2026-07-18)
- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations — also handles ~55% of compression via syslog-auto pool
- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract — handles ~15% of compression via syslog-auto pool
- **Strix Halo (64GB, 128K ctx, qwen3.6-35B-udq4)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs
- Agent profiles MUST route auxiliary tasks to the correct GPU:
- `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
- `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
- `auxiliary.compression.model: syslog-auto` (Strix Halo)
- `auxiliary.vision.model: gpu-light` (RTX 5070)
- `auxiliary.web_extract.model: gpu-light` (RTX 5070)
- `auxiliary.compression.model: syslog-auto` (distributed pool, switched from strix-moe 2026-07-18)
- Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
- For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
@@ -294,7 +292,6 @@ The following MUST be identical across ALL profiles:
- See `devops-hermes-compression` skill for full reference
### Rule 10: Default Model Must Be `syslog-auto` (All Agents)
- **Koby Exception**: Per captain ruling 2026-08-11, Koby is a DeepSeek-primary external agent; its primary model remains `deepseek-v4-flash` (via api.deepseek.com to preserve DeepSeek-specific reasoning, while other sections follow Rule 10.
- **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
- **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
- `syslog-auto` is the LiteLLM routing model — it load-balances between strix-moe
@@ -323,13 +320,10 @@ verify ALL FOUR of these against the live config. They are the only root causes
1. **max_context_window correct?** — BOTH `compression.max_context_window` AND `context.max_context_window`
MUST be `131072` (all GPUs are 128K). A value of `262144` causes instability near 100K and must NOT be used.
~83K instead of ~170K. Check: `grep -n max_context_window ~/.hermes/config.yaml`
2. **base_url uses authenticated path?** — `custom_providers[0].base_url`, `delegation.base_url`,
and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/litellm/v1` (Rule 5, canonical)
or `http://192.168.68.116/v1` (legacy, still authenticated via nginx). BOTH verified 200 with
key + 600s proxy_read_timeout on 2026-08-09. Never bare `:4000` direct.
Check: `grep -nE 'base_url: http://192.168.68.116(:4000)?/v1' ~/.hermes/config.yaml` — the
ONLY paths allowed are `/v1` or `/litellm/v1` (both via nginx :80).
`:4000` or missing `litellm/v1`/`v1` prefix = violation.
2. **base_url uses /v1 NOT /litellm/v1?** — `custom_providers[0].base_url`, `delegation.base_url`,
and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/v1` (Rule 5). nginx `/litellm/`
has a 60s default timeout → 504 on any inference >60s; `/v1/` has 600s.
Check: `grep -n 'litellm/v1' ~/.hermes/config.yaml` (must return NOTHING)
3. **LITELLM_API_KEY valid?** — The key must be a real LiteLLM key (`sk-` + 64 hex, 67 chars).
Malformed values (e.g. `sk-_SWAl_Vu_…`, 47 chars) return 401 → DeepSeek fallback.
Verify: `curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $LITELLM_API_KEY" http://192.168.68.116/v1/models` (must be 200)
@@ -346,17 +340,6 @@ curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $
### Rule 13: API Key Injection — Two Patterns (UPDATED 2026-07-16, WAL #1300)
### Rule 14: Hermes Context Detection Uses `max_model_tokens`, NOT `max_input_tokens`
**CRITICAL**: Hermes context detection reads `max_model_tokens` (128K), NOT `max_input_tokens` (64K cap).
- **Abiba and Hermes agents**: `max_model_tokens: 131072` (128K) — unlimited context
- **Crewmates (ops, tune, verify, auth-keys, build)**: `max_input_tokens: 64000` (64K) — capped
- If you see `max_input_tokens: 64000` in an Abiba/Hermes config, that's a mistake
- Using `max_input_tokens` for Hermes agents causes premature context loss
- Check: `grep -n 'max_model_tokens\|max_input_tokens' ~/.hermes/config.yaml`
- Expected output: `max_model_tokens: 131072` (not max_input_tokens)
Agents inject `LITELLM_API_KEY` via ONE of two mechanisms. Both are valid; the contract
requirement is that the key is a **valid LiteLLM virtual key** (HTTP 200 on /v1/models).
@@ -395,35 +378,6 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
- `/etc/environment` is NO LONGER the canonical key source (stale values there caused 401s).
- Do NOT leave a hardcoded stale key in `/etc/environment` — it shadows the drop-in/wrapper.
### Rule 14: Provider Name Must Match custom_providers Name (ADDED 2026-07-19, WAL #1471)
- `model.provider` MUST be `harness` (the `custom_providers[0].name`), NOT the literal string `custom`
- When `provider: custom`, Hermes' `_get_named_custom_provider("custom")` returns None (no provider is
named "custom" — it is named "harness"), causing a fall-through to the generic resolution path
(`source: env/config`) at `runtime_provider.py:1156`
- The generic path builds `api_key_candidates` from `model.api_key` (empty), host-gated
OLLAMA/OPENAI/OPENROUTER keys, and `_host_derived_api_key` (returns "" for IP addresses)
- **The generic path does NOT resolve `model.api_key_env` or `custom_providers.key_env`** —
`LITELLM_API_KEY` is never read, producing `api_key = "no-key-required"` → HTTP 401
- The named custom provider path (`source: custom_provider:harness`) DOES read `key_env` —
but only triggers when `provider` matches the `custom_providers[0].name`
- All sections MUST use `provider: harness`: `model`, `compression`, `auxiliary.vision`,
`auxiliary.web_extract`, `auxiliary.compression`, `delegation`
- Only `fallback_providers` uses a different provider (`deepseek`) for true fallback diversity
- **Diagnostic**: If you see `source: env/config` in a request dump or log, the provider name
is wrong. It should be `source: custom_provider:harness`.
- **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>`
before and after any config change to catch this and all other rule violations.
### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07)
- Every MCP server entry must point at the correct endpoint:
- ra-h-os = http://192.168.68.65:3100/mcp
- litellm = https://litellm.sysloggh.net/mcp
- MCP entries must carry a REAL key value in the header.
- Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP
endpoints and result in "Malformed API Key" floods.
- Ensure the header value is the actual key (e.g., `sk-...`).
## Execution
1. **Check current config** — Read the target agent's config.yaml
+16 -17
View File
@@ -34,12 +34,11 @@ Syslog is migrating away from **unauthenticated direct access** to the shared in
| Path | Auth | Status |
|------|------|--------|
| `http://192.168.68.116/v1` | Bearer `sk-*` key (nginx-fronted) | ✅ **VALID** — authenticated via nginx :80 (verified 2026-08-09: 401 without key, 200 with) |
| `http://192.168.68.116/litellm/v1` | Bearer `sk-*` key (nginx-fronted) | ✅ **CURRENT / CANONICAL** — captain-approved migration target; 600s proxy_read_timeout (verified) |
| `http://192.168.68.116:4000/v1` | Bearer `sk-*` key (direct container) | ❌ **FORBIDDEN** — bypasses nginx; port 4000 direct is not a config path |
| `http://192.168.68.116/v1` | None (direct) | ❌ **DEPRECATED** — being phased out |
| `http://192.168.68.116/litellm/v1/responses` | Bearer `sk-*` key | ✅ **CURRENT** — authenticated LiteLLM proxy |
All harness/litellm providers MUST use an authenticated nginx-fronted path (`/litellm/v1` canonical, `/v1` legacy-valid).
Any `base_url` pointing at `:4000` or a bare IP without nginx is a **migration violation**.
All harness/litellm providers MUST use the authenticated `/litellm/v1/responses` path.
Any `base_url` pointing to bare `/v1` on 192.168.68.116 is a **migration violation**.
### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
@@ -190,23 +189,23 @@ litellm_settings:
| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
|-------|-----|-----|---------------|------------|--------|-----------------|---------------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
| Mumuni | 105 (kagentz) | .14 | `mumuni` | Infisical vault | ✅ Fixed | systemd Hermes gateway | 2026-08-29 |
| Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed (DeepSeek-primary) | `infisical run` | 23:30 UTC Jul 5 |
| Koonimo | 113 | .114 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-08-09 |
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
| Kagenz0 | 105 | .14 | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
> **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
> LiteLLM key aliases use agent identity, not CT hostname.
### Migration Status: Authenticated Path
| Agent | `/litellm/v1` | Legacy `/v1` | Status |
|-------|--------------|-------------|--------|
| Mumuni | ✅ harness provider | ✅ auxiliary on /v1 (valid) | ✅ Authenticated (verified 2026-08-09) |
| Tanko | ✅ 5 sections | 0 | ✅ Migrated 2026-08-08, keys 200 |
| Koby | ✅ custom provider (harness name) | — | ✅ External DeepSeek primary (intentional, captain ruling 2026-08-11) |
| Koonimo | ✅ .114 (baggy) | — | ✅ 128K context applied 2026-08-09 |
| Agent | `/litellm/v1/responses` | Deprecated `/v1` | Status |
|-------|--------------------------|--------------------|--------|
| Mumuni | ✅ 5 sections | 0 | ✅ Authenticated |
| Tanko | ⚠️ No SSH access | — | Needs check |
| Koby | ⚠️ No route to host | — | Needs check |
| Koonimo | ⚠️ Connection timed out | — | Needs check |
### Systemd Service Pattern (2026-07-11 — vault migration)
@@ -285,13 +284,13 @@ auxiliary:
vision:
api_key: sk-<agent-key-from-vault> # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
base_url: http://192.168.68.116/v1
model: gemma-4-12b
provider: harness
compression:
api_key: sk-<agent-key-from-vault> # ← workaround (same as above)
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
base_url: http://192.168.68.116/v1
model: gemma-4-12b
provider: harness
```
+4 -4
View File
@@ -28,7 +28,7 @@ connectivity recovery including end-to-end DM validation.
| Param | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — on DSH since 2026-08-27, no Hermes plugin) |
| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
| `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
## Maintains
@@ -55,7 +55,8 @@ connectivity recovery including end-to-end DM validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------|
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
@@ -120,8 +121,7 @@ cp plugins/platforms/zulip/adapter.py \
plugins/platforms/zulip/plugin.yaml \
{{hermes_home}}/hermes-agent/plugins/platforms/zulip/
# Fix ownership (was Tanko-only, runs as jerome user)
# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (DSH).
# Fix ownership (Tanko only — runs as jerome user)
[ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
{{hermes_home}}/hermes-agent/plugins/platforms/zulip/
+8 -8
View File
@@ -1,9 +1,9 @@
---
report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
kind: function
name: hermes-zulip-restore
description: >
Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
path, verifies env credentials, restarts the gateway, and confirms Zulip
connects. Run this whenever a Hermes agent stops responding on Zulip or after
a fresh agent deployment.
@@ -12,7 +12,6 @@ version: 1.0.0
status: active
runtime_contract: 2
---
---
# Hermes Zulip Restore — Bring Any Agent Back to Good State
@@ -24,7 +23,7 @@ gateway restart, and connection validation.
| Param | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — DSH since 2026-08-27) |
| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
## Maintains
@@ -37,7 +36,7 @@ gateway restart, and connection validation.
- `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
- All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko, historical — DSH since 2026-08-27)
- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
- Gateway restarted and zulip platform reports state `connected`
- HTML stripping enabled for `/approve` and `/deny` slash command support
@@ -52,6 +51,8 @@ gateway restart, and connection validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------|
| Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
@@ -93,8 +94,8 @@ cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on DSH, no Hermes plugin)
chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/ # Tanko only (historical)
# Fix ownership (Tanko only)
chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/ # Tanko only
# Clean up
rm -rf /tmp/zulip-deploy
@@ -183,7 +184,6 @@ https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/z
Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
Pull request #33 is the primary integration branch.
---
---
**Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.
+2 -2
View File
@@ -87,8 +87,8 @@ call apply-liteLLM-routing
call apply-agent-compression
agent: mumuni
host: 192.168.68.14
config_path: /home/hermes/.hermes/config.yaml
host: 192.168.68.123
config_path: /root/.hermes/config.yaml
-- Phase 4: Enable llama.cpp prompt caching on GPU hosts
+40 -66
View File
@@ -12,12 +12,6 @@ description: >
never mutate infrastructure based on them without first confirming
against the live system. Policy fields are authoritative. See the
`verify-before-mutate` skill.
**Last verified:** 2026-08-15 — hwepve removed from Tabiri cluster
(now 5 nodes: minipve, amdpve, storepve, acerpve, ocupve). hwepve
(192.168.68.4) is a standalone PVE node + NetBird routing peer;
London relocation pending. CTs 100 (abiba) and 105 (kagentz) moved
to minipve.
---
# Infrastructure Control Pattern
@@ -35,19 +29,19 @@ description: >
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ Abiba │ │ Tanko │ │ Mumuni │
│ (pi) │ │ (Hermes) │ │ (Hermes) │
│ CT 100 │ │ CT 112 │ │ CT 100 │
│ CT 100 │ │ CT 112 │ │ CT 114 │
└──────┬──────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
└──────────────────┼────────────────────┘
▼
┌────────────────────────────────────┐
│ Proxmox Cluster API │
│ minipve.sysloggh.net:443 │
│ (monitoring@pve!mumuni token) │
└────┬──────┬──────┬──────┬──────────┘
│ │ │ │
┌────┘ ┌────┘ ┌────┘ ┌────┘
▼ ▼ ▼ ▼
┌──────────────────────────────────────┐
│ Proxmox Cluster API │
│ minipve.sysloggh.net:443 │
│ (monitoring@pve!mumuni token) │
└────┬──────┬──────┬──────┬──────┬─────┘
│ │ │ │ │
┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
▼ ▼ ▼ ▼ ▼
minipve amdpve storepve acerpve ocupve
(.12) (.15) (.6) (.9) (.5)
@@ -91,7 +85,7 @@ description: >
### Reachability Matrix
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.24) | Baggy (.114) |
| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
|-----------|---------|----------------|--------|-------------|---------------|----------------|
| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
@@ -105,26 +99,12 @@ description: >
| Node | IP | CPU | RAM | VMs/CTs | Role |
|------|----|-----|-----|---------|------|
| minipve | .12 | 16C | 30GB | abiba, kagentz, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute |
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat |
| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
| minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute |
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat |
| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs |
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
> minipve at .10, not acerpve. Abiba (CT 100) and kagentz (CT 105) are on
> minipve (moved from hwepve 2026-08-15). Mumuni runs inside Abiba CT100
> (.24); CT 114 (mumuni) no longer exists in the cluster.
>
> **hwepve (192.168.68.4) — STANDALONE (removed from Tabiri 2026-08-15):**
> Huawei MateBook 16 (KLVL-WXX9), pve-manager/9.2.10, kernel 7.0.14-8-pve.
> Zero VMs/CTs. Being relocated to London as a standalone PVE node + NetBird
> routing peer (relocation pending). Localizations applied: timezone
> Europe/London, lid-switch ignore, sleep/suspend/hibernate targets masked,
> cluster-shared storage removed (remaining: local, local-lvm, storage,
> mediastore). prometheus-node-exporter active on :9100; net.ipv4.ip_forward=1;
> NetBird client not yet installed (enrollment pending setup key).
### Checks (every 5 min)
```
@@ -170,7 +150,7 @@ description: >
|-------|------|-----------|
| **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis |
| **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
| **Home stack** | `/opt/home_stack/` | stirling-pdf, pulse (jdownloader decommissioned 2026-08-01 → dedicated CT 118 LXC) |
| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
| **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
@@ -187,15 +167,11 @@ description: >
- Compose: `/opt/home_stack/docker-compose.yml`
- Control script: `/opt/home_stack/infra-control.sh`
**JDownloader** (decommissioned from docker-vm 2026-08-01 — moved to dedicated CT 118 LXC):
- LXC: CT 118 on storepve, `192.168.68.20` (JDownloader + VNC 5900 + web UI 6080)
- Web UI: `http://192.168.68.20:6080` (noVNC via websockify)
- Docker container `jdownloader-2` on .7 removed; compose entry stripped
**JDownloader**:
- URL: `http://192.168.68.7:5800` (web UI via VNC)
**Pulse**:
- URL: `http://192.168.68.7:7655` (direct LAN)
- Public: `https://pulse.sysloggh.net` (NetBird CNAME proxy)
- Container: `rcourtman/pulse:5.1.35` in `/opt/home_stack` (port 7655, was 3001)
**Pulse** (Uptime Kuma):
- URL: `http://192.168.68.7:3001`
### Ecosystem B: CT 116 syslog-api (192.168.68.116)
@@ -378,18 +354,16 @@ fine. Services that resolve directly to a LAN IP are NetBird-independent.
|---------|--------|-------------|-------------|-------------|--------|
| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
| Authentik | auth.sysloggh.net:443 | 192.168.68.11:9000 | CNAME → netbird | **Yes** | ⚠️ |
| Gitea | git.sysloggh.net:443 | 192.168.68.17:3000 | CNAME → netbird | **Yes** | ⚠️ |
| Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
| Gitea | git.sysloggh.net:443 | 192.168.68.110 | CNAME → netbird | **Yes** | ⚠️ |
| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
| Pulse | pulse.sysloggh.net:443 | 192.168.68.7:7655 | CNAME → netbird | **Yes** | ✅ verified 2026-08-01 |
| DNS UI | dns.sysloggh.net:443 | 192.168.68.10:80 | CNAME → netbird | **Yes** | ⚠️ |
| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
| DNS UI | dns.sysloggh.net:443 | 192.168.68.102 | CNAME → netbird | **Yes** | ⚠️ |
| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
**Verified 2026-07-24:** NetBird VPS rebooted after a hang; all CNAME'd
**Verified 2026-07-02:** NetBird VPS rebooted after a hang; all CNAME'd
services recovered. LAN-IP-direct paths stayed up throughout the outage.
Also added `dns.sysloggh.net` route (was missing entirely).
See `scripts/netbird-add-domain.sh` for adding new proxy routes.
### 5.2 Checks (every 2 min)
@@ -537,8 +511,8 @@ enforced by the `routing-regression.config_url_violations` check in Section
| LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
| LiteLLM (nginx) | `http://192.168.68.116` | — |
| Grafana | `http://192.168.68.116:3001` | — |
| Authentik | `https://192.168.68.11:9000` | `https://auth.sysloggh.net` |
| Gitea | `http://192.168.68.17:3000` | `https://git.sysloggh.net` |
| Authentik | `https://192.168.68.11` | `https://auth.sysloggh.net` |
| Gitea | `http://192.168.68.110:3000` | `https://git.sysloggh.net` |
| Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` |
| SearXNG | `http://192.168.68.7:8888` | — |
| Firecrawl | `http://192.168.68.7:3002` | — |
@@ -601,25 +575,25 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| CT | Name | Node | IP | Role | Agent |
|----|------|------|----|------|-------|
| 100 | abiba | minipve | .24 | Pi agent | ✅ pi |
| 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi |
| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
| 102 | adguard | **minipve** | **.10** | DNS | ❌ |
| 102 | adguard | acerpve | — | DNS | ❌ |
| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
| 104 | authentik | minipve | .11 | OIDC | ❌ |
| 105 | kagentz | minipve | — | Agent Zero | ✅ |
| 105 | kagentz | amdpve | — | Agent Zero | ✅ |
| 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
| 107 | pbs | storepve | — | Backups | ❌ |
| 108 | media | storepve | — | Media | ❌ |
| 109 | docker-vm | storepve | .7 | Docker host | ❌ |
| 110 | gitea | minipve | **.17** | Git | ❌ |
| 110 | gitea | minipve | — | Git | ❌ |
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
| 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ |
| 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
| 113 | baggy | amdpve | ? | Hermes agent | ✅ |
| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
| 115 | scottdenya | amdpve | — | ? | ❌ |
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
| 117 | zulip | storepve | .19 | Chat | ❌ |
| 118 | jdownloader | storepve | .20 | JDownloader LXC (dedicated, migrated from docker-vm 2026-08-01) | ✅ |
| 119 | infisical-vault | minipve | — | Vault | ❌ |
| 117 | zulip | storepve | — | Chat | ❌ |
| 118 | jitsi | minipve | — | Video | ❌ |
## Appendix C: Docker Compose Files Location
@@ -639,27 +613,27 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
| CT | Name | Node | pct-run |
|-----|------|------|---------|
| 100 | abiba | minipve | `pct-run 100` |
| 105 | kagentz | minipve | `pct-run 105` |
| 100 | abiba | amdpve | `pct-run 100` |
| 105 | kagentz | amdpve | `pct-run 105` |
| 111 | tdunna | amdpve | `pct-run 111` |
| 112 | tanko | amdpve | `pct-run 112` |
| 113 | baggy | amdpve | `pct-run 113` |
| 115 | scottdenya | amdpve | `pct-run 115` |
| 104 | authentik | minipve | `pct-run 104` |
| 110 | gitea | minipve | `pct-run 110` |
| 114 | mumuni | minipve | `pct-run 114` |
| 116 | syslog-api | minipve | `pct-run 116` |
| 106 | ra-h-os | storepve | `pct-run 106` |
| 107 | proxmox-backup | storepve | `pct-run 107` |
| 108 | media | storepve | `pct-run 108` |
| 117 | zulip | storepve | `pct-run 117` |
| 102 | adguard | **minipve** | `pct-run 102` |
| 102 | adguard | acerpve | `pct-run 102` |
GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
```bash
ssh root@192.168.68.8 # RTX 3090
ssh root@192.168.68.110 # RTX 5070
ssh root@192.168.68.15 # Strix Halo
ssh root@192.168.68.4 # hwepve — standalone London node + NetBird routing peer (relocation pending)
```
## Section 7: Agent Health Check (consolidated — 2026-07-05)
+1
View File
@@ -140,6 +140,7 @@ After ALL updates (apt + images + restarts), verify every critical service is ba
| Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
| Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
| PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
| Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
+6 -55
View File
@@ -7,15 +7,13 @@ description: >
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
via existing /metrics Prometheus endpoint.
DEPLOYMENT STATUS (2026-08-09):
DEPLOYMENT STATUS (2026-07-09):
✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
(via proxmox-monitor contract). Grafana at :3001, all scrape targets active.
✅ GPU exporters DEPLOYED: all 3 GPU hosts (.8/.110/.15) run exporters on
:9400 (nvidia_gpu_exporter / amdgpu exporter) — verified 200 on 2026-08-09.
✅ LiteLLM /metrics scraping live (success_callback: prometheus; auth via
master key) + Alertmanager + Zulip bridge (alerts-infra) added 2026-08-09.
⚠️ This contract is target-state aspirational — but GPU export + alerting
are now as-built (verified 2026-08-09).
(via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
NVIDIA sidecar exporters (.8/.110:9400) never installed.
Router falls back to direct GPU /health probes.
⚠️ This contract is target-state aspirational — not as-built.
As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
version: 1.0.0
---
@@ -102,53 +100,6 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
- Stack persists across reboots (systemd for exporters, Docker restart policy)
## Execution
### check-health
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
```bash
# Zulip API health (POST ping)
source /etc/litellm-monitor.env
ZULIP_USER="abiba-bot@chat.sysloggh.net"
curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u "${ZULIP_USER}:${ZULIP_BOT_KEY}"
# Expected: 200 (HTTP 000 = unreachable/cache)
# PM2 process health
pm2 jlist
# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service)
# GPU exporters (may be down per DEPLOYMENT STATUS)
curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL"
curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL"
curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL"
# Router health (via nginx on port 80)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health
# Expected: 200 (Router is up and responding)
# LiteLLM health (via nginx on port 80)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health
# Expected: 200 (LiteLLM is up and responding)
# PVE API (401 expected for unauthenticated probe — API is up over https)
curl -s -o /dev/null -w '%{http_code}' https://192.168.68.116:8006/api2/json
# Expected: 401 (unauthorized — API is up; 000 = unreachable, 500 = API down)
# Prometheus targets
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
# Expected: All targets UP (may show some down if exporters not deployed)
# Grafana health
curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
# Expected: {"status":"ok","version":"..."}
# LiteLLM metrics (Prometheus endpoint)
curl -s http://192.168.68.116:4001/metrics | head -20
# Expected: Prometheus-formatted metrics output
```
**Report format**: Summarize actual results from each probe. If any probe returns non-200 or empty output, flag as alert.
### Phase 1: GPU Exporters
+8 -43
View File
@@ -59,7 +59,7 @@ Before ANY update wave:
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 105 (kagentz, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
@@ -77,7 +77,7 @@ Before ANY update wave:
|------|-------|---------|
| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Home stack (Pulse, Stirling PDF) — JDownloader moved to CT 118 LXC 2026-08-01 | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
@@ -91,52 +91,17 @@ Before ANY update wave:
- Firecrawl test: `curl :3002/`
- SearXNG test: `curl :8888`
## Wave 4: Proxmox Kernel Reboot
## Wave 4: Proxmox Kernel Reboot (if needed)
Only if `[ -f /var/run/reboot-required ]` on any node.
| Target | Action |
|--------|--------|
| Affected PVE node | Verify all CTs/VMs migrated or stopped |
| | `reboot` via PVE API (or `systemctl reboot -f` if dbus fails) |
| | `reboot` via PVE API |
| | Wait 120s for node to come back |
| | Start any stopped CTs |
### Post-reboot sweep (known gaps)
After every node reboot, run these checks:
1. **CT auto-start sweep** — LXC containers sometimes don't start despite
`onboot: 1`. Check every CT on the rebooted node and start any left stopped:
```bash
pct list | awk '/stopped/{print $1}' | xargs -I{} pct start {}
```
Known cases: scottdenya (CT 115 on amdpve), authentik (CT 104 on minipve).
2. **Zulip recovery** — When docker-vm or storepve reboots, the Zulip main
container loses its Docker network assignment (SIGKILL during storage
outage detaches it from `zulip_default` network). Run:
```bash
ssh root@192.168.68.19 'docker rm -f zulip-zulip-1 && cd /opt/zulip && docker compose up -d'
```
The compose restart recreates the container on the correct network.
3. **docker-vm Docker daemon** — After reboot, Docker can take 3-4 minutes
to become `active`. The docker-proxy for Pulse (port 7655) starts early,
so Pulse is accessible before `docker ps` reports ready. Wait for Docker
before checking other stacks.
### VPS ↔ docker-vm tunnel
After any VPS or docker-vm reboot, verify the dedicated WireGuard tunnel:
```bash
ssh root@72.61.0.17 'wg show wg1' | grep "latest handshake"
# If no handshake in >60s:
ssh root@72.61.0.17 'wg-quick up wg1'
```
The tunnel uses PersistentKeepalive=25 and is systemd-enabled, but should
be verified after a reboot.
## Rollback Protocol
If ANY verification fails:
@@ -159,12 +124,12 @@ Before Wave 1, snapshot these files:
/opt/home_stack/docker-compose.yml (VM 109 .7)
/opt/audiobookshelf/docker-compose.yml (VM 109 .7)
/root/.pi/agent/extensions/config.yaml (CT 100 .24)
/etc/systemd/system/strix-server.service (amdpve .15 — strix-moe)
/etc/systemd/system/ornith-server.service (amdpve .15 — strix-moe)
/etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110)
# Hermes agent configs (key enforcement — 2026-07-10)
/home/hermes/.hermes/config.yaml (Mumuni kagentz CT105; Tanko CT112 uses /home/jerome/.hermes)
/etc/systemd/system/hermes-gateway.service (Mumuni kagentz CT105 — system unit, User=hermes)
/etc/environment (LITELLM_API_KEY — legacy path, Mumuni now keys via Infisical)
/root/.hermes/config.yaml (Mumuni CT 114, Tanko CT 112, etc.)
/root/.config/systemd/user/hermes-gateway.service (Mumuni CT 114 — EnvironmentFile fixed)
/etc/environment (Mumuni CT 114 — LITELLM_API_KEY)
```
## MCP Gateway (2026-07-10)
+4 -46
View File
@@ -178,7 +178,7 @@ through its agent wrapper.
| Agent | Host | Pattern | Keys | Status |
|-------|------|---------|------|--------|
| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
| mumuni | .14 (kagentz CT105) | systemd unit hermes-gateway.service (user hermes) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
@@ -257,51 +257,9 @@ reads use per-agent identities. This eliminates the single shared token risk.
| Agent | .env Keys |
|-------|-----------|
| Mumuni | MUMUNI_LITELLM_API_KEY, MUMUNI_ZULIP_API_KEY |
|| Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY |
|| Koby | (wrapper injects from vault — .env has Telegram token) |
|| Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY |
|| Agent Zero (kagentz .14) | OPENROUTER_API_KEY (direct OpenRouter access) |
### Agent Zero (kagentz .14) — OpenRouter Integration (2026-09-01)
Agent Zero runs in Docker on kagentz (CT105) and uses **direct OpenRouter API access**,
not via the LiteLLM proxy. This is because Agent Zero's workflow (self-update manager,
UI bootstrap, model selection) is built around OpenRouter's native authentication.
**Key Storage:**
- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=sk-or-v1-…`)
- **Vault**: Infisical secret `OPENROUTER_API_KEY` (project=agents, env=production)
- **Fallback**: The container's .env is the primary source; vault sync is optional
(unlike fleet agents which require vault injection)
**Current Key (2026-09-01):**
- **Prefix**: `sk-or-v1-0af3f3…`
- **User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
- **Plan**: Paid (not free tier)
- **Usage**: 0 (as of 2026-09-01)
**Model Configuration:**
- **Preset**: "Cost Efficient" (`/a0/usr/plugins/_model_config/presets.yaml`)
- **Model**: `openrouter/moonshotai/kimi-k3`
- **API Base**: (empty — uses OpenRouter default)
**Why not LiteLLM proxy?**
Agent Zero's architecture was designed before the fleet adopted the LiteLLM proxy
standard. The container runs `/exe/self_update_manager.py` and `/a0/run_ui.py` which
directly call OpenRouter via Python's requests library. Converting would require:
1. Refactoring all LLM calls to use `litellm` library
2. Adding vault wrapper injection
3. Updating self_update_manager to use proxy-aware key handling
**Rotation Procedure:**
1. Generate new key in OpenRouter UI
2. Update container: `sed -i 's/^API_KEY_OPENROUTER=.*/API_KEY_OPENROUTER=<new_key>/' /a0/usr/.env`
3. Update vault: `infisical secrets set OPENROUTER_API_KEY=<new_key> --projectId=agents --env=production`
4. Restart container: `sudo docker exec agent-zero supervisorctl restart run_ui`
5. Verify: `curl -s https://openrouter.ai/api/v1/auth/key -H "Authorization: Bearer <new_key>"`
**Related Contract:**
- `agent-zero-openrouter-key.prose.md` — Full agent-zero key management contract
| Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY |
| Koby | (wrapper injects from vault — .env has Telegram token) |
| Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY |
## Key Rotation Log
-117
View File
@@ -1,117 +0,0 @@
---
kind: pattern
name: litellm-client-timeouts
description: >
Standard client timeout and retry policy for ALL agents calling LiteLLM
(CT 116, http://192.168.68.116). Created 2026-09-08 after the Sep 6 incident:
a backend stall 04:00-06:30 EDT produced 157 client-abandoned 408 failures
(82% from abiba-pi, 13 from mumuni) against a backend that was actually
succeeding at 20-70s per call once clients stopped giving up. Grounded in
measured data: syslog-auto (LiteLLM virtual model, no dedicated GPU — routes
to backends) averages 28.8s/request with 25.4s TTFT over 888 calls/24h;
nginx already allows 600s (Rule 5, verified 2026-08-09); the gap is entirely
client-side. Blast radius if wrong: agents fall back to DeepSeek silently
(key/timeout failures present as model degradation, not errors) or abandon
healthy-but-slow reasoning calls, fragmenting long tasks.
---
## Maintains
- client_timeout_standard: "litellm-client-timeouts v1.0 (2026-09-08)"
- applies_to: ALL agents and scripts calling http://192.168.68.116 (main model path, auxiliary tasks, health probes, benchmark jobs)
- verified_against: Prometheus litellm_* metrics 24h window ending 2026-09-08 ~10:00 EDT; live probe (syslog-auto tiny call 0.56s TTFB, 200 OK); nginx 600s proxy_read_timeout (Rule 5)
## The measured numbers these values come from
| Model | avg latency | avg TTFT | p-profile (24h) |
|---|---|---|---|
| syslog-auto | 28.8s | 25.4s | 68 calls took 30-120s; tail to ~300s under load |
| qwen3.6-27B-code | 23.0s | — | same backend class as syslog-auto |
| strix-moe | 7.5s | — | Strix Halo, healthy |
| gemma-4-12b | 2.6s | — | RTX 5070, healthy |
Sep 6 incident timeline: failures 04:00-07:00 EDT (0% GPU util = wedged
backend), full recovery 07:00-08:00 with ZERO client failures once requests
tolerated 20-70s — 392 successful slow calls in the three hours after recovery.
LiteLLM's internal queue time is ~0s; the latency is model inference, not
proxy queuing.
## Parameters
### 1. Primary model path (model.default / custom_providers) — NO client timeout below 300s
- The default Hermes HTTP timeout (~60s) is TOO SHORT for syslog-auto's healthy
28.8s average + 120-300s tail. Every 408 in the incident was a client
abandoning a request the backend would have answered.
- If the transport exposes a timeout setting for the main model, set it to
**300s or more**. If it does not (current Hermes custom-provider path has no
timeout knob), that is acceptable ONLY because nginx holds the request for
600s — but any wrapper, script, or direct API call you write MUST set its own
timeout >= 300s for syslog-auto/qwen-class calls.
- Never hardcode a shorter timeout "to fail fast" on this path — failing fast
here is what caused the incident.
### 2. Auxiliary tasks — keep template timeouts, one correction
- vision: 60s (keep), web_extract: 30s (keep) — gemma-4-12b averages 2.6s;
these are fine.
- compression: 300s (keep — this was already raised from 60 per gpu-fleet).
- **gpu-dense delegation/x_search: set timeout >= 120s.** The RTX 3090
(qwen3.6-27B-code backend, 23.0s avg) is the same speed class as
syslog-auto; delegation defaults that assume fast responses will 408 the
same way.
### 3. Retry policy — backoff, not repetition
- On timeout (408) or 5xx: retry up to **2 times** with exponential backoff
(**15s, 45s**) before giving up.
- Do NOT retry in a tight loop. The Sep 6 spike shape (65 failures in one hour
from one key) was a batch job retrying without backoff while the backend was
down — it multiplied load during recovery.
- On 401/403: do NOT retry — that is a key/permission problem (see
litellm-api-keys.prose.md and Rule 11); retrying just spams the log.
- On 429: honor the retry-after header if present, else back off 60s.
### 4. Health probes — identify yourself and time out sanely
- Probes MUST NOT appear as keyless, model-less failures in the metrics (12
such orphans appeared in the incident window and cost investigation time).
Send a real model name and use a real (probe-designated) key.
- Probe timeout: 30s. A probe that takes longer than 30s IS the alert —
report "backend slow (>30s)" rather than hanging.
- Probe cadence: at most hourly. The 6h litellm-health cron cadence is the
standard; sub-hourly synthetic traffic distorts latency baselines.
### 5. Batch/benchmark jobs — schedule away from 04:00-07:00 EDT and chunk
- The incident window showed bulk clients amplifying a backend stall 5:1.
- Batch jobs that can tolerate delay: schedule 09:00-17:00 EDT.
- Any batch loop over N requests MUST sleep >= 5s between requests and honor
the retry policy in section 3.
## Returns
- A single standard any agent or script can cite: timeouts >= 300s on the
syslog-auto path, >= 120s on gpu-dense delegation, backoff retries (2x,
15s/45s), identified probes at 30s/hourly, batch jobs chunked and
day-scheduled.
- Failure signature recognition: bulk 408s from multiple keys in one window =
backend event (check gpu_utilization_percent: 0% = wedged, ~100% = saturated);
single-key 408s = that client's timeout is too short.
- Cross-references: hermes-config-template.prose.md (Rule 5 nginx 600s,
auxiliary timeouts), gpu-fleet.prose.md (compression 300s precedent,
stable aliases), litellm-api-keys.prose.md (key/permission failures).
## Intentionally NOT changed
- No server-side LiteLLM timeout/cooldown changes proposed — the incident
self-recovered and the server is healthy (0.56s live probe); changing
server behavior without process-level root cause (CT116 requires root;
not reachable from kagentz) would be guessing.
- No change to the template's vision/web_extract/compression timeouts —
measured data says they are correct.
- No per-agent key permission changes — those are litellm-api-keys.prose.md
territory (and the open gpu-vision/gemma 403 items are already filed with
the key owners).
- No model routing changes — syslog-auto's weighted pool behaved correctly
throughout the incident.
-9
View File
@@ -18,15 +18,6 @@ description: >
Designed as a reusable contract for any Syslog agent.
Source of truth: gpu-fleet.prose.md
## Monitoring / Alerting (as-built 2026-08-09)
- LiteLLM /metrics scrape job: requires `litellm_settings.success_callback: [prometheus]`
(failure_callback alone does NOT mount /metrics — verified 2026-08-09).
Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/.
- Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver
firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09.
- Prometheus node job covers ALL 5 PVE nodes (.5/.6/.9/.12/.15:9100).
---
## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
+8 -42
View File
@@ -1,6 +1,4 @@
---
report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
kind: responsibility
name: litellm-self-heal
status: deployed
@@ -9,7 +7,7 @@ note: >
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
now reimplemented as `litellm-health-check.sh` on CT 116.
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs).
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
GPU monitoring integrated from gpu-monitor on .24:9100.
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
@@ -22,8 +20,7 @@ description: >
LiteLLM inference stack health monitoring + self-healing. Verifies the full
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
inference, and agent keys. Applies remediation rules for common failures.
Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
---
Reports every action via Zulip DM and RA-H OS knowledge graph.
---
# LiteLLM Operations — Health Check + Self-Heal
@@ -68,15 +65,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
## LiteLLM Model Surface (ground truth — `/opt/inference-harness/litellm_config.yaml` on CT 116)
`model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`, `crew-auto` (new 2026-08-20).
### Context Cap Split (2026-08-20)
- **Abiba (firstmate)**: 128K uncapped — unlimited context for primary workloads
- **Hermes agents** (mumuni, tanko, koby, koonimo): 128K uncapped
- **Crewmates** (ops, tune, verify, auth-keys, build): 64K capped — alias `crew-auto` enforces 64K limit
Preferred implementation: uncap shared pool, add capped alias for crew-only.
`model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`.
- `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200).
- `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively.
@@ -113,7 +102,7 @@ Preferred implementation: uncap shared pool, add capped alias for crew-only.
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): reads each agent's **live** `LITELLM_API_KEY` from its gateway process env via SSH — never hardcodes keys (hardcoded keys rot on rotation and caused 9×401/30min). Fleet roster: abiba, tanko, mumuni, koby, koonimo (legacy `tdunna`/`baggy` removed — never existed).
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
## Maintains
@@ -131,7 +120,6 @@ Preferred implementation: uncap shared pool, add capped alias for crew-only.
- Also wakes on user request
- On failure: re-check after 30s, escalate after 3 consecutive failures
---
---
## Health Check
@@ -174,7 +162,6 @@ Determine overall_status from individual check results:
- "degraded" — 1-2 non-critical checks fail
- "down" — critical checks fail
---
---
## Remediation Rules
@@ -218,15 +205,14 @@ Escalate → if SSH access unavailable, send Zulip DM
Router no longer in path so Redis active counters are unused. Rule retained
for reference but inactive. If Redis issues occur, check harness-redis container.
---
---
## Reporting
Every remediation cycle produces a structured report:
### 1. Gitea Log Entry
Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph.
### 1. RA-H OS Knowledge Graph Node
Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
### 2. Zulip DM to Owner
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
@@ -241,7 +227,6 @@ top actions, uptime.
If a fix requires another agent (e.g., Authentik restart), relay sent
to responsible agent with full context.
---
---
## Execution
@@ -267,11 +252,9 @@ call report-generator
health: health
actions: actions
-- Phase 4: Log to Gitea (not knowledge graph — hard rule)
call gitea-logger
-- Phase 4: Log to knowledge graph
call kg-logger
run_id: run_id
repo: SyslogSolution/health-logs
path: litellm/{run_id}.json
health: health
actions: actions
@@ -317,20 +300,3 @@ With failures:
]
}
```
## gitea-logger Implementation
When this step executes, write the report JSON to a temp file and push to Gitea:
```bash
REPO="https://abiba-bot:${GITEA_PAT}@git.sysloggh.net/SyslogSolution/health-logs"
DIR="litellm"
FILE="${run_id}.json"
echo "${report_json}" > /tmp/${FILE}
(cd /tmp && git clone --depth 1 "${REPO}" &&
cp ${FILE} health-logs/${DIR}/${FILE} &&
cd health-logs && git add ${DIR}/${FILE} &&
git commit -m "litellm-health: ${run_id}" && git push)
rm -rf /tmp/health-logs /tmp/${FILE}
```
+5 -6
View File
@@ -1,12 +1,9 @@
---
report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
name: memory-audit-maintenance
kind: responsibility
description: Shared memory audit and maintenance contract for Hermes agents (Mumuni, Koby, Koonimo). Tanko is no longer a Hermes agent (now on DSH/DeepSeek Harness since 2026-08-27) and uses DSH-native memory, so it is excluded from this Hermes roster. Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
id: 067NC4KG01RG50R40M30E20918
---
---
### Goal
@@ -18,10 +15,11 @@ Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md,
### Scope
This contract is the **Hermes Agent standard** for memory maintenance. It is shared across Hermes agents (Mumuni, Koby, Koonimo). **Tanko is excluded — it migrated to DSH (DeepSeek Harness) on 2026-08-27 and now uses DSH-native memory (mnemon), not `~/.hermes/memories/`.** Each agent runs it against its own memory files only no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
**Agent Roster (Hermes):**
**Agent Roster:**
- Mumuni
- Tanko
- Koby (CT 111 / tdunna)
- Koonimo (CT 113 / baggy)
@@ -345,3 +343,4 @@ return {
### Per-Agent Notes
Each Hermes agent (Mumuni, Tanko, Tdunna, Baggy) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
-180
View File
@@ -1,180 +0,0 @@
---
report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
kind: pattern
name: memory-fixer
description: >
Auto-fix low-hanging fruit in the RA-H OS knowledge graph. No judgment calls — only deterministic Level 1 operations.
Escalate anything that needs Kwame's input. Executes confirmed Kwame decisions to completion (state + updated_at).
version: 2.0.0
---
---
# Memory Fixer
> **Canonical copy:** `/root/.hermes/contracts/memory-fixer-v3.md` (used by the `memory-fixer-daily` cron job). This file is the institutional record of the same contract. When the two diverge, treat the v3 source in `/root/.hermes/contracts/` as executable truth.
## Purpose
Auto-fix low-hanging fruit in the graph. No judgment calls — only deterministic Level 1 operations. Escalate anything that needs Kwame's input. When Kwame replies to an escalation, **execute the decision to completion** (update state and timestamps), never leaving a node in review-pending forever.
**Type:** Write-only (Level 1 fixes only)
**Scope:** RA-H OS knowledge graph (192.168.68.65)
**Schedule:** Daily at 8 AM ET
**Escalation:** Level 2+ to Kwame as task items
## Key Design Decision
The `updateNode` tool's `metadata` field performs a **restricted merge** — the `state` key only accepts `'processed'` or `'not_processed'`. Additionally, new metadata keys cannot be added via the merge.
**Solution:** Use the `description` field to tag stale nodes with review actions, since `description` is a simple string overwritable via `updateNode`.
**Tag Format:** `[REVIEW: action] original description text...`
Where `action` is one of:
- `archive` — node is stale and should be archived
- `refresh` — node is stale and should be refreshed (infrastructure)
- `keep` — node has been confirmed as current
- `merge` — node is a duplicate candidate
**Query for finding review-tagged nodes:**
```sql
SELECT id, title, description
FROM nodes
WHERE description LIKE '[REVIEW:%';
```
## Level 1 Auto-Fixes (No Kwame Decision Needed)
### 1. Missing `type` Auto-Classification
```sql
SELECT id, title,
CASE
WHEN title LIKE '%infrastructure%' OR title LIKE '%proxmox%' OR title LIKE '%setup%' THEN 'infrastructure'
WHEN title LIKE '%skill%' OR title LIKE '%how to%' OR title LIKE '%guide%' THEN 'skill'
WHEN title LIKE '%doc%' OR title LIKE '%template%' OR title LIKE '%brand%' THEN 'documentation'
WHEN title LIKE 'WAL:%' OR title LIKE 'TASK:%' THEN 'note'
WHEN title LIKE '%[LEARN]%' THEN 'documentation'
ELSE 'note'
END as auto_type
FROM nodes
WHERE json_extract(metadata, '$.type') IS NULL;
```
### 2. Missing `namespace` Auto-Population
```sql
SELECT id, title, json_extract(metadata, '$.tenant') as tenant
FROM nodes
WHERE json_extract(metadata, '$.namespace') IS NULL;
```
### 3. Staleness Review Tagging
Using the type-based windows from the memory-monitor contract, tag nodes stale beyond their window. **Only process a maximum of 10 nodes per run** to avoid overwhelming Kwame. Prioritize infrastructure first, then dynamic, then ephemeral.
**Exclusion Rules:**
- Nodes with `state` = `review_pending`, `deprecated`, `archived`, or `not_processed` are NOT processed
- Nodes whose `description` already starts with `[REVIEW:` are NOT re-processed
```sql
SELECT id, title, json_extract(metadata, '$.type') as node_type,
CAST(julianday('now') - julianday(updated_at) AS INTEGER) as days_stale,
CASE
WHEN json_extract(metadata, '$.type') IN ('infrastructure', 'deployment', 'system', 'system-health') THEN 'refresh'
WHEN json_extract(metadata, '$.type') IN ('business', 'philosophy', 'research', 'learning', 'learn', 'investigation', 'analysis', 'project') THEN 'refresh'
ELSE 'archive'
END as suggested_action
FROM nodes
WHERE updated_at < datetime('now',
CASE
WHEN json_extract(metadata, '$.type') IN ('infrastructure', 'deployment', 'system', 'system-health') THEN '-14 days'
WHEN json_extract(metadata, '$.type') IN ('skill', 'documentation', 'template', 'protocol-enforcement', 'prd', 'architecture') THEN '-90 days'
WHEN json_extract(metadata, '$.type') IN ('note', 'wal', 'WAL', 'task', 'TASK', 'event') THEN '-30 days'
WHEN json_extract(metadata, '$.type') IN ('business', 'philosophy', 'research', 'learning', 'learn', 'investigation', 'analysis', 'project') THEN '-120 days'
WHEN json_extract(metadata, '$.type') IN ('deprecated-relay', 'audit', 'audit-report', 'incident', 'incident-report') THEN '-3650 days'
ELSE '-45 days'
END
)
AND json_extract(metadata, '$.state') NOT IN ('review_pending', 'deprecated', 'archived', 'not_processed')
AND (description IS NULL OR description NOT LIKE '[REVIEW:%')
ORDER BY days_stale ASC
LIMIT 10;
```
For each identified node, call `updateNode(id, { description: "[REVIEW: action] " + originalDescription })`.
## Level 2 Escalations (Kwame Decision Required)
1. **Stale nodes** flagged with `[REVIEW: …]` — Archive, refresh, or keep?
2. **Duplicate Nodes** (same title or >70% title overlap) — Merge or keep?
3. **Orphan Nodes >90 days old** — Archive or connect?
## Reporting Format
The fixer reports to Kwame via this Zulip DM:
```
🦅 Memory Fixer — [HH:MM UTC]
Level 1 fixes applied:
- Missing type: X nodes classified
- Missing namespace: Y nodes populated
Stale nodes needing review (max 10):
1. [Node #XXX] Title — X days stale, SUGGEST: refresh
2. [Node #YYY] Title — Y days stale, SUGGEST: archive
...
Duplicates needing decision:
1. [Node #AAA] vs [Node #BBB] — Same title
Orphans >90 days:
1. [Node #EEE] Title — X days stale, orphaned
Reply with:
- "archive #XXX, #YYY" to mark for archive
- "archive all" to archive all stale nodes listed
- "keep #XXX" to confirm a node is current
- "merge #AAA into #BBB" to merge duplicates
- "refresh #XXX" to mark as current
```
## Execution on Next Run
The fixer reads Kwame's previous response and **executes the decision to completion** — it must not leave a node in review-pending forever. Tagging alone is NOT enough; each confirmed decision must also update `state` and `updated_at` so the node drops out of the stale window on the next run.
> ⚠️ `updateNode` cannot set `state` to non-standard values (restricted to `processed`/`not_processed`) and cannot add metadata keys. For state transitions and `updated_at` bumps, use **direct SSH + SQLite** on the bridge host:
> ```bash
> ssh root@192.168.68.65 "sqlite3 /root/.local/share/RA-H/db/rah.sqlite \"UPDATE nodes SET metadata = json_set(metadata, '$.state', '<state>'), updated_at = datetime('now') WHERE id = <id>;\""
> ```
> Use `updateNode` only for description/source/title/link edits.
Decision → completed action mapping:
| Kwame reply | Description change | State | `updated_at` |
|---|---|---|---|
| `archive #XXX` | replace `[REVIEW: archive] ` → `[ARCHIVED] ` prefix | `archived` | bumped to now |
| `keep #XXX` / `refresh #XXX` | **clear the `[REVIEW: …]` tag entirely** | `active` | bumped to now |
| `merge #AAA into #BBB` | set `[REVIEW: merge_into #BBB]` on #AAA, then follow manual merge workflow | handled manually | bumped to now |
| `archive all` | apply the archive row to every node listed in the prior report | `archived` | bumped to now |
**Why `updated_at` must be bumped (critical):** the Level-1 staleness query keys off `updated_at < now - window`. If the fixer clears the tag but leaves a stale `updated_at`, the node is immediately re-flagged on the very next run and the cycle repeats forever. Bumping `updated_at` to now pushes the node back to the front of the window.
**Exclusion after action:** once an action is applied, the node's description no longer starts with `[REVIEW:` (archive → `[ARCHIVED]`, refresh/keep → original text), so it is not re-processed.
After all actions are applied, verify with:
```sql
SELECT id, json_extract(metadata, '$.state') FROM nodes WHERE description LIKE '[REVIEW:%';
```
The result must be 0 rows when all decisions are executed. Report what was done.
## Checks
- **State integrity:** archived nodes have `state: archived` + `[ARCHIVED]` prefix; kept nodes are `state: active` without a `[REVIEW:]` tag.
- **No review-pending forever:** after executing Kwame's decisions, `[REVIEW:%` node count must be 0.
- **Timestamps:** every executed decision bumps `updated_at`, so the node exits the stale window on the next run.
## Logging
Every Level 1 fix logged to `~/.hermes/logs/memory-fixer/YYYY-MM-DD.md`
Every Level 2 escalation logged and delivered to Kwame.
+3 -3
View File
@@ -6,7 +6,7 @@ description: >
delegation, verification, and delivery. Defines when to delegate, which
worker to use for what, how to handle failures, and the kanban board
protocol. Enforces context-window discipline and separation of concerns.
Runs on Mumuni (kagentz CT105, minipve, .14) via Hermes agent (Zulip gateway via systemd).
Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent.
version: 1.0.0
---
@@ -19,8 +19,8 @@ version: 1.0.0
## Topology
**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
**Manager:** Mumuni (kagentz CT105, minipve, .14) via Hermes agent
**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
**Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
This contract is infrastructure-agnostic in terms of which nodes are used.
+1 -1
View File
@@ -54,7 +54,7 @@ garbled input. When a user types these commands to Abiba:
- **`/approve`** → "Pi doesn't have pending approvals. Commands execute immediately."
- **`/approve session`** → Same response
- **`/deny`** → Same response + "For agents (Mumuni on Hermes, Tanko on DSH), these work with their built-in approval system."
- **`/deny`** → Same response + "For Hermes agents (Tanko, Mumuni), these work with their built-in approval system."
This keeps the UX consistent across agents — users can type `/approve` anywhere
without getting confused by LLM responses.
+12 -13
View File
@@ -2,17 +2,21 @@
kind: responsibility
name: pm2-self-heal
description: >
Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and
auto-restarts any that are stopped or errored. Logs every action to
the knowledge graph and alerts the owner via Zulip DM on failures.
CRITICAL: Never restart abiba-zulip — it runs this contract.
---
## Maintains
- abiba-telegram: { status: "online", uptime: string, restarts: number }
- abiba-zulip: { status: "online", uptime: string, restarts: number }
- gpu-watchdog: { status: "online", uptime: string, restarts: number }
- gpu-monitor: { status: "online", uptime: string, restarts: number }
- gitea-runner: { status: "online", uptime: string, restarts: number }
- spoton-service: { status: "online", uptime: string, restarts: number }
- zulip-watchdog: { status: "online", uptime: string, restarts: number }
- last_check: timestamp
> **Note (2026-07-04):** `abiba-zulip` removed — Zulip extension decommissioned.
## Continuity
@@ -27,15 +31,10 @@ description: >
- **Verify**: Re-check status after 5 seconds
- **Escalate**: If still failed after 2 retries, send Zulip DM to owner
### Rule 2: Process Restarting Too Often (crash-loop guard)
- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter) **or** a
process reporting a restart count > 1000 while showing "online" (a crash-loop mask)
### Rule 2: Process Restarting Too Often
- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter)
- **Note**: PM2 counter never decrements; only full delete+re-add resets it
- **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
- **Script guard (2026-08-16)**: `scripts/pm2-self-heal.sh` now restarts `abiba-telegram`
when `TEL_RESTARTS > 1000` even if the process reports "online" — catches a quiet
crash-loop that never toggles status to "stopped"/"errored" (e.g. the 10k-restarts
spoton incident). Alerts include the restart count.
- **Escalate**: Only when restarts > 30 — alerts to Zulip DM
- **Historical fix**: Previous cycles were caused by abiba-zulip extension's
stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2).
@@ -49,11 +48,11 @@ description: >
- If status is "online" → pass
- If status is "stopped" or "errored" → apply Rule 1
- If restarts > 5 → alert owner
3. **Check abiba-zulip** (live Zulip bridge, heartbeating):
3. **Check abiba-zulip** (self-process, read-only):
- If status is "online" → pass, log restarts count
- If status is "stopped" or "errored" → restart (`pm2 restart abiba-zulip` — fully restored)
- If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
- If restarts > 5 in last hour → alert owner with full diagnostics
4. **Log results** — Append to `SyslogSolution/health-logs/pm2/{timestamp}.md` in Gitea (not knowledge graph — hard rule)
4. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken
5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
6. **Wait 5 min** → repeat from step 1
+7 -32
View File
@@ -5,7 +5,7 @@ description: >
Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
agent: abiba
@@ -33,8 +33,8 @@ agent: abiba
| Exporter | Host:Port | Scope | Notes |
|----------|-----------|-------|-------|
| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
| prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
| node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. |
| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
## PVE API Token
@@ -48,7 +48,7 @@ agent: abiba
| UID | Title | Panels | Source |
|-----|-------|--------|--------|
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
@@ -77,9 +77,9 @@ agent: abiba
| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config |
## Cluster "Tabiri" — 5 Nodes
## Cluster "Tabiri" — 6 Nodes
| Node | IP | Role |
|------|----|----|
@@ -88,6 +88,7 @@ agent: abiba
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
| minipve | 192.168.68.12 | PVE |
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) |
| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 |
## Operations
@@ -100,32 +101,6 @@ Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
### check-targets
`curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'`
### check-health
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
```bash
# Prometheus health (bound to 0.0.0.0:9090 on .116)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9090/-/healthy
# Expected: 200 (Prometheus is up and healthy)
# Grafana health (bound to 0.0.0.0:3001 on .116)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:3001/api/health
# Expected: 200 (Grafana is up and healthy)
# Docker Stats exporter (bound to 127.0.0.1:9324 on .116 — must probe from .116 localhost)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9324/metrics"
# Expected: 200 (docker-stats-exporter is up and responding)
# PVE exporter (bound to 127.0.0.1:9221 on .116 — must probe from .116 localhost)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9221/metrics"
# Expected: 200 (pve-exporter is up and responding)
```
**Report format**: Summarize actual results from each probe. If any probe returns non-200, flag as alert.
**Note**: Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only) so they must be probed from .116 via SSH. Prometheus and Grafana are bound to 0.0.0.0 so they can be probed from the LAN.
### restart-exporter
`cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
+63 -338
View File
@@ -1,10 +1,10 @@
#!/usr/bin/env python3
"""
/root/scripts/agent-health-check.py — Consolidated Agent Health Verification v2
/root/scripts/agent-health-check.py — Consolidated Agent Health Verification
Verifies: LiteLLM keys (agent-specific), GPU port conflicts, agent Zulip streaming,
gateway liveness, gateway log health, CT liveness, config YAML integrity,
wrapper/CLI integrity, vault secret non-emptiness. NEVER restarts anything.
Single non-disruptive health check replacing 7 scattered scripts.
Verifies: LiteLLM keys, GPU port conflicts, agent Zulip streaming,
gateway liveness, and gateway log health. NEVER restarts anything.
Usage:
python3 /root/scripts/agent-health-check.py # Full check
@@ -12,62 +12,68 @@ Usage:
python3 /root/scripts/agent-health-check.py --quiet # Only output on failure
Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet
Changelog:
v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114),
abiba (.24).
"""
import subprocess, json, sys, os, time
from datetime import datetime
LITELLM = "http://192.168.68.116:80"
INFISICAL_PROJECT = "322fceab-39da-4854-a55a-568e76c0f13f"
INFISICAL_ENV = "prod"
# PVE node IPs for CT liveness checks
PVE_NODES = {
"amdpve": "192.168.68.15",
"minipve": "192.168.68.12",
"storepve": "192.168.68.6",
"acerpve": "192.168.68.9",
"ocupve": "192.168.68.5",
}
def _get_agent_key(agent_name):
"""Retrieve agent key from Infisical vault."""
try:
result = subprocess.run(
["infisical", "secrets", "get", "LITELLM_API_KEY",
"--project=agents", "--env=production", "--plain"],
capture_output=True, text=True, timeout=10
)
if result.returncode == 0:
return result.stdout.strip()
except Exception:
pass
# Agent definitions: ct, host, user, pve_node, vault_key_name
# Fallback: try exporting all secrets
try:
result = subprocess.run(
["infisical", "export", "--project=agents", "--env=production",
"--format=dotenv"],
capture_output=True, text=True, timeout=10
)
if result.returncode == 0:
for line in result.stdout.splitlines():
if line.startswith(f"LITELLM_API_KEY_{agent_name.upper()}") or \
(line.startswith("LITELLM_API_KEY=") and agent_name == os.uname().nodename):
return line.split("=", 1)[1].strip().strip('"').strip("'")
except Exception:
pass
return None
# Agent keys are pulled from Infisical vault at runtime.
# The 'key' field is populated dynamically below.
AGENTS = {
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
"abiba": {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "minipve", "vault_key": None}, # Pi agent + Mumuni Zulip, no vault key
"koby": {"ct": 111, "host": "192.168.68.129", "user": "root", "pve": "amdpve", "vault_key": "KOBY_LITELLM_API_KEY"},
"koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root", "pve": "amdpve", "vault_key": "KOONIMO_LITELLM_API_KEY"},
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome"},
"mumuni": {"ct": 114, "host": "192.168.68.123", "user": "root"},
"koby": {"ct": 111, "host": None, "user": None},
"koonimo": {"ct": 113, "host": None, "user": None},
}
# Inject keys from vault
for agent_name in AGENTS:
key = _get_agent_key(agent_name)
if key:
AGENTS[agent_name]["key"] = key
else:
AGENTS[agent_name]["key"] = None
GPU_HOSTS = {
"gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-server"},
"gpu-rtx5070 (.110)": {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
"gpu-strixhalo (.15)": {"host": "192.168.68.15", "port": 8080, "service": "strix-server"},
"gpu-strixhalo (.15)": {"host": "192.168.68.15", "port": 8080, "service": "ornith-server"},
}
FAIL = []
INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
# Fallback: if no env token, read the shared vault token file
if not INFISICAL_TOKEN:
_token_path = os.path.expanduser("~/.infisical-token")
if os.path.isfile(_token_path):
try:
with open(_token_path) as _f:
INFISICAL_TOKEN = _f.read().strip()
except (OSError, UnicodeDecodeError):
pass
# ── Helpers ──────────────────────────────────────────────────────────
def ssh(host, cmd, user="root"):
"""Execute a command on a remote host, return stdout or None."""
try:
@@ -106,81 +112,15 @@ def http_json(url, headers=None, timeout=5):
except:
return None
def run_infisical(args, quiet=True):
"""Run infisical CLI with env-based auth, return stdout or None."""
env = os.environ.copy()
env["INFISICAL_API_URL"] = INFISICAL_API_URL
if INFISICAL_TOKEN:
env["INFISICAL_TOKEN"] = INFISICAL_TOKEN
try:
result = subprocess.run(
["/usr/bin/infisical"] + args,
capture_output=True, text=True, timeout=15, env=env
)
return result.stdout.strip() if result.returncode == 0 else None
except:
return None
# ── KEY LOOKUP FIX ───────────────────────────────────────────────────
def _get_agent_key(agent_name, vault_key_name):
"""Retrieve agent-specific key from Infisical vault.
Uses {NAME}_LITELLM_API_KEY format (e.g., TANKO_LITELLM_API_KEY,
KOONIMO_LITELLM_API_KEY) which matches actual vault key names.
"""
if not vault_key_name:
return None
# Primary: get the agent-specific key by name
key = run_infisical([
"secrets", "get", vault_key_name,
"--projectId=" + INFISICAL_PROJECT,
"--env=" + INFISICAL_ENV,
"--plain",
])
if key and key.startswith("sk-"):
return key
# Fallback: export all and search for the key name
try:
export = run_infisical([
"export",
"--projectId=" + INFISICAL_PROJECT,
"--env=" + INFISICAL_ENV,
"--format=dotenv",
])
if export:
for line in export.splitlines():
if line.startswith(vault_key_name + "="):
value = line.split("=", 1)[1].strip().strip('"').strip("'")
if value.startswith("sk-"):
return value
except:
pass
return None
# Inject keys from vault for each agent
for agent_name in AGENTS:
info = AGENTS[agent_name]
key = _get_agent_key(agent_name, info.get("vault_key"))
AGENTS[agent_name]["key"] = key
# ═══════════════════════════════════════════════════════════════════
# CHECK 1: LiteLLM Key Validation (agent-specific keys)
# CHECK 1: LiteLLM Key Validation
# ═══════════════════════════════════════════════════════════════════
def check_keys():
for name, agent in AGENTS.items():
key = agent.get("key")
if not key:
print(f" ❌ {name}: NO KEY FOUND (vault empty or unreachable)")
FAIL.append(f"key:{name}:no-key")
continue
data = http_json(f"{LITELLM}/v1/models",
headers={"Authorization": f"Bearer {key}"})
headers={"Authorization": f"Bearer {agent['key']}"})
if data and data.get("data"):
model = data["data"][0].get("id", "?")
print(f" ✅ {name}: key valid → {model}")
@@ -190,7 +130,7 @@ def check_keys():
# ═══════════════════════════════════════════════════════════════════
# CHECK 2: GPU Port Conflict Detection (unchanged)
# CHECK 2: GPU Port Conflict Detection
# ═══════════════════════════════════════════════════════════════════
def check_gpu_ports():
@@ -218,6 +158,7 @@ def check_gpu_ports():
else:
print(f" ⚠️ {label}: svc={svc_status}, port owned by {port_owner}")
else:
# Verify health endpoint
health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
if health and '"status":"ok"' in health:
print(f" ✅ {label}: healthy (pid={port_owner})")
@@ -230,7 +171,7 @@ def check_gpu_ports():
# ═══════════════════════════════════════════════════════════════════
# CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents)
# CHECK 3: Agent Gateway Liveness + Streaming
# ═══════════════════════════════════════════════════════════════════
def check_agents():
@@ -238,36 +179,17 @@ def check_agents():
host = agent.get("host")
user = agent.get("user")
ct = agent["ct"]
report_only = agent.get("report_only", False)
# Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — it no longer runs a
# Hermes gateway, so skip the Hermes gateway/state/streaming/journal checks.
if agent.get("runtime") == "dsh":
live = ssh(host, "true", user=user)
print(f" {'✅' if live is not None else '❌'} {name}: DSH (DeepSeek Harness) — "
f"no Hermes gateway since 2026-08-27 (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
if live is None:
FAIL.append(f"unreachable:{name}")
continue
if not host or not user:
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
continue
# ⛔ KOBY IS NEVER REPAIRED — diagnostic only
if report_only:
print(f" 🔍 {name}: REPORT-ONLY mode (diagnostic only, no repairs on .129)")
# Still check gateway status for reporting purposes
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
if not pid:
pid = ssh(host, "pgrep -f 'hermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
if not pid:
print(f" ⚠️ {name}: GATEWAY NOT RUNNING (reported only)")
FAIL.append(f"gateway-down:{name}")
continue
else:
print(f" ✅ {name}: gateway running (pid={pid}, report-only mode)")
continue # Skip the rest of the check for Koby
# Gateway process (exclude the infisical bash wrapper that contains the same string)
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
if not pid:
print(f" ❌ {name}: GATEWAY NOT RUNNING")
FAIL.append(f"gateway-down:{name}")
continue
# Gateway state file
state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
@@ -281,7 +203,7 @@ def check_agents():
else:
gw_state, zulip = "no-state-file", "?"
# Zulip streaming check
# Zulip streaming: does adapter have edit_message?
adapter_paths = [
"~/.hermes/plugins/zulip-platform/adapter.py",
"~/.hermes/plugins/platforms/zulip/adapter.py",
@@ -298,191 +220,13 @@ def check_agents():
r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
user=user)
recent_errors = (recent_errors or "0").strip().split("\n")[-1]
recent_errors = (recent_errors or "0").strip().split("\n")[-1] # take last line
print(f" {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 4: CT Liveness (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_ct_liveness():
"""Check that all agent CTs are running on their PVE nodes."""
for name, agent in AGENTS.items():
ct = agent["ct"]
pve_node = agent.get("pve")
if not pve_node:
print(f" ⬜ {name} (CT {ct}): no PVE node mapped — skip")
continue
pve_ip = PVE_NODES.get(pve_node)
if not pve_ip:
print(f" ⬜ {name}: unknown PVE node '{pve_node}' — skip")
continue
status = ssh(pve_ip, f"pct status {ct} 2>/dev/null", user="root")
if not status:
print(f" ❌ {name} (CT {ct} on {pve_node}): PVE UNREACHABLE")
FAIL.append(f"ct-unreachable:{name}:{pve_ip}")
elif "running" in status:
print(f" ✅ {name} (CT {ct} on {pve_node}): running")
elif "stopped" in status:
print(f" ❌ {name} (CT {ct} on {pve_node}): STOPPED")
FAIL.append(f"ct-stopped:{name}")
else:
print(f" ⚠️ {name} (CT {ct} on {pve_node}): {status.strip()}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 5: Config YAML Integrity (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_config_integrity():
"""Verify agent config.yaml parses as valid YAML."""
for name, agent in AGENTS.items():
# Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no Hermes config.yaml.
if agent.get("runtime") == "dsh":
print(f" ⏭️ {name}: DSH — no Hermes config.yaml since 2026-08-27")
continue
host = agent.get("host")
user = agent.get("user")
if not host or not user:
print(f" ⬜ {name}: cannot SSH — skip config check")
continue
# Check YAML parses
yaml_ok = ssh(host,
"python3 -c "
'"import yaml; yaml.safe_load(open(\'/root/.hermes/config.yaml\')); print(\'OK\')" '
"2>&1 || echo 'FAIL'",
user=user)
if not yaml_ok:
print(f" ❌ {name}: SSH UNREACHABLE (config check skipped)")
FAIL.append(f"config-unreachable:{name}")
elif "OK" in yaml_ok:
print(f" ✅ {name}: config.yaml valid YAML")
else:
print(f" ❌ {name}: config.yaml YAML ERROR — {yaml_ok[:120]}")
FAIL.append(f"config-yaml-error:{name}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 6: Wrapper/CLI Integrity (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_wrapper_integrity():
"""Verify the hermes CLI wrapper exists and can reach hermes-real."""
for name, agent in AGENTS.items():
# Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no hermes CLI wrapper.
if agent.get("runtime") == "dsh":
print(f" ⏭️ {name}: DSH — no hermes CLI wrapper since 2026-08-27")
continue
host = agent.get("host")
user = agent.get("user")
if not host or not user:
print(f" ⬜ {name}: cannot SSH — skip wrapper check")
continue
# Check wrapper exists
wrapper = ssh(host, "ls -la /root/.local/bin/hermes 2>/dev/null", user=user)
if not wrapper:
# Check alternate wrapper locations
wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
if not wrapper:
print(f" ❌ {name}: NO HERMES CLI WRAPPER FOUND")
FAIL.append(f"wrapper-missing:{name}")
continue
else:
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
# Check wrapper has correct infisical path
infisical_path_valid = ssh(host,
"head -20 /root/.local/bin/hermes 2>/dev/null | grep -q '/usr/bin/infisical' && echo OK || echo MISS",
user=user)
if infisical_path_valid == "MISS":
# Check if infisical exists on path
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
if not inf_actual:
print(f" ❌ {name}: INFISICAL NOT INSTALLED (wrapper broken)")
FAIL.append(f"wrapper-no-infisical:{name}")
else:
print(f" ⚠️ {name}: wrapper infisical path may be wrong (infisical at {inf_actual})")
FAIL.append(f"wrapper-infisical-path:{name}")
# Check hermes-real exists
hermes_real = ssh(host,
"ls -la /root/.local/bin/hermes-real 2>/dev/null || echo MISS",
user=user)
if not hermes_real or hermes_real.strip() == "MISS":
# Check venv path
hermes_real = ssh(host,
"ls -la /usr/local/lib/hermes-agent/venv/bin/hermes 2>/dev/null || echo MISS",
user=user)
if not hermes_real or hermes_real.strip() == "MISS":
print(f" ❌ {name}: hermes-real NOT FOUND (wrapper broken)")
FAIL.append(f"wrapper-no-hermes-real:{name}")
else:
print(f" ✅ {name}: hermes-real at alt path")
# Check the .env file has the key
env_has_key = ssh(host,
"grep -c 'LITELLM_API_KEY' /root/.hermes/.env 2>/dev/null || echo 0",
user=user)
if env_has_key and env_has_key.strip() not in ("", "0"):
print(f" ✅ {name}: wrapper + .env key present")
else:
print(f" ⚠️ {name}: .env may be missing LITELLM_API_KEY entry")
# ═══════════════════════════════════════════════════════════════════
# CHECK 7: Vault Secret Non-Emptiness (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_vault_secrets():
"""Verify agent-specific vault secrets are non-empty and start with sk-."""
for name, agent in AGENTS.items():
vault_key_name = agent.get("vault_key")
if not vault_key_name:
continue
key = agent.get("key")
if not key:
print(f" ❌ {name}: vault secret {vault_key_name} MISSING or EMPTY")
FAIL.append(f"vault-empty:{name}:{vault_key_name}")
elif not key.startswith("sk-"):
print(f" ❌ {name}: vault secret {vault_key_name} WRONG FORMAT (starts '{key[:8]}...')")
FAIL.append(f"vault-bad-format:{name}:{vault_key_name}")
else:
print(f" ✅ {name}: vault {vault_key_name}=sk-...{key[-4:]}")
# ═══════════════════════════════════════════════════════════════════
# DEPLOY: copy updated script to /root/scripts/ on local host
# ═══════════════════════════════════════════════════════════════════
def deploy_self():
"""Copy this script to /root/scripts/agent-health-check.py if out of date."""
dest = "/root/scripts/agent-health-check.py"
try:
with open(__file__, "r") as f:
current = f.read()
if os.path.isfile(dest):
with open(dest, "r") as f:
existing = f.read()
if current == existing:
return # Already deployed
# Write new version
with open(dest, "w") as f:
f.write(current)
os.chmod(dest, 0o755)
print(f" 📦 Deployed updated script to {dest}")
except:
pass # Not fatal if deploy fails
# ═══════════════════════════════════════════════════════════════════
# MAIN
# ═══════════════════════════════════════════════════════════════════
@@ -491,12 +235,8 @@ def main():
quiet = "--quiet" in sys.argv
as_json = "--json" in sys.argv
# Self-deploy to canonical location
if not quiet and "--no-deploy" not in sys.argv:
deploy_self()
if not quiet:
print(f"🏥 Agent Health Check v2 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
print(f"🏥 Agent Health Check — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
print()
print("🔑 LiteLLM Keys:")
@@ -509,26 +249,11 @@ def main():
print("🤖 Agent Gateways:")
check_agents()
print()
print("🖥️ CT Liveness:")
check_ct_liveness()
print()
print("📝 Config Integrity:")
check_config_integrity()
print()
print("🔌 Wrapper/CLI Integrity:")
check_wrapper_integrity()
print()
print("🔐 Vault Secrets:")
check_vault_secrets()
if FAIL:
print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
if quiet:
# In quiet mode, only print failures as a single alert line
print(f"ALERT agent-health:{','.join(FAIL)}")
elif not quiet:
print("\n✅ All checks passed")
+19 -14
View File
@@ -296,20 +296,25 @@ def collect():
"pm2_uptime": pm2.get("uptime", "?"),
}
# Tanko (CT 112, IP 192.168.68.122) — DSH (DeepSeek Harness), no Hermes gateway
# since 2026-08-27. There is no ~/.hermes/gateway_state.json on CT 112 anymore;
# Zulip/Telegram connectivity is managed by the DSH harness, not the Hermes gateway.
# Tanko (CT 122)
tanko_state = ssh_jerome("192.168.68.122", "cat ~/.hermes/gateway_state.json 2>/dev/null")
tanko_data = {}
try:
tanko_data = json.loads(tanko_state) if tanko_state else {}
except:
tanko_data = {}
platforms = tanko_data.get("platforms", {})
report["agents"]["tanko"] = {
"platform": "dsh", "ct": 112, "ip": "192.168.68.122",
"gateway_state": "n/a (DSH)",
"zulip_state": "unknown",
"telegram_state": "unknown",
"gateway_pid": None,
"updated_at": "",
"platform": "hermes", "ct": 112, "ip": "192.168.68.122",
"gateway_state": tanko_data.get("gateway_state", "unknown"),
"zulip_state": platforms.get("zulip", {}).get("state", "unknown"),
"telegram_state": platforms.get("telegram", {}).get("state", "unknown"),
"gateway_pid": tanko_data.get("pid"),
"updated_at": tanko_data.get("updated_at"),
}
# Mumuni (CT 100, IP 192.168.68.24)
mumuni_state = ssh("192.168.68.24", "cat ~/.hermes/gateway_state.json 2>/dev/null")
# Mumuni (CT 114, IP 192.168.68.123)
mumuni_state = ssh("192.168.68.123", "cat ~/.hermes/gateway_state.json 2>/dev/null")
mumuni_data = {}
try:
mumuni_data = json.loads(mumuni_state) if mumuni_state else {}
@@ -317,7 +322,7 @@ def collect():
mumuni_data = {}
mumuni_platforms = mumuni_data.get("platforms", {})
report["agents"]["mumuni"] = {
"platform": "hermes", "ct": 100, "ip": "192.168.68.24",
"platform": "hermes", "ct": 114, "ip": "192.168.68.123",
"gateway_state": mumuni_data.get("gateway_state", "unknown"),
"telegram_state": mumuni_platforms.get("telegram", {}).get("state", "unknown"),
"zulip_state": mumuni_platforms.get("zulip", {}).get("state", "not_installed"),
@@ -325,7 +330,7 @@ def collect():
"hermes_version": "",
}
# Get Hermes version
ver = ssh("192.168.68.24", "hermes --version 2>/dev/null | head -1")
ver = ssh("192.168.68.123", "hermes --version 2>/dev/null | head -1")
if ver:
report["agents"]["mumuni"]["hermes_version"] = ver.split("·")[0].replace("Hermes Agent ","").strip()
@@ -538,7 +543,7 @@ th {{ color: #8b949e; font-weight: normal; }}
elif name == "tanko":
zulip_state = "✅" if agent.get("zulip_state") == "connected" else ("❌" if agent.get("zulip_state") == "disconnected" else "⬜")
gateway = agent.get("gateway_state", "?")
processed = "DSH"
processed = agent.get("updated_at", "")[:10]
elif name == "mumuni":
zulip_state = "⬜" if agent.get("zulip_state") == "not_installed" else ("✅" if agent.get("zulip_state") == "connected" else "⬜")
gateway = agent.get("gateway_state", "?")
-65
View File
@@ -1,65 +0,0 @@
#!/bin/bash
# Netbird Reverse Proxy — Add a new domain route
#
# Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]
#
# Example:
# netbird-add-domain.sh dns.sysloggh.net 192.168.68.10 80
#
# This script adds a domain to the Netbird proxy by inserting records
# directly into the management server's SQLite database, then restarting
# the proxy stack.
#
# Prerequisites: SSH root access to 72.61.0.17
# sqlite3 available on VPS
#
# Requires: The domain must already have a DNS CNAME to netbird.sysloggh.net
# pointing to 72.61.0.17.
set -euo pipefail
DOMAIN="${1:?Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]}"
BACKEND_IP="${2:?Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]}"
PORT="${3:-80}"
PROTOCOL="${4:-http}"
VPS="root@72.61.0.17"
DB_VOLUME="/var/lib/docker/volumes/root_netbird_data/_data"
DB="$DB_VOLUME/store.db"
echo "=== Adding Netbird proxy route ==="
echo "Domain: $DOMAIN"
echo "Backend: $BACKEND_IP:$PORT ($PROTOCOL)"
echo ""
ssh "$VPS" bash << REMOTESCRIPT
set -euo pipefail
# Generate unique ID using timestamp hash (Netbird format)
ID_SUFFIX=\$(date +%s | md5sum | head -c 16)
SVC_ID="d9\${ID_SUFFIX}ptsnc73\$(date +%s | md5sum | head -c 10)"
TGT_ID=\$(sqlite3 "$DB" "SELECT COALESCE(MAX(id), 100) + 1 FROM targets;")
ACCOUNT_ID="d88av3aptsnc73clmogg"
ZONE_ID="d8adqjaptsnc73fro5g0"
echo "Service ID: \$SVC_ID"
echo "Target ID: \$TGT_ID"
# Insert service
sqlite3 "$DB" "INSERT INTO services (id, account_id, name, domain, proxy_cluster, enabled, terminated, pass_host_header, rewrite_redirects, mode, source, port_auto_assigned, private) VALUES (\"\$SVC_ID\", \"\$ACCOUNT_ID\", \"$DOMAIN\", \"$DOMAIN\", \"netbird.sysloggh.net\", 1, 0, 1, 0, \"http\", \"permanent\", 0, 0);"
echo "Service: OK"
# Insert target
sqlite3 "$DB" "INSERT INTO targets (id, account_id, service_id, host, port, protocol, target_id, target_type, enabled, skip_tls_verify, request_timeout, session_idle_timeout, agent_network, disable_access_log) VALUES (\$TGT_ID, \"\$ACCOUNT_ID\", \"\$SVC_ID\", \"$BACKEND_IP\", $PORT, \"$PROTOCOL\", \"\$ZONE_ID\", \"subnet\", 1, 0, 0, 0, 0, 0);"
echo "Target: OK"
# Verify
sqlite3 -column "$DB" "SELECT s.name, t.host, t.port, t.protocol FROM services s JOIN targets t ON s.id=t.service_id WHERE s.name=\"$DOMAIN\";"
echo ""
echo "Restarting proxy stack..."
cd /root && docker compose restart netbird-server 2>/dev/null
sleep 15
docker compose restart proxy 2>/dev/null
echo "Done. Verify with: curl -sI https://$DOMAIN"
REMOTESCRIPT
+7 -7
View File
@@ -11,31 +11,31 @@ set -euo pipefail
# ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ──
declare -A CT_NODES=(
# amdpve (192.168.68.15)
[100]=amdpve # abiba
[105]=amdpve # kagentz
[111]=amdpve # tdunna
[112]=amdpve # tanko
[113]=amdpve # baggy
[115]=amdpve # scottdenya
# minipve (192.168.68.12)
[102]=minipve # adguard (was acerpve)
[104]=minipve # authentik
[110]=minipve # gitea
[114]=minipve # mumuni
[116]=minipve # syslog-api
[119]=minipve # infisical-vault
# storepve (192.168.68.6)
[106]=storepve # ra-h-os
[107]=storepve # proxmox-backup
[108]=storepve # media
[117]=storepve # zulip
[118]=storepve # jdownloader
# acerpve (192.168.68.9) — no CTs (bare metal GPU .8)
[100]=minipve # abiba (was hwepve)
[105]=minipve # kagentz (was hwepve)
# acerpve (192.168.68.9)
[102]=acerpve # adguard
# ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
#
# REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
# 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090)
# 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070)
# 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH)
# 118 jitsi → stopped, not in service
)
# Each node must be root-accessible via SSH hostname
@@ -74,7 +74,7 @@ main() {
if [[ $# -lt 1 ]]; then
echo "Usage: pct-run <CT_ID> [command...]" >&2
echo " pct-run 112 cat /etc/hostname" >&2
echo " pct-run 100 systemctl status hermes-gateway" >&2
echo " pct-run 114 systemctl status hermes-gateway" >&2
echo ""
echo "Known CTs:" >&2
for ct in $(echo "${!CT_NODES[@]}" | tr ' ' '\n' | sort -n); do
+1 -1
View File
@@ -33,7 +33,7 @@ TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram")
TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs)
TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs)
if [ "$TEL_STATUS" != "online" ] || [ "$TEL_RESTARTS" -gt 1000 ]; then
if [ "$TEL_STATUS" != "online" ]; then
pm2 restart abiba-telegram > /dev/null 2>&1
sleep 3
TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")
+7 -8
View File
@@ -47,24 +47,23 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
**Proxmox Cluster "Tabiri" (5 nodes):**
- amdpve (192.168.68.15): tanko, tdunna, baggy, scottdenya
- minipve (192.168.68.12): abiba, kagentz, adguard, authentik, gitea, syslog-api, infisical-vault
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip
- acerpve (192.168.68.9): llm-gpu
- amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya
- minipve (192.168.68.12): authentik, gitea, mumuni, syslog-api, jitsi
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip
- acerpve (192.168.68.9): llm-gpu, adguard
- ocupve (192.168.68.5): ocu-llm
**CT IDs (verified 2026-07-24 against PVE API):**
**CT IDs (verified 2026-07-04 against PVE API):**
100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os
107:pbs 108:media 110:gitea 111:tdunna 112:tanko
113:baggy 115:scottdenya 116:syslog-api 117:zulip
118:jdownloader 119:infisical-vault
113:baggy 114:mumuni 115:scottdenya 116:syslog-api 117:zulip
**NO CT 122, CT 123, or .19 exist in the cluster.**
**CRITICAL RULES (never regress):**
1. NO /grafana/ nginx route — it was tried and reverted on 2026-07-02. Grafana is direct LAN at :3001.
2. NO .19 IP — Zulip is CT 117 on storepve.
3. NO CT 122/123 — Tanko=CT 112, Mumuni=CT 100 (inside Abiba). No CT 114 anywhere.
3. NO CT 122/123 — Tanko=CT 112, Mumuni=CT 114.
4. Strix Halo :8080 is FIREWALLED to .116 only — cannot be probed from abiba (.24).
5. abiba-zulip PM2 process is DECOMMISSIONED (2026-07-04) — abiba uses Telegram only.
+4 -4
View File
@@ -12,10 +12,10 @@ echo ""
# Authorized agents for restricted contracts
# Format: contract_pattern|authorized_agents (comma-separated)
declare -A RESTRICTED
RESTRICTED["infrastructure-control.prose.md"]="abiba,abiba-bot,tanko,tanko-bot,mumuni,mumuni-bot"
RESTRICTED["proxmox-monitor.prose.md"]="abiba,abiba-bot"
RESTRICTED["hermes-config-template.prose.md"]="abiba,abiba-bot,mumuni,mumuni-bot,tanko,tanko-bot"
RESTRICTED["zulip-health.prose.md"]="abiba,abiba-bot,mumuni,mumuni-bot,tanko,tanko-bot"
RESTRICTED["infrastructure-control.prose.md"]="abiba"
RESTRICTED["proxmox-monitor.prose.md"]="abiba"
RESTRICTED["hermes-config-template.prose.md"]="abiba,mumuni,tanko"
RESTRICTED["zulip-health.prose.md"]="abiba,mumuni"
RESTRICTED["scripts/pm2-self-heal.sh"]="abiba"
RESTRICTED["scripts/prose-lint.sh"]="abiba"
RESTRICTED["scripts/prose-ai-review.sh"]="abiba"
-91
View File
@@ -1,91 +0,0 @@
#!/bin/bash
# swap-gpu-dense-model.sh — Swap RTX 3090 from qwen3.6-27B-code to SmartCode-Fable-5
# Run when download completes: ssh root@192.168.68.8 'bash -s' < this script
#
# Usage: bash swap-gpu-dense-model.sh
# Requires: new model at /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf
set -e
MODEL_PATH="/home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf"
OLD_WRAPPER="/home/llmuser/llama-wrapper.sh"
echo "═══ Swapping gpu-dense to SmartCode-Fable-5 ═══"
# 1. Verify model file
if [ ! -f "$MODEL_PATH" ]; then
echo "❌ Model not found at $MODEL_PATH"
echo " Download: curl -L -o $MODEL_PATH <huggingface-url>"
exit 1
fi
MODEL_SIZE=$(ls -lh "$MODEL_PATH" | awk '{print $5}')
echo "✅ Model found: $MODEL_SIZE"
# 2. Create new wrapper script for SmartCode-Fable-5
cat > /home/llmuser/llama-fable-wrapper.sh << 'WRAPPER'
#!/bin/bash
# SmartCode-Fable-5 llama-server wrapper for RTX 3090
# Sampler settings from model card: temp 0.9, top-p 0.95, top-k 60, repeat-penalty off
PORT=8080
GHOST_PID=$(ss -tlnp 2>/dev/null | grep -Po ":${PORT}\s+.*pid=\K[0-9]+" | head -1)
if [ -n "$GHOST_PID" ] && [ "$GHOST_PID" != "$$" ]; then
echo "[wrapper] Port $PORT occupied by ghost pid $GHOST_PID — cleaning up" >&2
kill -9 "$GHOST_PID" 2>/dev/null
sleep 2
fi
exec /usr/local/bin/llama-server \
--model /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf \
--ctx-size 131072 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--flash-attn 1 \
--cont-batching \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 1024 \
--n-gpu-layers 99 \
--temp 0.9 \
--top-p 0.95 \
--top-k 60 \
--min-p 0.0 \
--repeat-penalty 1.0 \
--api-key not-needed \
--port 8080 \
--host 0.0.0.0
WRAPPER
chmod 755 /home/llmuser/llama-fable-wrapper.sh
echo "✅ Created /home/llmuser/llama-fable-wrapper.sh"
# 3. Update systemd service to use new wrapper
echo "📝 Updating systemd service..."
sed -i 's|ExecStart=/home/llmuser/llama-wrapper.sh|ExecStart=/home/llmuser/llama-fable-wrapper.sh|' /etc/systemd/system/llama-server.service
systemctl daemon-reload
# 4. Stop old server, start new
echo "🔄 Restarting llama-server..."
systemctl stop llama-server
sleep 3
systemctl start llama-server
sleep 8
# 5. Verify
echo ""
echo "═══ Verification ═══"
systemctl is-active llama-server
echo ""
echo "Port 8080:"
ss -tlnp 2>/dev/null | grep ":8080" | head -1
echo ""
echo "GPU VRAM:"
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader 2>/dev/null
echo ""
echo "=== Health check ==="
curl -s --max-time 5 http://localhost:8080/health 2>/dev/null
echo ""
echo ""
echo "✅ Swap complete. Test via LiteLLM:"
echo " curl -s http://192.168.68.116/v1/chat/completions -H 'Authorization: Bearer <key>' -H 'Content-Type: application/json' -d '{\"model\":\"gpu-dense\",\"messages\":[{\"role\":\"user\",\"content\":\"write hello world in python\"}],\"max_tokens\":100}'"
+40 -38
View File
@@ -9,11 +9,10 @@ ZULIP_EMAIL="abiba-bot@chat.sysloggh.net"
ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
OWNER_ZULIP_ID="9"
LOG="/root/zulip-health-monitor.log"
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
ISSUES=0
echo "=== Zulip Health Check — $TIMESTAMP ===" >> "$LOG"
# Email config
GMAIL_USER="jtabiri@gmail.com"
GMAIL_PASS="rgbuomwcydxwbszd"
EMAIL_TO="jerome@sysloggh.com"
notify() {
local severity="$1" msg="$2"
@@ -25,14 +24,30 @@ notify() {
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-d "${form}" > /dev/null 2>&1 || true
# Zulip stream post to #agent-hub on topic 'zulip-health'
local stream_content="${severity} Zulip Monitor: ${msg}"
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-d "type=stream\&to=%5B7%5D\&topic=zulip-health\&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote(str()))")" \
> /dev/null 2>&1 || true
# Email alert
local subject="${severity} Zulip Monitor Alert"
python3 -c "
import smtplib
from email.mime.text import MIMEText
m = MIMEText('''${msg}''')
m['From'] = 'abiba@sysloggh.com'
m['To'] = '${EMAIL_TO}'
m['Subject'] = '${subject}'
s = smtplib.SMTP('smtp.gmail.com', 587)
s.starttls()
s.login('${GMAIL_USER}', '${GMAIL_PASS}')
s.sendmail('abiba@sysloggh.com', ['${EMAIL_TO}'], m.as_string())
s.quit()
" 2>/dev/null || true
}
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
ISSUES=0
LOG="/root/zulip-health-monitor.log"
echo "=== Zulip Health Check — $TIMESTAMP ===" >> "$LOG"
# ── Global: Zulip Server ──
SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \
https://chat.sysloggh.net/api/v1/server_settings \
@@ -66,39 +81,26 @@ else
echo " Abiba: ✅ Connected (processed=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('messages_processed',0))" 2>/dev/null))" >> "$LOG"
fi
# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ──
# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
# key availability varies — so probes run from the amdpve vantage via `pct exec`.
# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on amdpve
# (192.168.68.15). The gateway binds 127.0.0.1:3080 loopback-only by design — a
# remote :3080 probe is refused and is NOT a fault.
TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
"pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true)
[ -n "$TANKO_SVC" ] || TANKO_SVC="unknown"
TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
"pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true)
[ -n "$TANKO_HTTP" ] || TANKO_HTTP="000"
# ── Platform B: Hermes (Tanko) ──
TANKO_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 jerome@192.168.68.122 \
"cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}")
TANKO_ZULIP=$(echo "$TANKO_STATE" | python3 -c "
import sys,json
d=json.load(sys.stdin)
p=d.get('platforms',{}).get('zulip',{})
print(p.get('state','unknown'))
" 2>/dev/null)
if [ "$TANKO_SVC" != "active" ]; then
notify "🔴" "Tanko (DSH dsh-web) service state: $TANKO_SVC — needs restart"
if [ "$TANKO_ZULIP" != "connected" ]; then
notify "🔴" "Tanko (Hermes) Zulip state: $TANKO_ZULIP — needs restart"
ISSUES=$((ISSUES + 1))
echo " Tanko: ❌ service=$TANKO_SVC" >> "$LOG"
elif [ "$TANKO_HTTP" = "000" ]; then
notify "🔴" "Tanko (DSH dsh-web) HTTP :3080 connection refused/timeout — needs restart"
ISSUES=$((ISSUES + 1))
echo " Tanko: ❌ http=000 (refused/timeout)" >> "$LOG"
echo " Tanko: ❌ state=$TANKO_ZULIP" >> "$LOG"
else
case "$TANKO_HTTP" in
200|301|302|307|308|401|403)
echo " Tanko: ✅ service=active http=$TANKO_HTTP" >> "$LOG" ;;
*)
notify "🟡" "Tanko (DSH dsh-web) HTTP :3080 answered $TANKO_HTTP — running, unexpected status"
echo " Tanko: 🟡 service=active http=$TANKO_HTTP (running, warning)" >> "$LOG" ;;
esac
echo " Tanko: ✅ Zulip connected" >> "$LOG"
fi
# ── Platform B: Hermes (Mumuni) ──
MUMUNI_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.24 \
MUMUNI_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.123 \
"cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}")
MUMUNI_ZULIP=$(echo "$MUMUNI_STATE" | python3 -c "
import sys,json
+26 -81
View File
@@ -1,24 +1,22 @@
---
kind: responsibility
name: zulip-health
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Tanko on DSH / Mumuni on Hermes), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
title: Zulip Mesh Health Monitor — Multi-Platform
version: 3.0.0
runtime_contract: 2
agent: abiba
report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
---
# Zulip Mesh Health Monitor
Monitors ALL Zulip-connected agents across platforms (pi, Hermes, DSH, Agent Zero).
Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
Runs every 15 minutes in the background. Also triggers on session start.
## Requires
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); Mumuni (192.168.68.14, kagentz CT105 on minipve); and Agent Zero Docker host (192.168.68.14)
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
- **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -47,9 +45,11 @@ Runs every 15 minutes in the background. Also triggers on session start.
"severity": "healthy"
},
"tanko": {
"platform": "dsh",
"service_state": "active",
"http_status": 200,
"platform": "hermes",
"zulip_state": "connected",
"heartbeat_age_seconds": 45,
"gateway_pid": 1234,
"edit_fail_rate_pct": 0,
"severity": "healthy"
}
}
@@ -81,13 +81,13 @@ Log as "unreachable" — don't treat as critical unless it persists for 3+ conse
## Streaming Support (2026-07-05)
Zulip agents now support progressive message editing during agent generation.
When a Zulip agent (Tanko on DSH, Mumuni on Hermes) processes a message, the response is
When a Hermes agent (Tanko, Mumuni) processes a message, the response is
streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
- Adapter implements `edit_message()` using `_api_patch()` helper
- Gateway stream consumer progressively edits the Zulip message
- User sees real-time agent thinking instead of waiting for full response
- Verified: Tanko (CT 112) and Mumuni (kagentz CT 105) both have streaming active
- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
### Verification
```bash
@@ -182,90 +182,38 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
| `last_error` set | Log and monitor |
| Crash loop >10/h | Alert user |
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
### Step 3: Platform B — Tanko (DSH on amdpve CT 112) & Mumuni (Hermes)
Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve**
PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is not a dependency
of this contract — per-worker key availability varies — so CT 112 probes run
from the amdpve vantage via `pct exec`:
**B1: Gateway State**
```bash
ssh root@192.168.68.15 "pct exec 112 -- <command>"
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
```
> **By design (verified 2026-09-08):** the `dsh-web` gateway binds
> `127.0.0.1:3080` **loopback-only**. A remote probe against
> `192.168.68.122:3080` gets connection-refused — that is EXPECTED, NOT a fault,
> and must never be raised as Tanko down. Only loopback probes from inside
> CT 112 (or the public-URL fallback below) are valid health signals.
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
**B1: Gateway Service State (Tanko)**
```bash
ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web"
```
Expected: `active`. Anything else → gateway service down → apply the Tanko heal
(restart via DSH service, Platform B Actions table below).
**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**
```bash
ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
```
Alive = **ANY** HTTP status response from the endpoint — the expected set is
`200`/`301`/`302`/`307`/`308`/`401`/`403` (the gateway UI is token-gated and
legitimately answers with redirects/auth-challenges, so never require a bare
`200`), and any other status, including `404`/`5xx`, also counts alive: a
process answering `503` is running and self-heal must NOT restart-loop it.
Down = connection refused (`000`) or timeout only. Statuses outside the
expected set are logged/reported as a warning — reported, never healed on.
**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)**
```bash
curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
```
Fallback only — used when the monitoring node has no pct/SSH path to amdpve.
Alive = **ANY** HTTP status response from the endpoint — healthy signals are
`302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other
status, including `404`/`5xx`, also counts alive: the endpoint is up and
answering and must NOT be restart-looped. Down = connection refused (`000`) or
timeout only. Never expect a bare `200` — the public URL terminates in the
token-gated authentik chain. Statuses outside the healthy set are
logged/reported as a warning — reported, never healed on.
**B4: Gateway Process** (Hermes agent Mumuni only — Tanko runs no Hermes gateway)
**B2: Agent Process**
```bash
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
```
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more
than one `gateway run` process is found, the gateway has a collision (typically
one `--force` and one `--replace` process). Kill the newer/duplicate process,
then restart the remaining gateway per-agent (parameterized 2026-08-09, captain
ruling). Check the gateway log for "Gateway running with 2 platform(s)" (not 1)
to confirm Zulip reloaded.
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (`pm2 restart mumuni-zulip`). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
**B5: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
**B3: Heartbeat Verification**
```bash
ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
```
Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
Silence > 300s → warning. Silence > 600s → critical.
**B6: Response Delivery** (Hermes agent Mumuni only)
**B4: Response Delivery**
```bash
ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
```
> 50% fail rate → critical.
@@ -274,7 +222,7 @@ ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.her
| Condition | Action |
|-----------|--------|
| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` (Mumuni) / restart Tanko via DSH service |
| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
| No heartbeat in 10min | Same as above |
| `Failed to finalize` > 50% | Check PATCH API, Zulip server |
| Response empty/short | Check A2A endpoint / LiteLLM model |
@@ -284,11 +232,10 @@ ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.her
**C1: A2A Server Health**
```bash
# A2A server is on :50080 (not :8001) and is auth-gated (401 expected for unauthenticated)
ssh root@192.168.68.14 "curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:50080/a2a/"
ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
```
Expected: `401` (auth-gated, A2A server is up and responding) or `200` (if no auth required). Connection refused (000) → A2A server down.
Expected: `{"name":"kagentz",...}`. Connection refused → A2A server down.
**C2: Adapter Process**
@@ -309,14 +256,12 @@ Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0.
**C4: A2A Response Verification**
```bash
# A2A server is on :50080 (not :8001) and is auth-gated (401 expected for unauthenticated)
ssh root@192.168.68.14 "curl -s -X POST http://127.0.0.1:50080/a2a \
ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer $LITELLM_KEY' \
-d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
```
Expected: task ID with "working" status. Poll for completion with `tasks/get`. If 401, check LITELLM_KEY is set.
Expected: task ID with "working" status. Poll for completion with `tasks/get`.
**Platform C Actions**
+1 -1
View File
@@ -10,7 +10,7 @@ description: >
> **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and
> PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability
> monitoring now happens through Telegram. Agents (Mumuni on Hermes, Tanko on DSH) and
> monitoring now happens through Telegram. Hermes agents (Tanko, Mumuni) and
> Agent Zero (kagentz) continue to use Zulip.
## Maintains
+1 -1
View File
@@ -420,7 +420,7 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S
|-------|----------|-------------|--------------|-------------|
| **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
| **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
| **Mumuni** | Hermes (kagentz CT 105, migrated 2026-08-29) | ✅ Connected | No issues found | None needed |
| **Mumuni** | Hermes (CT 114) | ✅ Connected | No issues found | None needed |
### Key Fixes Applied
+6 -5
View File
@@ -13,8 +13,7 @@ triggers:
> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
> agents (Mumuni on Hermes, Tanko on DSH) continues through their own platform
> monitoring.
> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
## Maintains
@@ -66,7 +65,8 @@ triggers:
|------|----|------|---------|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
| Tanko | 192.168.68.122 (CT 112) | jerome | DSH (DeepSeek Harness) — restart via DSH service, not `hermes gateway restart` (no longer a Hermes agent since 2026-08-27) |
| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
## Debounce
@@ -75,8 +75,9 @@ Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
## Reporting
This contract is RETIRED — health-check logs are NOT knowledge graph content.
No graph nodes are created. Logs go to Gitea (SyslogSolution/health-logs).
Every cycle produces a knowledge graph node:
- Title: `[LEARN] zulip-self-heal: <timestamp>`
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"