Compare commits

...
Author SHA1 Message Date
root a48b947242 fix(alignment): correct tanko status - hybrid (DSH + Hermes), not DSH-only
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 21s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 21s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 20s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
PR #145 originally claimed 'the other 3 CTs are DSH-only' but all 4 agents
run a Hermes gateway. Only tanko additionally runs DSH, making it the hybrid
one. Fixed line 300 to say what the contract actually checks for each agent,
and fixed line 76 to stop asserting tanko's Hermes config is gone.
2026-09-28 12:57:00 +00:00
root 88e9243ce4 fix(alignment): clarify tanko's live LiteLLM proxy status in health-check description 2026-09-28 12:38:42 +00:00
root 2851a0cfc8 Merge PR #143: fix(agent-health): use llmuser for .8 GPU health probe instead of root
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 17s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 14s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 22s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 3s
2026-09-28 07:27:50 +00:00
root e0c9852de8 no-mistakes(document): Document llmuser SSH user for .8 GPU health probe
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 12s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 24s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 23s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
2026-09-28 07:22:13 +00:00
root bf3a1ba523 fix(agent-health): use llmuser for .8 GPU health probe instead of root
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Failing after 28s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
The rebuilt VM 101 (192.168.68.8) kept only one SSH key in root's
authorized_keys, so the health check's root probe returns Permission
denied and misreports the healthy host as UNREACHABLE. The llama-server
runs as llmuser, so that user can see the :8080 pid via ss -tlnp.

Add per-host user to GPU_HOSTS (default root, llmuser for .8) and pass
it through check_gpu_ports() into all ssh() calls.

Closes the gpu-unreachable:192.168.68.8 leg while leaving the root SSH
security decision for the captain.
2026-09-28 07:12:57 +00:00
root f230812e3a Merge PR #142: sync tanko CT 112 PVE mapping to minipve across consumers
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 18s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-09-28 07:01:19 +00:00
root 96769a103f no-mistakes(review): sync tanko CT 112 mapping to minipve across consumers
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 12s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 17s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s
2026-09-28 06:47:46 +00:00
root d697baa7b6 fix(agent-health): update tanko PVE mapping from amdpve to minipve (CT 112 migrated 2026-09-27)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 13s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 24s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 26s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-09-28 06:35:34 +00:00
abiba-bot fb7f351a2b Merge pull request 'fix(audit-hermes): handle fallback_providers as list or dict' (#141) from fix/audit-hermes-fallback-list-20260927 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-09-27 11:35:07 +00:00
root e42b970dec fix(audit-hermes): handle fallback_providers as list or dict
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 17s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
The audit assumed fallback_providers was always a dict (single provider).
Two live agents (koby, koonimo) carry it as a LIST of dicts (one entry per
fallback), so the script crashed with:

    File "audit-hermes-config.py", line 211, in audit
        fb.get("provider") == "deepseek",
    AttributeError: 'list' object has no attribute 'get'

Both are REAL agent configs, so this is not a malformed-input case — the
script simply could not audit two of the four agents it exists to audit.

Fix:
- Normalize fallback_providers to a list of entries (dict → [dict], list → list)
- Apply the existing checks to each entry
- A malformed entry (not a mapping) produces a reported VIOLATION naming the
  offending entry, NOT an uncaught exception

Adds regression test using the real failing shape (list of dicts) and proves
it bites against the pre-fix revision.

Real audit results after fix:
- mumuni: FAIL — 7 violations
- tanko: FAIL — 21 violations
- koby: FAIL — 16 violations (previously crashed)
- koonimo: FAIL — 10 violations (previously crashed)

No agent configs were changed. No existing rules were relaxed.
2026-09-27 11:17:18 +00:00
abiba-bot eadb927ec1 Merge pull request 'fix(hermes): clarify auxiliary model policy to match audit' (#140) from fix/hermes-aux-model-policy-20260927 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-09-27 10:05:44 +00:00
root d4e238047d fix(hermes): clarify auxiliary model policy to match audit
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
The 'Auxiliary Tasks (CONSISTENCY RULE)' header claimed ALL auxiliary
services must use an identical model (gpu-vision) and that syslog-auto
must never be used for auxiliary tasks. Both are false for compression,
which the script (and Rule 7/8) require to be syslog-auto. An agent
following the template produced a config the audit then failed.

- Split auxiliary into two classes: light (vision, web_extract/browsing)
  -> gpu-vision (RTX 5070); context-heavy (compression) -> syslog-auto,
  citing the existing 2026-07-23 OPERATIONAL DECISION in Rule 7.
- Remove the absolute 'Do NOT use syslog-auto for auxiliary tasks' line.
- State gpu-dense + strix-moe are the reasoning hosts; do not pin aux to them.
- Note the change in the frontmatter UPDATED log.

audit-hermes-config.py unchanged (it is the enforcement contract); prose
now matches it line-for-line on vision/web_extract/compression.

Verified: prose-lint.sh PASS, secret-scan.sh clean, 14/14 tests in
tests/test_audit_hermes_config_alias.py pass, audit PASSES on the
template's stated policy.
2026-09-27 09:46:37 +00:00
mumuni-bot 73d5097555 Merge pull request 'feat(memory-fixer): add duplicate-detection phase; correct stale canonical pointer + reporting model (v2.2.0)' (#139) from fix/memory-fixer-duplicate-detection-20260926 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 2s
Merge PR #139 — memory-fixer duplicate-detection phase (v2.2.0). All CI green; authorized by AGENTS.md step 4 (all green -> merge) and the authorization table (memory-fixer = normal sensitivity, any registered agent).
2026-09-27 02:16:43 +00:00
Mumuni c460ef905c feat(memory-fixer): add duplicate-detection phase; correct stale canonical pointer + reporting model (v2.2.0)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Level 1 gains phase 5, duplicate-node detection: runs
memory_dup_detect.py (read-only) and reports clusters by verdict.

- WRITER-DEFECT (run_family): one task creating a node per run -> writer
  fix, never merge (per-run nodes are the audit trail)
- SAFE-MERGE: identical bodies, still needs an explicit Kwame decision
- HUMAN-DECISION: same subject, bodies differ -> connect, never merge

Replaces the naive '>70% title overlap' duplicate rule, which false-positives
on distinct work: four client workflows (#357-#361) and two machines'
migrations (#1792/#1793) score high on titles with bodies 0.1-0.3 apart.

Also corrected in the same pass:
- the 'canonical copy' pointer named /root/.hermes/contracts/memory-fixer-v3.md,
  which does not exist on kagentz (no /root access); the job's instruction set is
  inline in ~/.hermes/cron/jobs.json
- 'reports to Kwame via this Zulip DM' described the pre-2026-09-21 model; the
  report is DROPped to the gate (comms_drop.py) and exit 0 means QUEUED, not sent
- added a Checks line so a skipped phase 5 is visible in the report
2026-09-26 19:32:36 +00:00
abiba-bot d99b552448 Merge pull request 'feat(search): agent-consumption layer — dedupe, filter, rerank, extract content' (#138) from feat/search-agent-consumption-20260926 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 12s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 10s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-09-26 16:06:47 +00:00
abiba-bot fc0cd7a032 Merge pull request 'feat(daily-digest): deliver via Zulip DM as an HTML attachment; drop mail entirely' (#137) from fix/daily-digest-zulip-delivery-20260926 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 12s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-09-26 16:06:35 +00:00
root 0a41a2d584 fix(daily-digest): probe Firecrawl on its real liveness path and classify endpoints per fleet policy
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Folds into the same branch as the Zulip delivery change, as instructed.

FINDING 1 - the Firecrawl probe path was wrong; the service is fine.
  scripts/daily-infra-report.py probed http://192.168.68.7:3002/health, which
  Firecrawl does not serve - it 404s. The root answers 200 with
  {"message":"Firecrawl API",...}. Live 2026-09-26:
    Firecrawl(/)        -> 200
    Firecrawl(/health)  -> 404   <- what the report was showing
  The probe is now the root, which is its liveness endpoint.

FINDING 2 - the Network Endpoints classification was wrong twice over.
  It read: color = green if code in (200,302,401) else (yellow if code >= 400
  else red). Two defects:
   (a) it ignored the fleet's own probe policy, codified 2026-09-14 in the
       monitoring contracts: ANY HTTP status proves the service answered, so the
       service is ALIVE, and only a failed CONNECTION is a failed probe. A 404
       from a wrong path is not a service fault.
   (b)  was a STRING comparison. Reproduced: 301 -> red (a live
       redirect rendered as a failure), 404 -> yellow, 500 -> yellow (a real
       server error softened to a warning).
  Replaced with classify_endpoint(), which returns:
     any 2xx/3xx/4xx -> green  'alive'          (code still shown)
     5xx             -> yellow 'server error'   (kept distinct from 4xx, as asked)
     000/no answer   -> red    'no connection'

  Verified against the live endpoints after the change:
    Gitea 200, Authentik 302, Zulip 302, Pulse 200, Proxmox 200, SearXNG 200,
    Firecrawl 200 - all green/alive; the only red state is a genuine no-connection.

ALSO CHECKED, as asked: scripts/search-stack-check.py does NOT depend on the
wrong route. It POSTs to {FIRECRAWL_URL}/v1/scrape with formats=[markdown], and
that path really works - live POST returned HTTP 200 and 180 chars of markdown
for https://example.com. It was never using /health.

prose-lint: PASSED.
2026-09-26 15:36:38 +00:00
root de32f54337 feat(daily-digest): deliver via Zulip DM as an HTML attachment; drop mail entirely
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 13s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Captain's decision 2026-09-26, clarified the same day: the digest is delivered to
his Zulip DM (user id 9) from abiba-bot as an HTML FILE - an attachment, not HTML
rendered in the message body and not a Markdown translation of it. Closes
daily-digest-mail-transport-20260921; the Google dependency is gone (no SMTP, no
EMAIL_PASSWORD, no app password, nothing to rotate).

WHAT CHANGES
* scripts/daily-infra-report.py: send_email() is replaced by send_zulip(), which
  writes the styled dashboard to /var/log/daily-infra-report/infra-report-<ts>.html,
  uploads it via POST /api/v1/user_uploads, then posts a SHORT Markdown pointer to
  user 9. The message body carries subject, top-line status and the attachment
  link; it does not reproduce the report.
* the 10,000-character cap is irrelevant here - it bounds message TEXT only, and
  the report travels as a file, so nothing is shrunk to fit.
* the key is abiba-bot's, already on the execution host at
  /root/.pi/agent/extensions/zulip/.env (mode 600). No vault entry was added:
  under the auth-keys charter that is a captain decision.
* daily-health-digest.prose.md -> v2.0.0 and contract-registry.yaml updated:
  transport, healthy/degraded definitions, and exit codes now match observed
  behaviour. There is NO degraded delivery leg any more - delivery is the only
  output path, so a missing or rejected key is a real failure (exit 1).
* queued defect folded in: a failed delivery used to print only the transport
  error while the report body never surfaced. Now the HTML is printed to stdout
  AND persisted on every failure, and the message names which step failed.

EVIDENCE (all against the live stack)
* real send: message id 86221 to user 9, attachment 16208 bytes at
  /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html
* the message is type=private, sender abiba-bot@chat.sysloggh.net, recipients
  [9, 21], body carries the top-line status and the attachment link, and does NOT
  contain a <table> - i.e. it does not reproduce the report
* the attachment fetches HTTP 200, 16208 bytes, content-type text/html, starts
  with <!DOCTYPE html>, and contains <style>, <table> and 16 class="card" blocks -
  it opens as a standalone styled document
* failure path: a bad key gives 'Delivery FAILED at upload: Malformed API key',
  EXIT=1, the HTML is printed to stdout and persisted to disk
* scheduled path: the run's own output is pasted in the PR

prose-lint: PASSED (19 warnings); secret scan clean.
2026-09-26 15:35:36 +00:00
23 changed files with 678 additions and 167 deletions
+1 -1
View File
@@ -24,7 +24,7 @@ Runs every 4 hours (2, 6, 10, 14, 18, 22 UTC at :35) via cron (`35 2,6,10,14,18,
## Requires
- **LiteLLM admin key** for key validation (retrieved from `/root/.pi/agent/env.sh`)
- **SSH access** to GPU hosts (.8, .110, .15) and agent CTs (.122, .129, .114, .24)
- **SSH access** to GPU hosts — `llmuser` on .8 (owns `llama-server`), `root` on .110 and .15 — and agent CTs (.122, .129, .114, .24)
- **Python 3** for script execution
- **Network access** to LiteLLM (:4000), GPU exporters (:9400), and gateway endpoints
+36 -17
View File
@@ -94,7 +94,16 @@ def audit(path):
cfg = yaml.safe_load(f)
model = cfg.get("model", {})
fb = cfg.get("fallback_providers", {})
fb_raw = cfg.get("fallback_providers", {})
# Normalize: fallback_providers may be a dict (single provider) or a list of dicts
# (one entry per fallback). Both shapes are valid; we must handle both without crashing.
if isinstance(fb_raw, dict):
fb_entries = [fb_raw]
elif isinstance(fb_raw, list):
fb_entries = fb_raw
else:
fb_entries = [fb_raw] # Let it fail the check below as malformed
fb = fb_entries[0] if fb_entries else {}
comp = cfg.get("compression", {})
aux = cfg.get("auxiliary", {})
deleg = cfg.get("delegation", {})
@@ -207,22 +216,32 @@ def audit(path):
"Rule 14",
f"delegation.provider must be 'harness' (got {deleg.get('provider')!r})",
)
check(
fb.get("provider") == "deepseek",
"Rule 14",
f"fallback_providers.provider must be 'deepseek' (got {fb.get('provider')!r}) — "
f"true fallback diversity, not same endpoint as primary",
)
check(
fb.get("model") == "deepseek-v4-flash",
"Rule 14",
f"fallback_providers.model must be 'deepseek-v4-flash' (got {fb.get('model')!r})",
)
check(
fb.get("api_key_env") == "DEEPSEEK_API_KEY",
"Rule 14",
f"fallback_providers.api_key_env must be DEEPSEEK_API_KEY (got {fb.get('api_key_env')!r})",
)
# Check each fallback entry. A malformed entry (not a mapping) is a VIOLATION, not a crash.
for idx, entry in enumerate(fb_entries):
prefix = f"fallback_providers[{idx}]"
if not isinstance(entry, dict):
check(
False,
"Rule 14",
f"{prefix} must be a mapping (got {type(entry).__name__})",
)
continue
check(
entry.get("provider") == "deepseek",
"Rule 14",
f"{prefix}.provider must be 'deepseek' (got {entry.get('provider')!r}) — "
f"true fallback diversity, not same endpoint as primary",
)
check(
entry.get("model") == "deepseek-v4-flash",
"Rule 14",
f"{prefix}.model must be 'deepseek-v4-flash' (got {entry.get('model')!r})",
)
check(
entry.get("api_key_env") == "DEEPSEEK_API_KEY",
"Rule 14",
f"{prefix}.api_key_env must be DEEPSEEK_API_KEY (got {entry.get('api_key_env')!r})",
)
# --- custom_providers sanity ---
check(
+15 -11
View File
@@ -643,7 +643,7 @@ contracts:
timeout: 120
requires:
- Zulip API key for abiba-bot@chat.sysloggh.net
- SSH access to amdpve (192.168.68.15) for Tanko (CT 112) and the Agent Zero Docker host (.14)
- SSH access to minipve (192.168.68.12) for Tanko (CT 112) and the Agent Zero Docker host (.14)
verification:
postconditions:
- check: bot registration active
@@ -1965,17 +1965,21 @@ contracts:
verify_commands:
- infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-email
- python3 -m pytest tests/test_daily_infra_report.py -q
email_dependency:
transport: smtp.gmail.com:587
identity: jtabiri@gmail.com
secret: EMAIL_PASSWORD (must be a Google app password)
status: DEGRADED as of 2026-09-25 - 534 5.7.9 Application-specific password required
note: A delivery failure is a credential dependency, not a code defect. Tracked
as daily-digest-mail-transport-20260921.
delivery:
transport: zulip-dm-attachment
recipient_user_id: 9
sender: abiba-bot@chat.sysloggh.net
key_source: abiba-bot Zulip key already on the execution host, read from the
600-mode env file /root/.pi/agent/extensions/zulip/.env
key_policy: do NOT add a vault entry - that is a captain decision under the auth-keys charter
body: short Markdown pointer; the HTML attachment IS the report
artifact: /var/log/daily-infra-report/infra-report-<UTCstamp>.html
note: Replaced SMTP/mail on 2026-09-26 by captain decision. Removes the Google
dependency entirely; closes daily-digest-mail-transport-20260921.
exit_semantics:
'1': missing PVE_TOKEN, unreachable Proxmox probe, or failed email send - raises an alert
'0': healthy, or a deliberate DEGRADED leg where the email credential is absent
and the report is still produced
'1': missing PVE_TOKEN, unreachable Proxmox probe, missing/rejected Zulip
credential, or a failed upload/post - raises an alert
'0': healthy delivery only - there is no degraded delivery leg any more
depends_on: []
last_run: null
last_status: null
+48 -32
View File
@@ -16,11 +16,12 @@ description: >
Exit-code semantics (as they actually behave, verified 2026-09-25):
* missing PVE_TOKEN, or an unreachable Proxmox probe -> exit 1 + alert
* missing EMAIL credential -> deliberate DEGRADED leg, exit 0, report still
produced
* email send failure -> exit 1 (a delivery fault, not a code defect)
* missing or rejected Zulip credential -> exit 1 (delivery is the only
output path, so it is a real failure, not a degraded leg)
* delivery failure -> exit 1, and the report body is printed AND persisted
so the content is never swallowed
version: 1.0.0
version: 2.0.0
---
## Purpose
@@ -93,14 +94,13 @@ $ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json
EXIT=0
```
and in mail mode:
and in delivery mode:
```
Sending email...
✅ All legs fully credentialed
📋 Summary:
Proxmox: 5/5 nodes online
VMs/CTs: 22/22 running
report ready: 16208 chars of HTML (delivered as a file attachment)
Sending to the captain's Zulip DM...
✅ Delivered to Zulip DM (user 9), message id 86221, attachment 16208 bytes
at /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html
```
Healthy means: every probe reports `ok`, `nodes_online == node_count`, and the
@@ -115,9 +115,8 @@ Verified on 2026-09-25 by running each case deliberately.
| all probes reachable, email sent | 0 | — | healthy |
| **missing `PVE_TOKEN`** | **1** | yes | `PROBE FAILURES: proxmox: node list unreachable (PVE_TOKEN missing or API down)`, and `cluster resources unreachable` |
| **Proxmox probe unreachable** | **1** | yes | same path as above; `pve_probe_status: unreachable` |
| **missing `EMAIL_PASSWORD`** | **0** | no | deliberate **DEGRADED** leg (`credential-missing: EMAIL_PASSWORD`); the report is still produced |
| **email send fails** | **1** | yes | e.g. Gmail `534 5.7.9 Application-specific password required` |
| degraded legs present (non-email) | 0 | no | logged under `⚠️ Degraded legs` |
| **missing/rejected Zulip credential** | **1** | yes | delivery is the only output path; report printed and persisted |
| **upload or message post fails** | **1** | yes | report printed and persisted; message names which step failed |
The distinction is deliberate and must not be flattened:
@@ -130,22 +129,31 @@ The distinction is deliberate and must not be flattened:
`PROBE_FAILURES` and `DEGRADED_LEGS` are separate lists for exactly this
reason. Do not merge them.
## Email-delivery dependency
## Delivery: Zulip DM carrying the report as an HTML ATTACHMENT
Delivery is a **credential dependency, not a code path**. The producer
authenticates to `smtp.gmail.com:587` as `jtabiri@gmail.com` with
`EMAIL_PASSWORD` from the vault and sends to `jerome@sysloggh.com`.
Captain's decision 2026-09-26, clarified the same day: the digest is delivered to
his **Zulip DM (user id 9)** from `abiba-bot@chat.sysloggh.net`, as an **HTML
FILE** — an attachment, not HTML rendered in the message body and not a Markdown
translation of it.
* Since that Google account has two-step verification, `EMAIL_PASSWORD` must be
a Google **app password**, not the account password.
* As of 2026-09-25 delivery is **failing** with
`534 5.7.9 Application-specific password required`; the fix is for the
captain to generate a fresh app password and place it in Infisical
(`infrastructure/production`) as `EMAIL_PASSWORD`.
* **A delivery failure is not a code defect.** Investigation of a failed send
should start at the credential, not the script. Chasing it as a code bug
wastes the effort; verify the credential path first with `--test-email`.
* Tracked separately as `daily-digest-mail-transport-20260921`.
* the styled dashboard is built exactly as before and written to
`/var/log/daily-infra-report/infra-report-<UTCstamp>.html`;
* it is uploaded through `POST /api/v1/user_uploads`;
* the **message body stays short Markdown** — subject line, top-line status
(nodes online, guests running, any degraded legs), and a link to the
attachment. The attachment IS the report; the body does not reproduce it.
This removes the Google dependency entirely: **no SMTP, no `EMAIL_PASSWORD`, no
app password, nothing to rotate.** `daily-digest-mail-transport-20260921` is
closed under this option.
The **10,000-character message cap does not apply** — it bounds message TEXT
only, and the report travels as a file. Do not shrink the report to fit it.
The credential is abiba-bot's Zulip key already on the execution host at
`/root/.pi/agent/extensions/zulip/.env` (`ABIBA_ZULIP_API_KEY`, mode 600,
root-readable). **Do not place a new credential in the vault** — under the
auth-keys charter that is a captain decision.
## What counts as a failure
@@ -153,10 +161,18 @@ A run FAILS (exit 1) when the report cannot be trusted or delivered:
* any probe is unreachable, so a section would silently be empty;
* `PVE_TOKEN` is missing;
* the email send fails.
* the Zulip credential is missing or rejected, or the upload/post fails.
A run is DEGRADED (exit 0, report still produced) when a non-load-bearing
credential is absent, currently only `EMAIL_PASSWORD`.
There is **no degraded delivery leg any more**. Delivery is the only output
path, so a missing credential is a failure rather than a survivable degradation —
the previous "missing `EMAIL_PASSWORD` still exits 0" rule is retired with the
mail transport.
**A delivery failure must never swallow the report.** On failure the script
prints the report body to stdout *and* leaves the HTML artifact on disk, so the
content is always recoverable from the run log. That closes the queued defect
where a failed send printed only the transport error and the report never
surfaced.
## Failure behaviour
@@ -175,7 +191,7 @@ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json \
| grep -E 'pve_probe_status|node_count|nodes_online'
# delivery path
infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-email
infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-zulip
```
Regression tests: `tests/test_daily_infra_report.py` (7 tests). Four of them
@@ -183,5 +199,5 @@ fail against the pre-fix script, which is what makes them bite.
## Maintains
- daily-infra-dashboard: { status: "degraded", reason: "email credential", last_check: timestamp }
- daily-infra-dashboard: { status: "ok|undelivered", transport: zulip-dm-attachment, last_check: timestamp }
- pve-probe: { status: "ok|unreachable", last_check: timestamp }
+1 -1
View File
@@ -414,7 +414,7 @@ one-off GPU builds. No automated post-migration cleanup was in place.
| 108 | media | storepve | lxc | ✅ reachable |
| 110 | gitea | minipve | lxc | ✅ reachable |
| 111 | tdunna | **storepve** | lxc | ⛔ **REPORT-ONLY** (192.168.68.129, Theo's box — no GC at any level) |
| 112 | tanko | amdpve | lxc | ✅ reachable |
| 112 | tanko | minipve | lxc | ✅ reachable |
| 113 | baggy | amdpve | lxc | ✅ reachable |
| 115 | scottdenya | amdpve | lxc | ✅ reachable |
| 116 | syslog-api | minipve | lxc | ✅ reachable |
+3 -3
View File
@@ -70,10 +70,10 @@ Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → GPU (llama-server)
## Config Pattern — Mandatory Fields
### For Hermes Agents (Mumuni, Koonimo)
### For Hermes Agents (Mumuni, Koonimo, Tanko-hybrid)
Every Hermes agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
(Tanko is excluded — migrated to DSH/DeepSeek Harness on 2026-08-27, no longer uses Hermes config.)
(Tanko is hybrid — runs both DSH and Hermes since 2026-08-27, so its Hermes config is also checked.)
### 1. Main Model
```yaml
@@ -297,7 +297,7 @@ Run the consolidated health check:
```bash
python3 /root/scripts/agent-health-check.py
```
This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
This validates each agent's live LiteLLM key against the gateway, including tanko, which runs HYBRID (DSH + Hermes) since 2026-08-27; detects GPU port conflicts (ghost processes),
verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
and counts recent errors. Non-disruptive — never restarts anything.
+13 -5
View File
@@ -5,6 +5,11 @@ description: >
Standard Hermes configuration template for Syslog Solution LLC agents.
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
RA-H OS MCP) while keeping agent-specific API keys and model choices.
UPDATED 2026-09-27: Clarified the Auxiliary Tasks policy — light aux (vision,
web_extract/browsing) -> gpu-vision (RTX 5070); context-heavy aux (compression) ->
syslog-auto (2026-07-23 decision, Rule 7). Removed the false "one model for all
auxiliary" / "never syslog-auto" claim; stated gpu-dense + strix-moe are the reasoning
hosts and aux should not be pinned to them. Now matches audit-hermes-config.py line-for-line.
UPDATED 2026-08-07: Added litellm MCP server entry; updated Rule 15 (MCP Validation)
to enforce REAL key headers (not env-vars) from the 2026-08-07 keyless-MCP incident.
Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
@@ -167,13 +172,16 @@ compression:
abort_on_summary_failure: false
# ─── Auxiliary Tasks (CONSISTENCY RULE) ───
# All auxiliary services MUST use identical model, base_url, and api_key_env:
# model: gpu-vision # stable alias (NOT a raw model name)
# Auxiliary tasks split into TWO model classes — do NOT assume one model for all:
# Light auxiliary (vision, web_extract/browsing) -> model: gpu-vision # RTX 5070
# Keeps the reasoning hosts (gpu-dense / strix-moe) free for agent prompts.
# Context-heavy auxiliary (compression) -> model: syslog-auto # weighted pool
# Deliberate per the 2026-07-23 OPERATIONAL DECISION in Rule 7: summarization
# runs against long histories and must be able to use the pool.
# Do NOT pin auxiliary work to the reasoning hosts (gpu-dense / strix-moe).
# All auxiliary services share identical ROUTING (base_url + api_key_env), not model:
# base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
# api_key_env: LITELLM_API_KEY
# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
# gpu-vision = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
# Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
# NEVER use retired model names (qwen3.6-27B-code, qwen3.6-35B-udq4; gemma-4-12b is retired
# and no longer resolves) in agent configs — use the stable aliases so model swaps don't break agents.
auxiliary:
+2 -2
View File
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------|
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
@@ -72,7 +72,7 @@ connectivity recovery including end-to-end DM validation.
### Step 1: Resolve Target
Map `target` to host, CT ID, hermes_home, and user from the live-state table.
For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
For CT112 route through `ssh root@minipve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
### Step 2: Pull Latest Plugin Source
+1 -1
View File
@@ -67,7 +67,7 @@ gateway restart, and connection validation.
### Step 1: Locate Target
Map `target` to connectivity parameters from the live-state table above.
For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
For CT112 route through `ssh root@minipve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
### Step 2: Deploy Zulip Adapter
+4 -4
View File
@@ -105,8 +105,8 @@ description: >
| Node | IP | CPU | RAM | VMs/CTs | Role |
|------|----|-----|-----|---------|------|
| minipve | .12 | 16C | 30GB | abiba, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | kagentz, tanko, baggy, scottdenya, adguard2 | Agents, compute |
| minipve | .12 | 16C | 30GB | abiba, tanko, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | kagentz, baggy, scottdenya, adguard2 | Agents, compute |
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat |
| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
@@ -682,7 +682,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| 109 | docker-vm | storepve | .7 | Docker host | ❌ |
| 110 | gitea | minipve | **.17** | Git | ❌ |
| 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
| 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ |
| 112 | tanko | minipve | .122 | DSH (DeepSeek Harness) agent | ✅ |
| 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
@@ -712,7 +712,7 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
| 100 | abiba | minipve | `pct-run 100` |
| 105 | kagentz | amdpve | `pct-run 105` |
| 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) |
| 112 | tanko | amdpve | `pct-run 112` |
| 112 | tanko | minipve | `pct-run 112` |
| 113 | baggy | amdpve | `pct-run 113` |
| 115 | scottdenya | amdpve | `pct-run 115` |
| 104 | authentik | minipve | `pct-run 104` |
+1 -1
View File
@@ -59,7 +59,7 @@ Before ANY update wave:
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, minipve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 105 (kagentz, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
+44 -6
View File
@@ -6,13 +6,19 @@ name: memory-fixer
description: >
Auto-fix low-hanging fruit in the RA-H OS knowledge graph. No judgment calls — only deterministic Level 1 operations.
Escalate anything that needs Kwame's input. Executes confirmed Kwame decisions to completion (state + updated_at).
version: 2.1.0
version: 2.2.0
---
---
# Memory Fixer
> **Canonical copy:** `/root/.hermes/contracts/memory-fixer-v3.md` (used by the `memory-fixer-daily` cron job). This file is the institutional record of the same contract. When the two diverge, treat the v3 source in `/root/.hermes/contracts/` as executable truth.
> **Executable copy:** the `okyeame-memory-fixer` cron job on kagentz (`hermes cron list`) holds its instruction
> set **inline in `~/.hermes/cron/jobs.json`** (`hermes cron edit <id> --prompt …`; there is no `--prompt-file`, and
> `~/.hermes/cron/memory-fixer-prompt.md` is a synced draft, not the live instruction). This file is the institutional
> record of the same contract; when the two diverge, the job prompt is what actually runs — diff it against this file
> before claiming a prompt change landed.
> ⚠️ Corrected 2026-09-26: the previous pointer (`/root/.hermes/contracts/memory-fixer-v3.md`) does not exist on
> kagentz — no `/root` access from this container — and was verified unreachable, not merely stale.
## Purpose
Auto-fix low-hanging fruit in the graph. No judgment calls — only deterministic Level 1 operations. Escalate anything that needs Kwame's input. When Kwame replies to an escalation, **execute the decision to completion** (update state and timestamps), never leaving a node in review-pending forever.
@@ -125,15 +131,44 @@ updateNode(id, {
**Archive candidates are identified by the fix 3 query's `suggested_action = 'archive'` branch** (the `ELSE 'archive'` case: anything not an infrastructure/skill/documentation/strategic/audit type).
### 5. Duplicate-Node Detection (Level 1 — read-only, every run)
The graph's duplicate problem is rarely an agent mistyping a title: it is **recurring writers creating a new
node per run instead of updating one**. This phase detects that class and reports it. It is read-only and
**never merges**.
```bash
python3 /home/hermes/.hermes/scripts/memory_dup_detect.py --json
```
Read-only, ~15s over the whole graph, exit 0. That script is the source of truth for the clustering logic —
do not re-implement it in the prompt or hand-count "duplicates" from titles.
Consume each `items[]` entry's `verdict` field; do not invent your own:
| `verdict` | Meaning | Required action |
|---|---|---|
| `WRITER-DEFECT` (`run_family: true`) | ONE scheduled task writes a new node per run | Report the ids, the `agents` (the writer) and `span_days`. **Never merge** — each node is that run's audit record. If the family grew since the last report, say `UNFIXED` and name the writer. |
| `SAFE-MERGE` | Bodies identical | Still requires an explicit `merge #A into #B` decision from Kwame. |
| `HUMAN-DECISION` | Same subject, bodies differ | Propose **connect (an edge)**, never merge. |
- **Title overlap alone is not duplication.** Four distinct client workflows of one family (#357-#361) and two
different machines' migrations (#1792/#1793) both score high on title tokens while their bodies sit 0.1-0.3
apart. Confirm against body similarity before calling anything a duplicate.
- Report clusters as **candidates for Kwame's decision**, never as established duplicates — a wrong auto-merge
destroys distinct content irrecoverably.
- Per-run history nodes are kept deliberately. Bulk-merging a run family destroys the audit trail the family exists for.
## Level 2 Escalations (Kwame Decision Required)
1. **Refresh-suggested stale nodes** flagged with `[REVIEW: refresh]` — refresh or keep? (Archive-suggested nodes are auto-archived under fix 4 and are not escalated.)
2. **Duplicate Nodes** (same title or >70% title overlap) — Merge or keep?
2. **Duplicate Nodes** — as detected by fix 5, by `verdict`, never by raw title overlap. `WRITER-DEFECT` is a writer fix (update one canonical node), not a merge decision; `SAFE-MERGE` and `HUMAN-DECISION` clusters are escalated for merge-or-connect.
3. **Orphan Nodes >90 days old** — Archive or connect?
## Reporting Format
The fixer reports to Kwame via this Zulip DM:
The fixer does **not** send anything. Under the single-egress model (2026-09-21) every report leaves the node
through Mumuni's gate (`comms_drop.py` for the queue, `comms_gate.py` to release and read-back verify), so
exit 0 means QUEUED, never delivered. A report body is written to a file and handed to the outbox helper:
```
🦅 Memory Fixer — [HH:MM UTC]
@@ -147,8 +182,10 @@ Stale nodes needing review (max 10):
2. [Node #YYY] Title — Y days stale, SUGGEST: archive
...
Duplicates needing decision:
1. [Node #AAA] vs [Node #BBB] — Same title
Duplicate clusters (candidates — Kwame decides; the fixer never merges unilaterally):
1. [WRITER-DEFECT] #AAA/#BBB/#CCC — writer <agent>, N nodes, span Nd (UNFIXED if it grew since the last report)
2. [HUMAN-DECISION] #DDD/#EEE — same subject, bodies differ, SUGGEST: connect
3. "none" when the scan returned no clusters
Orphans >90 days:
1. [Node #EEE] Title — X days stale, orphaned
@@ -195,6 +232,7 @@ The result must be 0 rows when all decisions are executed. Report what was done.
- **State integrity:** archived nodes have `state: archived` + `[ARCHIVED]` prefix; kept nodes are `state: active` without a `[REVIEW:]` tag.
- **Auto-archive applied:** no node should ever be left tagged `[REVIEW: archive]` — that tag is retired. Any `[REVIEW: archive]` found means fix 4 was skipped; archive it and report.
- **No review-pending forever:** after executing Kwame's decisions, `[REVIEW:%` node count must be 0.
- **Duplicate scan ran:** every report carries the fix 5 block (`none` when there were no clusters). A report with no duplicate section means phase 5 was skipped — a silently skipped detection phase is the failure this phase exists to prevent.
- **Timestamps:** every executed decision (and every auto-archive) bumps `updated_at`, so the node exits the stale window on the next run.
## Logging
+12 -6
View File
@@ -50,6 +50,11 @@ Changelog:
(kagentz CT 105 on minipve, .14, dedicated `hermes` user) and is monitored
from her side. This script must not probe mumuni or .24 — the v2 changelog
roster line was the last reference still placing her at .24 / CT100.
v6 (2026-09-28): .8 GPU health probe now runs as `llmuser` instead of `root`.
Root SSH to .8 was lost when the guest was rebuilt, so every .8 leg read as
UNREACHABLE for a healthy host. llmuser owns llama-server and can read
`systemctl is-active`, `systemctl show -p MainPID`, and the :8080 pid.
.110 and .15 keep the default `root` user.
"""
import subprocess, json, sys, os, time, re, io, contextlib
@@ -70,7 +75,7 @@ PVE_NODES = {
# Agent definitions: ct, host, user, pve_node, vault_key_name
AGENTS = {
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "minipve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
# abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
# local env file (key_env below), not from the shared vault or .bashrc.
# runtime=pi: abiba has run pi-only since the harness purge. There is no
@@ -95,7 +100,7 @@ AGENTS = {
# .110 rtx5070 (ocu-llm VM) -> llama-server.service (active)
# .15 strixhalo (amdpve) -> strix-server.service (active)
GPU_HOSTS = {
"gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-chat-api.service"},
"gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-chat-api.service", "user": "llmuser"},
"gpu-rtx5070 (.110)": {"host": "192.168.68.110", "port": 8080, "service": "llama-server.service"},
"gpu-strixhalo (.15)": {"host": "192.168.68.15", "port": 8080, "service": "strix-server.service"},
}
@@ -300,13 +305,14 @@ def check_gpu_ports():
host = gpu["host"]
port = gpu["port"]
svc = gpu["service"]
user = gpu.get("user", "root") # default root, overridden per-host where needed
# `systemctl is-active` exits non-zero when the unit is inactive or
# missing, which the ssh() helper would swallow as an SSH failure and
# report as UNREACHABLE. `|| true` keeps the real state word so we can
# tell "unit inactive" from "host unreachable".
svc_status = ssh(host, f"systemctl is-active {svc} || true")
port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")
svc_status = ssh(host, f"systemctl is-active {svc} || true", user=user)
port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1", user=user)
if not svc_status:
print(f" ❌ {label}: UNREACHABLE")
@@ -317,14 +323,14 @@ def check_gpu_ports():
print(f" ❌ {label}: PORT {port} NOT LISTENING (svc={svc_status})")
FAIL.append(f"gpu-no-port:{label}")
elif svc_status != "active":
svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2")
svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2", user=user)
if svc_pid and port_owner != svc_pid:
print(f" ❌ {label}: GHOST PROCESS — port owned by pid {port_owner}, svc pid {svc_pid} (svc={svc_status})")
FAIL.append(f"gpu-ghost:{label}:{port_owner}")
else:
print(f" ⚠️ {label}: svc={svc_status}, port owned by {port_owner}")
else:
health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health", user=user)
if health and '"status":"ok"' in health:
print(f" ✅ {label}: healthy (pid={port_owner})")
elif health and '"status":"no slot available"' in health:
+169 -38
View File
@@ -243,7 +243,7 @@ def collect():
("Pulse", "https://pulse.sysloggh.net"),
("Proxmox", "https://192.168.68.12:8006"),
("SearXNG", "http://192.168.68.7:8888"),
("Firecrawl", "http://192.168.68.7:3002/health"),
("Firecrawl", "http://192.168.68.7:3002/"), # Firecrawl serves no /health - the root is its liveness endpoint
]
report["endpoints"] = []
for name, url in endpoints:
@@ -389,6 +389,28 @@ def collect():
# ── HTML Dashboard ──
def classify_endpoint(code):
"""Classify an endpoint probe per the fleet's probe policy.
Codified 2026-09-14 in the monitoring contracts: ANY HTTP status proves the
service answered, so the service is ALIVE - 200/301/302/401/403/404 alike.
Only a failed CONNECTION (000 / timeout / refused) is a failed probe. A 404
from a wrong path is not a service fault and must not render as one.
This replaces a string comparison that was wrong in both directions
(`ep["code"] >= "400"`): it rendered 301 as red, 404 as yellow, and a real
500 as yellow. 5xx is kept as its own "server error" signal rather than
being merged with 4xx.
"""
if not code or code == "000":
return "red", "no connection"
if code.startswith("5"):
return "yellow", "server error"
if code.startswith(("2", "3", "4")):
return "green", "alive"
return "yellow", f"unexpected {code}"
def build_html(r):
issues = []
@@ -624,7 +646,7 @@ Proxmox: {r.get('pve_probe_status', 'ok')} ({r['nodes_online']}/{r['node_count']
# ── Network Endpoints ──
html += '<div class="card"><h2>🌐 Network Endpoints</h2><table><tr><th>Service</th><th>Status</th></tr>'
for ep in r["endpoints"]:
color = "green" if ep["code"] in ("200","302","401") else ("yellow" if ep["code"] >= "400" else "red")
color = classify_endpoint(ep["code"])[0]
html += f'<tr><td>{ep["name"]}</td><td class="{color}">HTTP {ep["code"]}</td></tr>'
html += '</table></div>'
@@ -688,42 +710,152 @@ Proxmox: {r.get('pve_probe_status', 'ok')} ({r['nodes_online']}/{r['node_count']
return html
# ── Send Email ──
# ── Delivery: Zulip DM carrying the report as an HTML ATTACHMENT ──
#
# Captain's decision, clarified 2026-09-26: the report is sent as an HTML FILE,
# i.e. an attachment - NOT HTML rendered in the message body, and NOT a Markdown
# translation of it. So the styled dashboard is built exactly as before, uploaded
# through Zulip's file-upload API, and the message body stays short: subject,
# top-line status, and a pointer to the attachment.
#
# This removes the Google dependency entirely (no SMTP, no EMAIL_PASSWORD).
# The 10,000-character message cap does not apply: it bounds message TEXT only,
# and the report travels as a file.
def send_email(html_content, subject_prefix=""):
FROM = "abiba@sysloggh.com"
TO = "jerome@sysloggh.com"
SUBJECT = f"{subject_prefix}{'🏗️ Infrastructure Report — ' + DATE_STR}"
msg = MIMEMultipart("alternative")
msg["From"] = FROM
msg["To"] = TO
msg["Subject"] = SUBJECT
msg.attach(MIMEText("Infrastructure report in HTML format — enable images to view.", "plain"))
msg.attach(MIMEText(html_content, "html"))
ZULIP_SITE = "https://chat.sysloggh.net"
ZULIP_BOT_EMAIL = "abiba-bot@chat.sysloggh.net"
CAPTAIN_USER_ID = 9
ZULIP_KEY_FILE = "/root/.pi/agent/extensions/zulip/.env"
REPORT_ARTIFACT_DIR = "/var/log/daily-infra-report"
def zulip_key():
"""abiba-bot's Zulip key, from the env or the on-host 600 file."""
key = os.environ.get("ABIBA_ZULIP_API_KEY")
if key:
return key.strip()
try:
EMAIL_PASSWORD = os.environ.get("EMAIL_PASSWORD") or os.environ.get("SMTP_PASSWORD") or os.environ.get("MAIL_PASSWORD")
if not EMAIL_PASSWORD:
print(" ⚠️ Degraded leg: credential-missing: EMAIL_PASSWORD (or SMTP_PASSWORD/MAIL_PASSWORD)", file=sys.stderr)
DEGRADED_LEGS.append("credential-missing: EMAIL_PASSWORD")
return True, "✅ Email leg degraded (no credential) — report still produced"
GMAIL_EMAIL = "jtabiri@gmail.com"
server = smtplib.SMTP("smtp.gmail.com", 587)
server.starttls()
server.login(GMAIL_EMAIL, EMAIL_PASSWORD)
server.sendmail(FROM, [TO], msg.as_string())
server.quit()
return True, "✅ Email sent to jerome@sysloggh.com"
except Exception as e:
return False, f"❌ Email failed: {e}"
with open(ZULIP_KEY_FILE) as fh:
for line in fh:
if line.startswith("ABIBA_ZULIP_API_KEY="):
return line.split("=", 1)[1].strip()
except OSError:
return None
return None
def build_summary(r, filename, test=False):
"""Short Markdown body: subject, top-line status, pointer to the attachment.
Deliberately NOT a reproduction of the report - the attachment is the report.
"""
nodes = f"{r.get('nodes_online', 0)}/{r.get('node_count', 0)} nodes online"
guests = f"{r.get('running_vms', 0)}/{r.get('total_vms', 0)} guests running"
lines = [
("\U0001F9EA **TEST — **" if test else "") + "\U0001F3D7\uFE0F **Infrastructure Report — " + DATE_STR + "**",
f"**{nodes}** \u00b7 **{guests}** \u00b7 generated {TIME_STR}",
]
problems = []
if r.get("pve_probe_status") != "ok":
problems.append(f"\u274c Proxmox probe: {r.get('pve_probe_status')}")
if r.get("resources_probe_status") != "ok":
problems.append(f"\u274c Resources probe: {r.get('resources_probe_status')}")
lit = r.get("litellm", {}) or {}
checks = lit.get("checks", []) or []
if checks:
passed = sum(1 for c in checks if c.get("status") == "pass")
if passed != len(checks):
problems.append(f"\u274c LiteLLM: {passed}/{len(checks)} checks pass")
if not (r.get("zulip_ext", {}) or {}).get("connected"):
problems.append("\u274c Zulip extension: not connected")
for leg in DEGRADED_LEGS:
problems.append(f"\u26a0\uFE0F degraded: {leg}")
lines.append("\n".join(problems) if problems else "\u2705 All monitored services healthy")
lines.append(f"\U0001F4CE **Full report attached:** `{filename}`")
return "\n\n".join(lines)
def _curl(args, timeout=60):
r = subprocess.run(["curl", "-s", "-m", str(timeout)] + args,
capture_output=True, text=True)
try:
return json.loads(r.stdout or "{}"), r.stdout
except json.JSONDecodeError:
return {}, r.stdout
def _curl_json(args, timeout=90):
r = subprocess.run(["curl", "-s", "-m", str(timeout)] + args,
capture_output=True, text=True)
try:
return json.loads(r.stdout or "{}"), r.stdout
except json.JSONDecodeError:
return {}, r.stdout
def send_zulip(html_content, report, test=False):
"""Upload the styled HTML and post a short pointer to the captain's DM.
Returns (ok, message). On ANY failure the report body is also printed to
stdout and persisted to disk, so a delivery failure can never swallow the
content - the defect this folds in.
"""
os.makedirs(REPORT_ARTIFACT_DIR, exist_ok=True)
stamp = NOW.strftime("%Y%m%d-%H%M%S")
filename = f"infra-report-{stamp}.html"
html_path = os.path.join(REPORT_ARTIFACT_DIR, filename)
try:
with open(html_path, "w") as fh:
fh.write(html_content)
except OSError as e:
print(f" \u26a0\uFE0F could not persist report artifact: {e}", file=sys.stderr)
key = zulip_key()
if not key:
print(html_content) # never swallow the content
return False, ("\u274c Delivery FAILED: no Zulip credential "
"(ABIBA_ZULIP_API_KEY unset and "
f"{ZULIP_KEY_FILE} unreadable). Report persisted to {html_path}")
auth = ["-u", f"{ZULIP_BOT_EMAIL}:{key}"]
# 1. Upload the report as a file.
up, up_raw = _curl_json(auth + [
"-X", "POST", f"{ZULIP_SITE}/api/v1/user_uploads",
"-F", f"file=@{html_path};type=text/html",
])
if up.get("result") != "success" or not up.get("uri"):
print(html_content)
return False, (f"\u274c Delivery FAILED at upload: {up.get('msg') or up_raw[:160]} "
f"(report persisted to {html_path})")
uri = up["uri"]
size = os.path.getsize(html_path)
# 2. Post a short message pointing at it.
body = build_summary(report, filename, test=test)
link = f"[{filename}]({uri})"
body = body.replace(f"`{filename}`", link)
payload, raw = _curl_json(auth + [
"-X", "POST", f"{ZULIP_SITE}/api/v1/messages",
"-d", "type=private",
"-d", f"to=[{CAPTAIN_USER_ID}]",
"--data-urlencode", f"content={body}",
])
if payload.get("result") == "success":
return True, (f"\u2705 Delivered to Zulip DM (user {CAPTAIN_USER_ID}), "
f"message id {payload.get('id')}, attachment {size} bytes at {uri}")
print(html_content)
return False, (f"\u274c Delivery FAILED at message post: {payload.get('msg') or raw[:160]} "
f"(uploaded {uri}; report persisted to {html_path})")
# ── Main ──
if __name__ == "__main__":
is_test = "--test-email" in sys.argv
is_test = ("--test-email" in sys.argv) or ("--test-zulip" in sys.argv)
print(f"{'🧪 TEST MODE' if is_test else '📊'} Collecting infrastructure data...")
report = collect()
@@ -738,15 +870,14 @@ if __name__ == "__main__":
print(" Building dashboard...")
html = build_html(report)
print(f" report ready: {len(html)} chars of HTML (delivered as a file attachment)")
if is_test:
prefix = "🧪 TEST — "
print(" Sending test email...")
print(" Sending TEST message to the captain's Zulip DM...")
else:
prefix = ""
print(" Sending email...")
ok, msg = send_email(html, subject_prefix=prefix)
print(" Sending to the captain's Zulip DM...")
ok, msg = send_zulip(html, report, test=is_test)
print(f" {msg}")
# Show summary
+2 -2
View File
@@ -101,8 +101,6 @@ GUESTS: list[Guest] = [
# amdpve (192.168.68.15)
Guest(ct_id="105", hostname="kagentz", ip="192.168.68.105", node="amdpve",
access_method="ssh-host", probe_target="kagentz (CT 105, amdpve)"),
Guest(ct_id="112", hostname="tanko", ip="192.168.68.112", node="amdpve",
access_method="pct-run", probe_target="tanko (CT 112, amdpve)"),
Guest(ct_id="113", hostname="baggy", ip="192.168.68.113", node="amdpve",
access_method="pct-run", probe_target="baggy (CT 113, amdpve)"),
Guest(ct_id="115", hostname="scottdenya", ip="192.168.68.115", node="amdpve",
@@ -110,6 +108,8 @@ GUESTS: list[Guest] = [
Guest(ct_id="120", hostname="adguard2", ip="192.168.68.120", node="amdpve",
access_method="pct-run", probe_target="adguard2 (CT 120, amdpve)"),
# minipve (192.168.68.12)
Guest(ct_id="112", hostname="tanko", ip="192.168.68.112", node="minipve",
access_method="pct-run", probe_target="tanko (CT 112, minipve)"),
Guest(ct_id="100", hostname="abiba", ip="192.168.68.100", node="minipve",
access_method="pct-run", probe_target="abiba (CT 100, minipve)"),
Guest(ct_id="102", hostname="adguard", ip="192.168.68.102", node="minipve",
+1 -1
View File
@@ -12,7 +12,6 @@ set -euo pipefail
declare -A CT_NODES=(
# amdpve (192.168.68.15)
[105]=amdpve # kagentz (was hwepve — corrected 2026-09-12; live per pvesh)
[112]=amdpve # tanko
[113]=amdpve # baggy
[115]=amdpve # scottdenya
[120]=amdpve # adguard2 (added 2026-09-12)
@@ -21,6 +20,7 @@ declare -A CT_NODES=(
[102]=minipve # adguard (was acerpve)
[104]=minipve # authentik
[110]=minipve # gitea
[112]=minipve # tanko (was amdpve — migrated 2026-09-27; live per pvesh)
[116]=minipve # syslog-api
[119]=minipve # infisical-vault
# storepve (192.168.68.6)
+2 -2
View File
@@ -47,8 +47,8 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
**Proxmox Cluster "Tabiri" (5 nodes):**
- amdpve (192.168.68.15): kagentz, tanko, baggy, scottdenya, adguard2
- minipve (192.168.68.12): abiba, adguard, authentik, gitea, syslog-api, infisical-vault
- amdpve (192.168.68.15): kagentz, baggy, scottdenya, adguard2
- minipve (192.168.68.12): abiba, tanko, adguard, authentik, gitea, syslog-api, infisical-vault
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna
- acerpve (192.168.68.9): llm-gpu
- ocupve (192.168.68.5): ocu-llm
+6 -6
View File
@@ -135,16 +135,16 @@ case "$PI_VERDICT" in
esac
# -- abiba-leg-end
# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ──
# ── Platform B: Tanko (DSH dsh-web on minipve CT 112) ──
# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
# key availability varies — so probes run from the amdpve vantage via `pct exec`.
# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on amdpve
# (192.168.68.15). The gateway binds 127.0.0.1:3080 loopback-only by design — a
# key availability varies — so probes run from the minipve vantage via `pct exec`.
# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on minipve
# (192.168.68.12). The gateway binds 127.0.0.1:3080 loopback-only by design — a
# remote :3080 probe is refused and is NOT a fault.
TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.12 \
"pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true)
[ -n "$TANKO_SVC" ] || TANKO_SVC="unknown"
TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.12 \
"pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true)
[ -n "$TANKO_HTTP" ] || TANKO_HTTP="000"
@@ -0,0 +1,236 @@
"""Regression test for the fallback_providers list-shape crash in audit-hermes-config.py.
WHY THIS FILE EXISTS: audit-hermes-config.py assumed `fallback_providers` was always a dict
(single provider). Two live agents (koby, koonimo) carry it as a LIST of dicts (one entry per
fallback), so the script crashed with:
File "audit-hermes-config.py", line 211, in audit
fb.get("provider") == "deepseek",
AttributeError: 'list' object has no attribute 'get'
Both are REAL agent configs, so this is not a malformed-input case — the script simply could not
audit two of the four agents it exists to audit. Until fixed, the key-hygiene check had no
coverage for half the fleet while appearing to run.
These tests execute the real CLI (`python3 audit-hermes-config.py <config>`) and assert:
1. A config whose `fallback_providers` is a LIST of valid dicts does NOT crash (exit code is 0 or 1,
never a traceback/AttributeError).
2. A config whose `fallback_providers` contains a MALFORMED entry (a list element that is not a
mapping) reports a VIOLATION naming the offending entry, NOT an uncaught exception.
3. The dict shape still works (existing tests must stay green).
No network, vault, or SSH access is required.
"""
from __future__ import annotations
import pathlib
import subprocess
import sys
ROOT = pathlib.Path(__file__).resolve().parent.parent
AUDIT = ROOT / "audit-hermes-config.py"
# A valid config where fallback_providers is a LIST of dicts (the real koby/koonimo shape).
# One entry, well-formed: provider=deepseek, model=deepseek-v4-flash, api_key_env=DEEPSEEK_API_KEY.
# This must produce a real verdict (PASS or FAIL) without crashing.
LIST_SHAPE_VALID = """
model:
api_key: ""
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
max_tokens: 4096
default: syslog-auto
provider: harness
fallback_providers:
- provider: deepseek
model: deepseek-v4-flash
api_key_env: DEEPSEEK_API_KEY
compression:
model: syslog-auto
provider: harness
threshold: 0.65
max_context_window: 131072
auxiliary:
vision:
model: gpu-vision
provider: harness
web_extract:
model: gpu-vision
provider: harness
compression:
model: syslog-auto
provider: harness
delegation:
provider: harness
custom_providers:
- name: harness
key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
"""
# A valid config where fallback_providers is a LIST with TWO entries (multiple fallbacks).
# Both entries well-formed. Must not crash and should produce a real verdict.
LIST_SHAPE_MULTI = """
model:
api_key: ""
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
max_tokens: 4096
default: syslog-auto
provider: harness
fallback_providers:
- provider: deepseek
model: deepseek-v4-flash
api_key_env: DEEPSEEK_API_KEY
- provider: deepseek
model: deepseek-v4-flash
api_key_env: DEEPSEEK_API_KEY
compression:
model: syslog-auto
provider: harness
threshold: 0.65
max_context_window: 131072
auxiliary:
vision:
model: gpu-vision
provider: harness
web_extract:
model: gpu-vision
provider: harness
compression:
model: syslog-auto
provider: harness
delegation:
provider: harness
custom_providers:
- name: harness
key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
"""
# A config where fallback_providers is a LIST containing a MALFORMED entry:
# one element is a plain string, not a mapping. The checker must report a VIOLATION
# naming the offending entry (fallback_providers[1]) and NOT crash.
LIST_SHAPE_MALFORMED = """
model:
api_key: ""
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
max_tokens: 4096
default: syslog-auto
provider: harness
fallback_providers:
- provider: deepseek
model: deepseek-v4-flash
api_key_env: DEEPSEEK_API_KEY
- "not-a-mapping"
compression:
model: syslog-auto
provider: harness
threshold: 0.65
max_context_window: 131072
auxiliary:
vision:
model: gpu-vision
provider: harness
web_extract:
model: gpu-vision
provider: harness
compression:
model: syslog-auto
provider: harness
delegation:
provider: harness
custom_providers:
- name: harness
key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
"""
# The original DICT shape (single provider) must still work — existing behaviour preserved.
DICT_SHAPE_VALID = """
model:
api_key: ""
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
max_tokens: 4096
default: syslog-auto
provider: harness
fallback_providers:
provider: deepseek
model: deepseek-v4-flash
api_key_env: DEEPSEEK_API_KEY
compression:
model: syslog-auto
provider: harness
threshold: 0.65
max_context_window: 131072
auxiliary:
vision:
model: gpu-vision
provider: harness
web_extract:
model: gpu-vision
provider: harness
compression:
model: syslog-auto
provider: harness
delegation:
provider: harness
custom_providers:
- name: harness
key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
"""
def _run_config(tmp_path, name, text):
cfg = tmp_path / name
cfg.write_text(text)
proc = subprocess.run(
[sys.executable, str(AUDIT), str(cfg)],
capture_output=True, text=True,
)
return proc.returncode, proc.stdout, proc.stderr
def test_list_shape_single_entry_does_not_crash(tmp_path):
"""A LIST with one valid dict must not raise AttributeError; exit 0 (PASS)."""
code, out, err = _run_config(tmp_path, "list-single.yaml", LIST_SHAPE_VALID)
# Must NOT be a crash (traceback). A clean run exits 0 (PASS) or 1 (FAIL), never 2+ (exception).
assert code in (0, 1), f"Expected clean exit 0 or 1, got {code}\nSTDOUT:\n{out}\nSTDERR:\n{err}"
assert "AttributeError" not in err, f"Crashed with AttributeError:\n{err}"
assert "Traceback" not in err, f"Crashed with uncaught exception:\n{err}"
# The valid single-entry list should PASS (all rules satisfied).
assert code == 0, f"Expected PASS but got {code}\n{out}"
assert "RESULT: PASS" in out
def test_list_shape_multiple_entries_does_not_crash(tmp_path):
"""A LIST with two valid dicts must not raise AttributeError; exit 0 (PASS)."""
code, out, err = _run_config(tmp_path, "list-multi.yaml", LIST_SHAPE_MULTI)
assert code in (0, 1), f"Expected clean exit 0 or 1, got {code}\nSTDOUT:\n{out}\nSTDERR:\n{err}"
assert "AttributeError" not in err, f"Crashed with AttributeError:\n{err}"
assert "Traceback" not in err, f"Crashed with uncaught exception:\n{err}"
assert code == 0, f"Expected PASS but got {code}\n{out}"
assert "RESULT: PASS" in out
def test_list_shape_malformed_entry_reports_violation_not_crash(tmp_path):
"""A LIST containing a non-mapping element must be a reported VIOLATION, not a crash."""
code, out, err = _run_config(tmp_path, "list-malformed.yaml", LIST_SHAPE_MALFORMED)
# Must NOT be a crash.
assert "AttributeError" not in err, f"Crashed with AttributeError:\n{err}"
assert "Traceback" not in err, f"Crashed with uncaught exception:\n{err}"
# Should be a FAIL (exit 1) because the malformed entry is a violation.
assert code == 1, f"Expected FAIL (exit 1) but got {code}\n{out}"
assert "RESULT: FAIL" in out
# The violation must name the offending entry (fallback_providers[1]).
assert "fallback_providers[1]" in out, f"Violation did not name the offending entry:\n{out}"
def test_dict_shape_still_passes(tmp_path):
"""The original DICT shape (single provider) must still PASS — existing behaviour preserved."""
code, out, err = _run_config(tmp_path, "dict-valid.yaml", DICT_SHAPE_VALID)
assert code == 0, f"Expected PASS but got {code}\n{out}\nSTDERR:\n{err}"
assert "RESULT: PASS" in out
+2 -2
View File
@@ -49,7 +49,7 @@ HEALTH_CONTRACT = ROOT / "zulip-health.prose.md"
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
MUMUNI_IP = "192.168.68.24" # Mumuni's old (decommissioned) deployment
TANKO_VANTAGE = "192.168.68.15" # amdpve — Tanko CT 112 via pct exec
TANKO_VANTAGE = "192.168.68.12" # minipve — Tanko CT 112 via pct exec
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
@@ -82,7 +82,7 @@ done
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
cmd="${*: -1}"
case "$host" in
192.168.68.15)
192.168.68.12)
case "$cmd" in
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
*curl*) printf '%s' "$TANKO_HTTP" ;;
+53
View File
@@ -104,6 +104,59 @@ def test_koby_ct111_is_on_storepve(ahc):
assert ahc.AGENTS["koby"]["pve"] == "storepve"
def test_tanko_ct112_is_probed_on_minipve(ahc, monkeypatch, capsys):
# CT 112 (tanko) was live-migrated to minipve (.12) on 2026-09-27; the
# amdpve mapping made `pct status 112` fail and read as ct-unreachable.
# Execute the probe and assert the host the script actually contacts.
probes = []
monkeypatch.setattr(
ahc, "ssh",
lambda host, cmd, user="root": probes.append((host, cmd)) or "status: running",
)
ahc.FAIL.clear()
ahc.REPORT_ONLY.clear()
try:
ahc.check_ct_liveness()
tanko_hosts = [h for h, cmd in probes if cmd == "pct status 112 2>/dev/null"]
assert tanko_hosts == ["192.168.68.12"]
finally:
ahc.FAIL.clear()
ahc.REPORT_ONLY.clear()
def test_gpu_rtx3090_probe_uses_llmuser_not_root(ahc, monkeypatch, capsys):
# 2026-09-28: root SSH to .8 was lost when the guest was rebuilt; llmuser
# owns llama-server and can read systemctl status and the :8080 pid. A root
# probe reads as UNREACHABLE for a healthy host (the reported bug). Execute
# check_gpu_ports() against an SSH boundary that only accepts llmuser@.8 and
# assert the .8 leg does not produce the false UNREACHABLE failure.
seen = []
def fake_ssh(host, cmd, user="root"):
seen.append((host, user))
if host == "192.168.68.8" and user != "llmuser":
return None # root SSH denied -> baseline false UNREACHABLE
if cmd.startswith("systemctl is-active"):
return "active"
if cmd.startswith("ss -tlnp"):
return "48351"
if cmd.startswith("curl"):
return '{"status":"ok"}'
return None
monkeypatch.setattr(ahc, "ssh", fake_ssh)
ahc.FAIL.clear()
try:
ahc.check_gpu_ports()
out = capsys.readouterr().out
assert "gpu-unreachable:192.168.68.8" not in ahc.FAIL
assert "\u2705 gpu-rtx3090 (.8): healthy" in out
assert ("192.168.68.8", "llmuser") in seen
assert not any(host == "192.168.68.8" and user == "root" for host, user in seen)
finally:
ahc.FAIL.clear()
def test_report_only_legs_never_count_as_failures(ahc):
for agent, report_only in (("koby", True), ("koonimo", False), ("tanko", False)):
ahc.FAIL.clear()
+2 -2
View File
@@ -34,7 +34,7 @@ ROOT = pathlib.Path(__file__).resolve().parents[1]
ZULIP_MONITOR = ROOT / "scripts" / "zulip-monitor.sh"
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
TANKO_VANTAGE = "192.168.68.15" # amdpve — Tanko CT 112 via pct exec
TANKO_VANTAGE = "192.168.68.12" # minipve — Tanko CT 112 via pct exec
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
@@ -52,7 +52,7 @@ done
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
cmd="${*: -1}"
case "$host" in
192.168.68.15)
192.168.68.12)
case "$cmd" in
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
*curl*) printf '%s' "$TANKO_HTTP" ;;
+24 -24
View File
@@ -28,7 +28,7 @@ session start.
## Requires
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
- **SSH access** to minipve (192.168.68.12) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
- **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `kagentz.sysloggh.net` (C3 public path), `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -228,20 +228,20 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
| Crash loop >10/h | Alert user |
### Step 3: Platform B — Tanko (DSH on amdpve CT 112)
### Step 3: Platform B — Tanko (DSH on minipve CT 112)
Mumuni is out of scope for this host (see the note above): she runs on her own
container and is monitored on her side.
Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve**
PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is not a dependency
as the `dsh-web` systemd unit inside **CT 112**, which resides on the **minipve**
PVE host (**192.168.68.12**). Direct SSH to 192.168.68.122 is not a dependency
of this contract — per-worker key availability varies — so CT 112 probes run
from the amdpve vantage via `pct exec`:
from the minipve vantage via `pct exec`:
```bash
ssh root@192.168.68.15 "pct exec 112 -- <command>"
ssh root@192.168.68.12 "pct exec 112 -- <command>"
```
> **By design (verified 2026-09-08):** the `dsh-web` gateway binds
@@ -253,7 +253,7 @@ ssh root@192.168.68.15 "pct exec 112 -- <command>"
**B1: Gateway Service State (Tanko)**
```bash
ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web"
ssh root@192.168.68.12 "pct exec 112 -- systemctl is-active dsh-web"
```
Expected: `active`. Anything else → gateway service down → apply the Tanko heal
@@ -262,7 +262,7 @@ Expected: `active`. Anything else → gateway service down → apply the Tanko h
**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**
```bash
ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
ssh root@192.168.68.12 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
```
Alive = **ANY** HTTP status response from the endpoint — the expected set is
@@ -273,13 +273,13 @@ process answering `503` is running and self-heal must NOT restart-loop it.
Down = connection refused (`000`) or timeout only. Statuses outside the
expected set are logged/reported as a warning — reported, never healed on.
**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)**
**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to minipve)**
```bash
curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
```
Fallback only — used when the monitoring node has no pct/SSH path to amdpve.
Fallback only — used when the monitoring node has no pct/SSH path to minipve.
Alive = **ANY** HTTP status response from the endpoint — healthy signals are
`302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other
status, including `404`/`5xx`, also counts alive: the endpoint is up and
@@ -404,29 +404,29 @@ ExecStartPost=/bin/systemctl --no-block start dsh-web-token.service
4. Every later request through `/` presents that cookie; the token is not needed
again until the cookie expires or a new browser is used.
**Verification** (amdpve vantage):
**Verification** (minipve vantage):
```bash
# 1. Login endpoint is Authentik-gated: unauthenticated -> 302 (not 200/303).
ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}\n' \
ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}\n' \
-H 'Host: tankodhs.sysloggh.net' http://127.0.0.1/dsh-web-login"
# Expected: 302
# 2. Legacy :8081 endpoint is gone (connection refused -> 000).
ssh root@192.168.68.15 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \
ssh root@192.168.68.12 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \
-w '%{http_code}\n' http://192.168.68.122:8081/"
# Expected: 000
# 3. Backend cookie mint + reuse (exactly what /dsh-web-login proxies to).
TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token")
ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \
TOKEN=$(ssh root@192.168.68.12 "pct exec 112 -- cat /etc/dsh-web/launch-token")
ssh root@192.168.68.12 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
# Expected: 200 — the minted dsh-auth-... cookie (authority
# tankodhs.sysloggh.net) is replayed on the next request and accepted.
# 4. Token refresh is non-disruptive and idempotent.
ssh root@192.168.68.15 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh"
ssh root@192.168.68.12 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh"
# Expected: "token unchanged; nginx not reloaded" when nothing changed
```
@@ -437,32 +437,32 @@ fresh cookie. Both verified live 2026-09-11.
```bash
# 5. Cookie survives a dsh-web restart, and the new token mints a new cookie.
ssh root@192.168.68.15 "pct exec 112 -- systemctl restart dsh-web"
ssh root@192.168.68.12 "pct exec 112 -- systemctl restart dsh-web"
# dsh-web is Type=simple: restart returns before :3080 is listening. Bounded-poll
# until the socket answers (any status but 000) before asserting the cookie.
for i in $(seq 1 60); do
UP=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
UP=$(ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
-H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/")
[ "$UP" != "000" ] && break
sleep 2
done
ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
# Expected: 200 — the pre-restart cookie is still accepted.
# The restart's ExecStartPost (or the 2-minute timer) refreshes the include. A
# manual run may no-op on the flock, so poll until the include carries a token
# the running process accepts (bounded wait) before the mint+reuse check.
for i in $(seq 1 60); do
TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'")
CODE=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
TOKEN=$(ssh root@192.168.68.12 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'")
CODE=$(ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'")
[ "$CODE" = "303" ] && break
sleep 2
done
# Expected: 303 — the include now holds the token the running process accepts.
ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh-new.jar -o /dev/null \
ssh root@192.168.68.12 "pct exec 112 -- curl -s -c /tmp/dsh-new.jar -o /dev/null \
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null \
ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null \
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
# Expected: 200 — the refreshed token minted a fresh cookie.
```