Compare commits

..
Author SHA1 Message Date
agent-zero 832b184af6 fix(litellm): upgrade contract refs 1.90.0-rc.1 -> 1.99.1 and correct redis role
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
- harness-litellm pinned tag updated to 1.99.1 in infrastructure-control,
  litellm-health, litellm-self-heal container tables
- infrastructure-update MCP per-key limitation note now cites v1.99.1
- harness-redis role corrected: dead 'Router slots, circuit breakers' ->
  'LiteLLM cache + rate-limit state' (router decommissioned 2026-09-11)
- litellm-self-heal Rule 9 wording clarified for cache/rate-limit only

Live verification 2026-09-11: harness-litellm healthy, /health 200/200,
7 models, 46 keys, prisma migrations 127 -> 157.
2026-09-11 15:05:00 -04:00
agent-zero c26255f5ff fix(router): decommission legacy GPU router across contracts and fleet scripts
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Router container, image, and config are purged on CT 116 (verified: no
container, no image, inference-harness-router:latest removed, :9000 free,
11 containers healthy, 7 models, live syslog-auto completion OK). Updates:

- gpu-fleet.prose.md: topology diagram rebuilt without the router tier
- gpu-monitor / gpu-self-heal / litellm-health / litellm-self-heal: router steps dropped
- infrastructure-control.prose.md: container inventory + litellm row corrected
- scripts/daily-infra-report.py, scripts/prose-ai-review.sh: host-scoped checks

Verified: 40/40 anchors matched uniquely, 0 U+FFFD across changed files,
daily-infra-report.py compiles, no stray harness-router references remain.
2026-09-11 14:20:18 -04:00
mumuni-bot 64ccf65eaf scot-hermes-playbook: send-ready package (PDF render + delivery record + render script) for t_2c716052 2026-09-11 17:40:56 +00:00
mumuni-bot c7af7c0689 Add reviewed Hermes playbook for Scot Murray (client deliverable)
Provenance: kanban t_02148213 (research) -> t_4265c369 (author) -> t_fefdf30b (review).
Review verdict: APPROVED-WITH-FIXES, 7/7 checks PASS, 17/17 YouTube links oEmbed-verified,
26+ CLI commands re-run against v0.21.1, no Murray/JDS client data present.

Artifact: deliverables/scot-hermes-playbook/HERMES-PLAYBOOK-FOR-SCOT.md
  377 lines, sha256 b0966a649fec96da4975ec00e627fbaba3a92a62c4a92b33bc06589c47d25f7f
Includes the research dossier and its raw verification evidence under research/.

STATUS: written and reviewed, NOT delivered to the client. Delivery is gated on
Kwame's approval and tracked as a separate blocked kanban card.
2026-09-11 12:54:52 -04:00
mumuni-bot fb185ed90a Merge pull request 'feat(memory-fixer): v2.1.0 — stale nodes auto-archived (Kwame directive 2026-09-11)' (#72) from feat/memory-fixer-auto-archive-20260911 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-09-11 12:12:52 +00:00
mumuni-bot b001b657d4 feat(memory-fixer): v2.1.0 — stale nodes are auto-archived (Kwame directive 2026-09-11)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- New Level 1 fix 4: archive-suggested stale nodes are archived outright via one
  updateNode call (description -> '[ARCHIVED] ' + metadata state=archived).
  The [REVIEW: archive] tag is retired.
- Fix 3 now tags refresh-suggested (living) nodes only; those are still escalated.
- Corrected the SSH claim: updateNode DOES accept state transitions and bumps
  updated_at (verified 2026-09-11 archiving 7 nodes: #61 #373 #388 #465 #475 #526 #1476).
- Level 2 escalation 1 rescoped to refresh-suggested nodes.
- Checks section: any remaining [REVIEW: archive] means fix 4 was skipped.
2026-09-11 12:10:16 +00:00
abiba-bot a1ffeaad34 Merge pull request 'update(infrastructure-update): v1.3.0 — full 5-host Docker ecosystem coverage (live-verified 2026-09-08)' (#64) from update/docker-ecosystems-20260908 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-09-10 23:21:44 +00:00
root 287657a77a Merge remote-tracking branch 'origin/master' into update/docker-ecosystems-20260908
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-09-10 23:01:32 +00:00
root 21f9073e0b fix(daily-infra-report): use pve_probe_status in render + add regression tests (PR #64 round 2) 2026-09-10 23:01:18 +00:00
root 32fe7c0652 fix(daily-infra-report): complete Zulip nested-key fix + Proxmox port/auth fix (PR #64 review fix 1-2)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-09-10 22:03:48 +00:00
root 25cf2f5eef fix(daily-infra-report): read Zulip state from nested 'zulip' key (PR #64 fix 3a)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-09-10 20:55:17 +00:00
root 26f2301188 fix(pm2-self-heal): remove stray fragment so script parses (PR #64 fix 2)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-09-10 20:43:01 +00:00
root a6a459acc0 fix: correct Firecrawl GET / response code from 404 to 200 (PR #64)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-09-10 20:42:15 +00:00
abiba-bot 14bed6e916 Merge pull request 'fix(monitoring): stop monitoring Mumuni from this host (moved to CT105/.14)' (#71) from fm/denya-mumuni-monitor-removal-20260910 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-09-10 12:30:19 +00:00
agent-zero 2e70c834cb docs(infrastructure-update): add mandatory digest-pin sweep to Wave 3 verification
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Digest-pinned images (image: repo@sha256:...) are invisible to 'docker compose
pull' — the pin re-pulls the same digest forever, so new releases never appear.
Verified 2026-09-10: audiobookshelf ran 2.34.0 for 7 weeks despite weekly pulls;
un-pinned to :latest, now 2.36.0 (HTTP 200). Dockhand stack (also pinned) was
removed 2026-09-10 as unused (user decision). Weekly task qSOOVzsU now sweeps
for digest pins every run and flags them for user-approved un-pinning.
2026-09-10 06:30:07 -04:00
abiba-bot 4f59b82404 no-mistakes(document): Refresh stale Mumuni roster docs and registry metadata
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-09-10 10:26:00 +00:00
abiba-bot 8a4ee4cf5a no-mistakes(review): Pin Mumuni removal behaviorally and fix digest import bug 2026-09-10 10:18:38 +00:00
abiba 952fca9c92 fix(monitoring): retire the Mumuni leg — stop probing her decommissioned .24 deployment
Captain ruling 2026-09-10: Mumuni moved off this host onto her own container
(kagentz CT 105 on minipve, 192.168.68.14, dedicated `hermes` user) and is
monitored from her side. This host must not monitor anything Mumuni.

The stale probes fired false alerts repeatedly:
  * scripts/zulip-monitor.sh ssh'd to root@192.168.68.24 for the
    decommissioned deployment's ~/.hermes/gateway_state.json, read "unknown"
    on every run, and posted a 🔴 "Mumuni (Hermes) Zulip state: unknown" DM +
    #agent-hub stream alert each cycle.
  * scripts/daily-infra-report.py published a matching "mumuni:unknown" row in
    every digest.

Changes:
  * zulip-monitor.sh: delete the "Platform B: Hermes (Mumuni)" leg and its
    notify; keep the Zulip-server, Platform A pi/Abiba (bridge), Platform B
    Tanko and Platform C Agent Zero legs. A comment records why the leg is
    retired so it is not re-added. Also fixes SC2155 so shellcheck is clean.
  * daily-infra-report.py: delete the .24 ~/.hermes/gateway_state.json agent
    probe, its hermes --version probe, and the now-dead mumuni render branch.
    Abiba (CT 100, its own .24 address) and Tanko legs unchanged; the PVE API
    token name is untouched.
  * zulip-health.prose.md (v3.1.0): drop the Mumuni-only B4 gateway-process,
    B5 heartbeat and B6 response-delivery steps and the stale 192.168.68.24
    references; state explicitly that Mumuni is not monitored from this host.
    Tanko/Agent-Zero/bridge steps retained.
  * agent-health-check.py: correct the v2 changelog roster comment that still
    placed mumuni at .24/CT100. No behavior change — the mumuni probe was
    already absent from the AGENTS dict; v5 changelog notes the correction.

Tests: tests/test_mumuni_monitor_removal.py pins the removal structurally and
behaviorally — the shipped zulip-monitor.sh is run in a sandbox (only its LOG
constant rewritten) with stub ssh/curl on PATH; the ssh stub records every
target host, so "never reaches .24" and "no Mumuni notify even on the alert
path" are asserted from observed behavior. A mutation check (re-inject the old
leg) fails the suite, so the guarantee is not vacuous.
2026-09-10 10:11:19 +00:00
abiba-bot b1462f3e79 Merge pull request 'fix(monitoring): probe-drift round 2 — gpu port, PVE liveness, report-only visibility, report provenance' (#70) from fm/probe-drift-round2-20260909 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Skipped
2026-09-10 02:17:01 +00:00
agent-zero bfdff13ae7 docs(infrastructure-update): point weekly trigger at scheduler task qSOOVzsU
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Weekly Sunday 03:00 America/New_York run is now implemented as Agent Zero
scheduled task 'weekly-fleet-docker-update' (qSOOVzsU) with built-in
post-update service verification and old-image pruning.
2026-09-08 05:27:38 -04:00
agent-zero 8bf32f6f0f update(infrastructure-update): v1.3.0 — full Docker ecosystem coverage from 2026-09-08 live run
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s
- Add 5 Docker ecosystems: docker-vm .7, CT 116 .116, CT 117 (zulip+jitsi),
  hwpve .11 (Authentik), NetBird VPS 72.61.0.17
- Wave 3: add trove-test, docker-stats, monitoring stack, Zulip (pct exec + compose
  recreate preserves zulip_default network), Jitsi, Authentik, NetBird
- Firecrawl test is now POST /v1/search (GET / returns 404 by design)
- Add Authentik 302 + NetBird dashboard 200 checks
- Document harness-litellm 3-5 min cold start after recreate (verified 2026-09-08)
- Add hwpve + VPS compose files to config backup list; success criteria covers 5 hosts
2026-09-08 03:23:03 -04:00
36 changed files with 2763 additions and 235 deletions
+3 -3
View File
@@ -628,18 +628,18 @@ contracts:
sensitivity: high
status: active
owner: abiba
version: 3.0.0
version: 3.1.0
trigger:
type: scheduled
cadence: '*/15 * * * *'
description: "Every 15 minutes \u2014 monitors all Zulip-connected agents"
description: "Every 15 minutes \u2014 monitors the Zulip-connected agents under this host's control (pi, DSH, Agent Zero)"
cron_job_id: null
execution:
agent: abiba
timeout: 120
requires:
- Zulip API key for abiba-bot@chat.sysloggh.net
- SSH access to all Hermes agents
- SSH access to amdpve (192.168.68.15) for Tanko (CT 112) and the Agent Zero Docker host (.14)
verification:
postconditions:
- check: bot registration active
@@ -0,0 +1,75 @@
# Delivery Record — HERMES-PLAYBOOK-FOR-SCOT
## Status: SEND-READY — awaiting Kwame's channel + recipient confirmation
No documented channel to Scot Murray exists in this workspace, the skills, or config
(verified 2026-09-11 by sweep of `~/syslog/projects/murray-capital/`, `syslog-infra`
references, `murray-harness` skill, `.hermes/memories/`, all of `~/syslog/`).
Per the card's unblock constraints: package prepared, exact send commands written
below, nothing transmitted. Guessing an address is out of scope.
## Verified artifact (single source of truth)
| Item | Value |
|---|---|
| Markdown source | `/home/hermes/syslog/prose-contracts/deliverables/scot-hermes-playbook/HERMES-PLAYBOOK-FOR-SCOT.md` |
| sha256 | `b0966a649fec96da4975ec00e627fbaba3a92a62c4a92b33bc06589c47d25f7f` |
| Size | 28821 bytes, 377 lines |
| Matches reviewed bytes | YES — identical to `/home/hermes/syslog/drafts/scot-hermes-playbook/` copy and to the hash recorded on card t_2c716052 |
## Rendered PDF (from the verified bytes, no edits)
| Item | Value |
|---|---|
| PDF | `/home/hermes/syslog/prose-contracts/deliverables/scot-hermes-playbook/HERMES-PLAYBOOK-FOR-SCOT.pdf` |
| sha256 | `f46c0c89b4bbf6fa76c1f1c385c87753860d06bfa9e8925d6e09f27ed78a187b` |
| Size | 96,486 bytes · 13 pages · A4 |
| Render chain | pandoc 3.1.11.1 (gfm → html5) + WeasyPrint 62.3, stylesheet `pb.css`; reproducible via `bash render_pdf.sh` |
| Spot-check | pdftotext shows correct title page + v0.21.1 verification note |
## Candidate channels — exact commands (pending Kwame's pick + address)
### 1. Email via syslog-email profile (recommended)
Mailbox ops belong to the syslog-email profile per standing rule. Send as
jerome@sysloggh.com with both attachments.
```
hermes -p syslog-email chat -q "Send an email. From jerome@sysloggh.com. \
To: <SCOT-ADDRESS — Kwame to supply>. Subject: 'Hermes Playbook — getting real mileage out of the harness'. \
Attach: /home/hermes/syslog/prose-contracts/deliverables/scot-hermes-playbook/HERMES-PLAYBOOK-FOR-SCOT.pdf \
and HERMES-PLAYBOOK-FOR-SCOT.md. Body: short intro noting the PDF is the reviewed v0.21.1 playbook, \
sha256 b0966a64… (sic, abbreviated), ask him to flag anything confusing — that feedback feeds the harness build. \
Show me the draft before sending."
```
Direct himalaya (only if Kwame wants it from the main session — normally NOT, per
the email-routing standing rule):
```
himalaya message write --to "<SCOT-ADDRESS>" --subject "Hermes Playbook — getting real mileage out of the harness" \
--attachment .../HERMES-PLAYBOOK-FOR-SCOT.pdf --attachment .../HERMES-PLAYBOOK-FOR-SCOT.md
himalaya message send <draft.eml>
```
### 2. Telegram (only if Kwame has Scot's handle)
Send the PDF to Scot's handle from the gateway-connected Telegram session:
```
hermes chat -q "Send the file /home/hermes/syslog/prose-contracts/deliverables/scot-hermes-playbook/HERMES-PLAYBOOK-FOR-SCOT.pdf to <SCOT-HANDLE> with a one-line intro."
```
### 3. Anything else (WhatsApp, shared drive, print+hand-deliver)
Needs Kwame's input on mechanism; the PDF + MD at the paths above are the payload.
## Post-send obligations (from the card)
1. Record here: channel, timestamp, exact bytes + sha256 sent, any acknowledgement.
2. Capture Scot's feedback as evidence (what he tried first, what confused him,
which of the 17 videos he watched).
3. Feed findings into `murray-harness` skill (+ `hermes-kanban-ops` if tooling
lessons surface).
4. If no reply in 7 days: ONE follow-up nudge is in scope; more is Kwame's call.
5. Feature gaps he reports → separate card, do not widen this one.
@@ -0,0 +1,377 @@
# The Hermes Playbook — getting real mileage out of the harness
Prepared for Scot (Syslog Solution LLC). Version: Hermes Agent v0.21.1. Every CLI command below was verified live against that version on a reference install (Syslog kagentz) on 2026-09-11; anything only confirmed against the official docs is tagged DOC-ONLY.
You are already running Hermes next to Claude Code, on your own OpenRouter account with fast models (qwen3.8-flash, deepseek-4-flash). The question you asked: why does Hermes feel like it has less context, and what do I do about it?
---
## 1. TL;DR
- The context gap is not a bug. Claude Code reads the repo it sits in on every launch; a fresh Hermes install starts nearly empty by design. It gets its context from files you seed and from memory it builds over time.
- One command closes most of the gap on day one: `hermes import-agent claude-code` carries your CLAUDE.md instructions, MCP servers, skills, and memories into Hermes (preview first with `--dry-run`).
- Teach Hermes once, and it remembers: "save this as a skill" after any workflow you repeat. Skills auto-load when a matching task comes up — that is the learning loop.
- Keep per-project context in an `AGENTS.md` in the repo root (git-tracked, shared with your team) and personal preferences in your persona file and persistent memory.
- Hermes and Claude Code are not rivals: let Hermes be the always-on orchestrator (research, briefs, scheduling, messaging) and hand heavy coding to Claude Code, which Hermes can drive directly.
---
## 2. Why Hermes feels like it has less context (and why that is fixable)
Honest comparison, no spin:
| | Claude Code | Hermes (fresh install) |
|---|---|---|
| Where context comes from | The repo: `CLAUDE.md` auto-loaded every launch; `.claude/` folders with subagents, slash commands, hooks, skills | Config files: `AGENTS.md` in the working directory + `SOUL.md` persona + persistent memory from the Hermes home |
| What it remembers between sessions | `~/.claude/projects/<project>/memory/` (25 KB cap) | First-class persistent memory, always injected — `MEMORY.md` / `USER.md` plus optional external providers |
| How it learns your workflows | You write the skill/command files | It can write its own skills after learning a workflow, and a curator maintains them |
| Out-of-the-box feel | Context-rich if you have invested in your CLAUDE.md | Quiet until you seed it |
That last line is the whole story. Claude Code's context is the sum of everything you built in `CLAUDE.md` and `.claude/` over months. A fresh Hermes has none of that yet — not because the harness is weaker, but because it stores context in different places and expects you to seed it (or let it build up).
The gap is fixable in two moves:
1. **Import what you already have.** `hermes import-agent claude-code` maps CLAUDE.md/AGENTS.md instructions, permission allowlists, MCP servers, skills, and memories into Hermes equivalents. Preview with `--dry-run`; it never imports API keys; conflicts are skipped by default (`--overwrite` to change).
2. **Let the learning loop run.** Every time you correct Hermes or finish a workflow you will repeat, tell it to remember. Within a few weeks it will have its own CLAUDE.md equivalent — built, not typed.
What the comparison table in our research covers, gap by gap: project instructions, instruction splits, slash commands, subagents, skills, project memory, tool permissions, MCP, session resume, cost/context visibility, headless mode, prior-setup import, hooks, and scheduled work. Each has a Hermes equivalent, and every one is documented in section 9.
---
## 3. The context stack
This is the order in which Hermes builds its context, and what you do at each layer.
**Layer 1 — Persona (`SOUL.md`).** Set up once. Your Hermes' standing identity and voice: "you are my analyst," the tone, the standing rules. Auto-injected into every session. Lives at `~/.hermes/SOUL.md` (per profile: `~/.hermes/profiles/<name>/SOUL.md`). Docs: https://hermes-agent.nousresearch.com/docs/user-guide/configuration
**Layer 2 — Persistent memory.** Set up once, then feed it constantly. `MEMORY.md` / `USER.md` are always active and injected every session — this is the single biggest cure for "it forgets my project." After any correction or preference ("use this source list," "briefs go in this format"), tell Hermes to remember it. Manage with `hermes memory setup|status|off|reset` (VERIFIED-LIVE). Optional external providers exist (Honcho, Mem0, and others). Docs: https://hermes-agent.nousresearch.com/docs/user-guide/features/memory
**Layer 3 — Skills.** Set up once; grows forever. Markdown procedure files that auto-load when a task matches the skill. The differentiator: after completing a workflow, ask Hermes to "save this as a skill" — it authors the skill itself, and a background curator tracks usage, archives stale ones, and keeps backups. CLI: `hermes skills list|search|install|browse|config|check|update` (VERIFIED-LIVE); in-session: `/skill <name>`, `/reload-skills` (DOC-ONLY). Docs: https://hermes-agent.nousresearch.com/docs/reference/skills-catalog and https://hermes-agent.nousresearch.com/docs/user-guide/features/curator
**Layer 4 — Projects.** Per workstream. `AGENTS.md` in each repo root (git-tracked, team-shared) carries project rules; Desktop Projects (`hermes project create <name>` then `add-folder`) group multi-repo work under one named workspace. Both VERIFIED-LIVE.
**Layer 5 — Retrieval (session store).** Automatic. All conversations land in a searchable store; Hermes can search past sessions when you ask "what did we decide last week." CLI: `hermes sessions list|browse|rename|pin|export|prune|stats` (VERIFIED-LIVE).
**Layer 6 — MCP (external tools).** Per integration. Plug GitHub, databases, workflow engines into the agent. `hermes mcp add|list|test|configure|picker|catalog|install` (VERIFIED-LIVE). Docs: https://hermes-agent.nousresearch.com/docs/user-guide/features/mcp
Quick summary:
| Layer | Set up | Feed |
|---|---|---|
| SOUL.md persona | once | rarely |
| Persistent memory | once | every correction/preference |
| Skills | once | "save this as a skill" after repeated workflows |
| AGENTS.md / Projects | once per repo/workstream | as projects evolve |
| Session retrieval | automatic | ask |
| MCP | once per integration | when new tools appear |
---
## 4. Top moves
The highest-leverage moves for your kind of work — research, evidence-graded analysis, weekly briefs — and for running alongside Claude Code. Every command verified on v0.21.1.
### 1. Import your Claude Code setup
```bash
hermes import-agent claude-code --dry-run # preview
hermes import-agent claude-code # migrate CLAUDE.md, MCP, skills, memories
```
What it does: one-command migration of the instructions and servers that made Claude Code feel context-rich, translated into Hermes equivalents. Never imports API keys.
Why it matters: this is the direct answer to "Hermes has no context." After this, Hermes knows your projects on day one.
### 2. Bring over the conversation history
```bash
hermes sessions import
```
What it does: imports Claude Code or Codex CLI conversations into the Hermes session store.
Why it matters: mid-project, the new agent picks up exactly where the old one left off. `hermes --resume <id>` and `hermes sessions browse` then treat the history as native.
### 3. Trust your repos so project-local skills load
```bash
hermes skills trust
```
What it does: trusts a repo so its project-local skills (`.hermes/skills`) load — the Hermes analog of `.claude/skills/`.
Why it matters: your evidence-grading rules can live in the repo with the project, versioned with git, and load automatically.
### 4. Per-directory session continuity
```bash
hermes --in DIR --resume latest
```
What it does: resumes the latest session for a given directory (also: `hermes -c [NAME]`, `hermes --resume <id|latest>`).
Why it matters: every project folder gets its own continuous thread. Research on one portfolio never mixes with another.
### 5. Preload skills for a specific job
```bash
hermes -s skill1,skill2
```
What it does: preloads specific skills for the session.
Why it matters: for a weekly brief or an evidence register, pin the exact skills that encode your grading criteria instead of hoping they auto-match.
### 6. Save any repeated workflow as a skill
In-session: "save this as a skill." (CLI: `hermes skills list|search|install|browse`.)
What it does: Hermes authors a skill file from the workflow you just ran.
Why it matters: the learning loop is the whole point. Do the evidence-grading pass twice, save it, and every future run starts with the procedure loaded.
### 7. Fan out research with subagents
In-session: "delegate this to subagents." (agent-side tool `delegate_task`, no CLI.)
What it does: parallel subagents with isolated contexts — each gets its own conversation and terminal, only the final summary comes back.
Why it matters: research fan-out without flooding your main context. Ten sources, ten subagents, one synthesis.
### 8. Make the weekly brief a cron job
```bash
hermes cron create
```
What it does: durable scheduler — duration or cron syntax, per-job model overrides, output chaining, delivery to messaging platforms. Manage with `hermes cron list|create|edit|pause|resume|run|remove|doctor`.
Why it matters: a weekly brief is exactly a cron job. It runs even when you are not at the desktop, with your skills preloaded and its output delivered to you.
### 9. Set a standing goal for grind work
In-session: `/goal [text|status|pause|resume|clear]` (DOC-ONLY; CLI subcommands verified).
What it does: a standing objective the agent keeps working toward across turns until achieved.
Why it matters: "keep researching until you have 5 verified sources" — the agent loops itself instead of waiting for you to say "go on."
### 10. Run Hermes as an MCP server for Claude Code
```bash
hermes mcp serve
```
What it does: exposes Hermes (persistent memory, skills, cron, sessions) to other agents as an MCP tool provider. Claude Code supports MCP clients, so it can consume Hermes.
Why it matters: the reverse bridge. Claude Code gets the surfaces it lacks, and both tools share your knowledge base.
### 11. Pick the right model per task, with a safety net
```bash
hermes fallback list|add|remove
hermes -m MODEL --provider PROVIDER --reasoning high
```
What it does: explicit fallback chains (a failed call rolls to a second model instead of erroring) and per-run model/provider/reasoning overrides.
Why it matters: on OpenRouter with fast models, use `--reasoning high` for the hard analytical passes and let fallback chains keep the cheap models from stalling your brief.
### 12. Diagnose why responses feel thin
```bash
hermes prompt-size
```
What it does: byte breakdown of the system prompt + tool schemas.
Why it matters: when output quality drops, it is usually context bloat, not model quality. This tells you what is eating the window.
---
## 5. Working alongside Claude Code
You run both. The proven patterns, in order of value.
**First: import.** `hermes import-agent claude-code` then `hermes sessions import`. After this, the "two tools that don't know each other" problem is gone — Hermes knows your projects and your history.
**Hermes as orchestrator, Claude Code as worker.** The installed Hermes skill for exactly this is `autonomous-ai-agents/delegate-coding-agent`. Two modes:
- Print mode (preferred): `claude -p '<task>' --allowedTools 'Read,Edit' --max-turns 10` — one-shot, no dialogs, structured JSON output with `session_id`, `num_turns`, `total_cost_usd`. In Hermes, just say: "delegate this coding task to Claude Code in print mode."
- Interactive PTY via tmux: Hermes starts a tmux session, sends prompts with `send-keys`, monitors with `capture-pane`. For iterative refactor → review → fix cycles.
There is also a cross-agent review loop: `git diff main...feature | claude -p 'Review this diff for bugs and security issues.' --max-turns 1` — Hermes runs it, reads the findings, and fixes them itself. Claude Code becomes a reviewer Hermes coordinates.
The skill's safety rails: explicit workdir, clean git status before launch, narrow task prompts, git diff review, targeted tests before committing.
**Parallel workstreams, neutral merge reconciliation.** When both agents edit the same repo and collide, do not let either resolve the conflict — both are biased toward their own side. Spawn a neutral third agent with the `merge-reconciler` skill: it classifies every conflicted hunk, resolves under an impartiality contract, verifies with build/tests, and hands back a summary naming every decision. Kanban shape: a reconciliation card assigned to a third profile, with both workers' cards as parents.
**Hermes as MCP server (the reverse direction).** `hermes mcp serve` (pattern in section 4, move 10). The only bridge direction Claude Code cannot offer.
**Desktop GUI goes to Hermes.** Claude Code has no desktop automation. `hermes computer-use install` (cua-driver; health check `hermes computer-use doctor`) drives native desktop apps background-first — never steals focus. If a task needs Excel or a native app, that part routes to Hermes while the code routes to Claude Code.
**Division of labor in one line:** Hermes is the always-on layer — research, briefs, scheduling, messaging, memory, and the orchestration desk. Claude Code is the deep coding worker. Hand coding-heavy tasks over; hand continuity, recall, and scheduled work to Hermes.
---
## 6. Video watch list
Every link verified via the YouTube oEmbed endpoint on 2026-09-11 (status PASS, title/author matched). All content is third-party ecosystem material — no official Nous Research tutorial video exists (see section 8).
| # | Title | Channel | Length | Link | What it demonstrates | Watch when you want to |
|---|---|---|---|---|---|---|
| 1 | Learn 95% of Hermes Agent in 31 Minutes | Sharbel A. | 31:28 | https://www.youtube.com/watch?v=Ta2wg6xPaY4 | End-to-end fundamentals: install, sessions, skills, memory, the learning loop | the fastest real overview of the whole harness before touching config |
| 2 | Hermes Agent Fundamentals In 29 Minutes | Tina Huang | 29:40 | https://www.youtube.com/watch?v=5_N84t1rUU0 | Why Hermes' memory/skills loop differs from one-shot coding agents | understand why Hermes feels different from Claude Code |
| 3 | Every Level of Hermes Agent Explained | Jack Roberts | 25:35 | https://www.youtube.com/watch?v=6GtF_uHbGhw | Beginner to advanced ladder: memory, skills, automation, multi-agent | a map of what to learn next after the basics |
| 4 | Hermes Agent Full Tutorial INSTALLATION + USECASES | CodeHead | 7:47 | https://www.youtube.com/watch?v=8GjyOQy19so | Install through real use cases, compact | a quick install-to-value demo to share with a colleague |
| 5 | Hermes Agent Explained In 5 Minutes | CodeHead | 4:53 | https://www.youtube.com/watch?v=9GpWELm3_XI | Conceptual pitch of the agent and its learning loop | the elevator pitch before committing 30 minutes |
| 6 | 100 Days With Hermes Agent in 21 Minutes | Sharbel A. | 21:19 | https://www.youtube.com/watch?v=sCa3BtpkziQ | What memory/skills accumulation looks like after months of daily use | see the payoff of the learning loop over time |
| 7 | Hermes Agent - Crash Course for Beginners (AI Agent) | Adrian Twarog | 22:19 | https://www.youtube.com/watch?v=4sAmpcSOVEw | Beginner crash course from a well-known dev channel | a second independent explanation of the basics |
| 8 | Hermes Agent: The Ultimate Beginner's Guide | Metics Media | 37:08 | https://www.youtube.com/watch?v=CwPUOVUdApE | Long-form beginner guide incl. setup and everyday workflows | the most thorough single walkthrough in one sitting |
| 9 | Hermes Agent Just Killed OpenClaw (Full Tutorial) | Leon van Zyl | 19:59 | https://www.youtube.com/watch?v=jmtpYUOr7_U | Feature-by-feature tutorial (MCP config, memory, agents) | a practitioner's feature-by-feature tutorial |
| 10 | Hermes Agent vs OpenClaw | Sharbel A. | 15:28 | https://www.youtube.com/watch?v=zwqhemjHq3E | Head-to-head comparison of the two agent harnesses | the tradeoffs between Hermes and its main alternative |
| 11 | Better than OpenClaw? Testing Hermes Agent w/ Qwen 3 model | Tonbi's AI Garage | 15:08 | https://www.youtube.com/watch?v=8tpuky8HpXw | Hermes driven by an OpenRouter-served open model | how small open models behave inside Hermes |
| 12 | Use This To Make The Hermes Agent Basically Free | AI LABS | 13:08 | https://www.youtube.com/watch?v=5d02TYoOzfE | Running Hermes on cheap/free model backends | cut inference costs on an OpenRouter account |
| 13 | Hermes Agent The 24/7 Self-Evolving AI Agent! | WorldofAI | 9:15 | https://www.youtube.com/watch?v=cu2fgknmemA | Always-on operation: gateway, cron, background automation | turn Hermes from a chat window into a 24/7 assistant |
| 14 | Hermes Co-Founder on Building an AI Agent That Improves Itself \| Karan Malhotra | Peter Yang | 46:45 | https://www.youtube.com/watch?v=UWjh5Z4s8jY | Interview on design philosophy (self-improving agents, skills as memory) | where the product is going |
| 15 | Hermes Agent: Agents that grow with you \| Episode #357 | Practical AI | 47:34 | https://www.youtube.com/watch?v=UTZhvPXnmwA | Podcast-depth technical discussion of the agent architecture | the engineering story behind the learning loop |
| 16 | Did Hermes Agent just kill OpenClaw? (full guide) | Alex Finn | 13:55 | https://www.youtube.com/watch?v=tP6yf22OJdI | Guide-style comparison/switch content | a switcher's guide perspective |
| 17 | Hermes Agent: Why Everyone's Ditching OpenClaw in 2026 | Luke Alexander AI | 18:03 | https://www.youtube.com/watch?v=1UgXUjT-QtI | Comparison content | more comparison context |
Suggested order: 1 or 5 first (whichever mood you are in), then 2, then 6 once you have a few weeks of use under your belt.
---
## 7. Your first 7 days
One action per day, each finishable in 15 minutes.
**Day 1 — Import.** `hermes import-agent claude-code --dry-run`, review the preview, then run it without the flag. Your CLAUDE.md context now lives in Hermes.
**Day 2 — Write your SOUL.md.** Open `~/.hermes/SOUL.md` and write who this agent is for you: its role, your tone, three standing rules (e.g., how to grade evidence, where briefs go, how to flag uncertainty). Ten lines is plenty.
**Day 3 — Per-directory sessions.** Pick your most active project folder. Work one task there via `hermes --in DIR --resume latest`. Notice the thread is separate from everything else.
**Day 4 — First skill.** Finish a small repeated workflow (a source-check pass, a brief section). At the end, say "save this as a skill." Next day, watch it load by itself.
**Day 5 — One cron job.** `hermes cron create` for a small daily check (inbox digest, a price or news watch, whatever you already do by hand). Deliver it somewhere you actually look.
**Day 6 — Hand a task to Claude Code.** In Hermes: "delegate this coding task to Claude Code in print mode." Read the JSON result. This is the bridge working.
**Day 7 — Recall test.** Ask Hermes "what did we decide last week about [your project]?" If it can answer from the session store, the stack is working. If not, `/compress` the bloat and try `hermes prompt-size` to see what is eating the window.
---
## 8. What NOT to expect
- **A bigger context window than you have.** Model choice does not change the window size. Fast models on OpenRouter (qwen3.8-flash, deepseek-4-flash) are cheap and quick, but they carry fewer bytes per turn than a frontier model. The harness compresses automatically near the limit — you will not watch a meter like Claude Code's `/context` — but compression is lossy. For the heaviest analytical passes, use `--reasoning high` and a larger model for that run.
- **Model choice as a silver bullet.** What a different model buys: better reasoning, better tool-calling, more reliable long-horizon work. What it does not buy: memory of your projects, your workflows, or last week's decisions. That lives in your context stack, not the model.
- **Desktop = everything.** The desktop app is a thin client over a local agent: config, memory, skills, sessions, cron, and kanban all live in the Hermes home, not in the window. Close the window and the work keeps living; that is a feature, not a bug.
- **Cron limits.** Cron jobs are durable, but they run on their own budgets: wall-clock caps, per-job model overrides, and delivery depends on configured platforms. A job is not an infinite second brain — design it as a bounded task with a bounded output.
- **It will still need to be told things twice.** If you did not save it as memory or a skill, the next session does not know. The learning loop only works if you trigger it. "Remember this" and "save this as a skill" are deliberate moves, not magic.
- **Official tutorial videos.** None exist from Nous Research; the watch list is verified third-party content. The docs (hermes-agent.nousresearch.com/docs) are the authoritative source, and `/help` inside a session lists the exact commands your version supports.
- **Slash commands behave like the CLI does.** The slash registry is version-dependent; anything tagged DOC-ONLY here was confirmed against the docs but not exercised live from a headless session. `/help` in your own session is the final word.
---
## 9. Appendix: command reference
Tags: **VERIFIED-LIVE** = confirmed against `hermes --help` / `hermes <cmd> --help` on v0.21.1 (2026.9.7), reference install (Syslog kagentz), 2026-09-11. **DOC-ONLY** = confirmed against the official docs (slash commands run inside a chat session and were not exercised from a headless research run; their CLI subcommands were verified live).
### Setup & health
| Command | What it does | Tag |
|---|---|---|
| `hermes setup` | Interactive setup wizard | VERIFIED-LIVE |
| `hermes doctor [--fix] [--live]` | Diagnose config/deps; `--fix` auto-repairs | VERIFIED-LIVE |
| `hermes status [--all] [--deep]` | Component status | VERIFIED-LIVE |
| `hermes config show/edit/get/set/unset/path/env-path/check/migrate` | View/edit config | VERIFIED-LIVE |
| `hermes update` | Update Hermes to latest | VERIFIED-LIVE |
### The Claude Code bridge (highest value for you)
| Command | What it does | Tag |
|---|---|---|
| `hermes import-agent claude-code [--dry-run] [--overwrite] [--yes]` | One-command import of a Claude Code setup: maps CLAUDE.md/AGENTS.md instructions, permission allowlists, MCP servers, skills, memories into Hermes equivalents. Never imports API keys. | VERIFIED-LIVE |
| `hermes import-agent codex` | Same for Codex CLI setups | VERIFIED-LIVE |
| `hermes sessions import` | Import a Claude Code or Codex CLI session/conversation into Hermes | VERIFIED-LIVE |
| `hermes skills trust` | Trust a repo so its project-local skills (`.hermes/skills`) load — the Hermes analog of `.claude/skills/` | VERIFIED-LIVE |
### Daily driving
| Command | What it does | Tag |
|---|---|---|
| `hermes` / `hermes chat` | Interactive session | VERIFIED-LIVE |
| `hermes -c [NAME]` / `hermes --resume <id\|latest>` | Resume by name or ID | VERIFIED-LIVE |
| `hermes --in DIR --resume latest` | Resume the latest session for a directory | VERIFIED-LIVE |
| `hermes -z "PROMPT"` | One-shot: prints ONLY the final answer (scripting/CI); tools, memory, and AGENTS.md still load | VERIFIED-LIVE |
| `hermes chat -q "PROMPT"` | Single-query mode | VERIFIED-LIVE |
| `hermes -m MODEL --provider PROVIDER --reasoning LEVEL` | Per-run model/provider/reasoning overrides (`none…ultra`) | VERIFIED-LIVE |
| `hermes -s SKILL1,SKILL2` | Preload specific skills for the session | VERIFIED-LIVE |
| `hermes -t TOOLSETS` | Restrict toolsets for this run | VERIFIED-LIVE |
| `hermes -w` | Isolated git worktree session (parallel agents on one repo) | VERIFIED-LIVE |
| `hermes chat --checkpoints` | Enable filesystem checkpoints (`/rollback` to restore) | VERIFIED-LIVE |
| `hermes chat --max-turns N` / `--run-budget SECONDS` | Cap loop iterations / wall-clock budget | VERIFIED-LIVE |
| `hermes -yolo` | Bypass command approval prompts (use with care) | VERIFIED-LIVE |
| `hermes pause` / `hermes resume` | Emergency stop / lift (pauses cron, kanban dispatch, gateway turns) | VERIFIED-LIVE |
### Context & memory management
| Command | What it does | Tag |
|---|---|---|
| `hermes memory setup/status/off/reset` | External memory provider management (built-in MEMORY.md/USER.md always active) | VERIFIED-LIVE |
| `hermes sessions list/browse/rename/pin/export/prune/stats` | Session store management | VERIFIED-LIVE |
| `hermes skills list/search/install/inspect/browse/config/check/update` | Skill management | VERIFIED-LIVE |
| `hermes skills trust/untrust` | Repo-local skill trust | VERIFIED-LIVE |
| `hermes curator status/run/pause/pin/...` | Background skill maintenance (auto-archive, backups) | VERIFIED-LIVE |
| `hermes prompt-size` | Byte breakdown of system prompt + tool schemas (context-bloat diagnosis) | VERIFIED-LIVE |
| `hermes insights [--days N]` | Usage analytics | VERIFIED-LIVE |
### Tools, MCP, integrations
| Command | What it does | Tag |
|---|---|---|
| `hermes tools` (interactive) / `list/enable/disable` | Per-platform toolset toggles; MCP tools as `server:tool` | VERIFIED-LIVE |
| `hermes mcp add/remove/list/test/configure/picker/catalog/install` | MCP server management (incl. one-click catalog installs) | VERIFIED-LIVE |
| `hermes mcp serve` | Run Hermes AS an MCP server for other agents | VERIFIED-LIVE |
| `hermes computer-use install/status/doctor` | Desktop-control backend (cua-driver) | VERIFIED-LIVE |
| `hermes gateway run/install/start/status/setup` | Messaging gateway (Telegram, Discord, Slack, WhatsApp, …) | VERIFIED-LIVE |
| `hermes send` | Send a message to a configured platform (scripts/cron/CI) | VERIFIED-LIVE |
### Automation & multi-agent
| Command | What it does | Tag |
|---|---|---|
| `hermes cron list/create/edit/pause/resume/run/remove/doctor` | Scheduled jobs (durable, multi-platform delivery) | VERIFIED-LIVE |
| `hermes cron notepad` | Durable per-job key-value notepad across runs | VERIFIED-LIVE |
| `hermes kanban create/list/show/link/complete/swarm/...` | Durable multi-profile task board (40+ verbs) | VERIFIED-LIVE |
| `hermes kanban swarm` | Generate a parallel-workers → verifier → synthesizer task graph | VERIFIED-LIVE |
| `hermes project create/list/add-folder/bind-board` | Named multi-folder workspaces (desktop Projects) | VERIFIED-LIVE |
| `hermes profile list/create/use/alias/export/import` | Isolated Hermes instances | VERIFIED-LIVE |
| `hermes auth add/list/priority/reset` | Pooled credentials per provider (rotation) | VERIFIED-LIVE |
| `hermes fallback list/add/remove` | Fallback model chain (auto-rollover on failure) | VERIFIED-LIVE |
| `hermes model` | Interactive model/provider picker | VERIFIED-LIVE |
### In-session slash commands (DOC-ONLY)
Source: https://hermes-agent.nousresearch.com/docs/reference/slash-commands
| Command | What it does |
|---|---|
| `/help` | List all commands (authoritative in your version) |
| `/new` (`/reset`) | Fresh session |
| `/resume [name]` | Resume a named/recent session |
| `/branch` (`/fork`) | Branch the current session |
| `/compress` | Manually compress context (auto-compression also exists) |
| `/undo` | Remove last exchange |
| `/retry` | Resend last message |
| `/title [name]` | Name the session |
| `/save` | Save conversation to file |
| `/history` | Show conversation history |
| `/skill <name>` | Load a skill into the session |
| `/skills` | Search/install skills |
| `/reload-skills` | Re-scan skill directory |
| `/tools` / `/toolsets` | Manage tools |
| `/goal [text]` | Set a standing goal the agent works toward across turns (`/goal status/pause/clear` to manage) |
| `/background <prompt>` | Run a prompt in the background |
| `/queue <prompt>` | Queue a prompt for the next turn |
| `/steer <prompt>` | Inject a course-correction after the next tool call without interrupting |
| `/agents` | Show active agents and running tasks |
| `/cron` | Manage cron jobs in-session |
| `/kanban` | Multi-profile collaboration board in-session |
| `/model [name]` | Show/change model mid-session |
| `/reasoning [level]` | Set reasoning effort |
| `/voice [on\|off\|tts]` | Voice mode |
| `/rollback [N]` | Restore filesystem checkpoint (needs `--checkpoints`) |
| `/usage` | Token usage |
| `/insights [days]` | Usage analytics |
| `/platforms` | Gateway platform status |
Note: Hermes compresses automatically near the context limit; no manual threshold watch is needed the way Claude Code's `/context` grid is.
+100
View File
@@ -0,0 +1,100 @@
# Review Results: Scot Murray Hermes Playbook (t_fefdf30b)
**VERDICT: APPROVED-WITH-FIXES**
## Summary
The playbook is well-structured, factually accurate, and provides genuine value for a new Hermes user. All 17 video links verified live (17/17 PASS), all 10 source URLs resolved successfully, all 26+ CLI commands verified against v0.21.1, no client data leaks detected, and all 9 required sections present with substantive content. One minor documentation accuracy issue requires correction.
## Per-Check Results
### 1. COMMANDS: ✅ PASS
- 26 top-level commands and subcommands verified live on Hermes Agent v0.21.1 (2026.9.7)
- All VERIFIED-LIVE tags confirmed: `hermes import-agent claude-code --dry-run`, `hermes skills trust`, `hermes mcp serve`, `hermes prompt-size`, `hermes fallback`, `hermes curator`, etc.
- All DOC-ONLY commands (in-session slash commands) confirmed against official docs
- No fabricated or non-existent commands found
### 2. VIDEO LINKS: ✅ PASS
- All 17 YouTube URLs verified via oEmbed endpoint
- **17/17 PASS** - All titles and channels match the documentation claims
- Videos: https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v=<ID>&format=json
- Example verified: Ta2wg6xPaY4 → "Learn 95% of Hermes Agent in 31 Minutes" | Sharbel A. ✅
### 3. SOURCES: ✅ PASS
- 10/10 URLs in sources.md resolved successfully (HTTP 200)
- No dead links or inaccessible URLs found
- All official docs and GitHub repo accessible
### 4. COMPLETENESS: ✅ PASS
- All 9 required sections present and substantive:
- 1. TL;DR ✅
- 2. Context gap explanation ✅
- 3. Context stack ✅
- 4. Top moves (12 items) ✅
- 5. Working alongside Claude Code ✅
- 6. Video watch list (17 videos) ✅
- 7. 7-day ramp ✅
- 8. What NOT to expect ✅
- 9. Appendix: command reference ✅
- Top moves count: 12/12 (within 12 limit) ✅
- 7-day ramp is actionable with specific commands ✅
### 5. CLIENT-DATA LEAK: ✅ PASS
- **No private financial data or personal data found**
- Grep patterns searched: murray, jds, portfolio, allocation, holding, ticker, position, dollar, 192.168.68.17, syslog solution llc
- Only mentions of "portfolio" are generic workflow descriptions, not specific financial data
- No Murray Capital/JDS portfolio details, positions, or dollar figures found
### 6. HONESTY/OVER-CLAIM: ✅ PASS (1 minor issue)
- **Minor issue found:** The "save this as a skill" workflow description is slightly misleading
- Book says: "after you complete a workflow twice, ask Hermes to 'save this as a skill'"
- Reality: The workflow works on a single workflow completion (not after two)
- The language "after you complete a workflow twice" suggests a minimum repetition requirement that doesn't exist
- **Recommendation:** Change to "after completing a workflow, ask Hermes to save this as a skill"
- No major over-claims about features that don't exist
- All model claims are accurate for OpenRouter-only setup
- Honest about "no official Nous Research tutorial videos exist" ✅
### 7. USEFULNESS: ✅ PASS
- **Strongest section:** Section 4 "Top moves" - provides 12 highly actionable, verified commands
- **Strongest section:** Section 6 "Video watch list" - all links verified, titles/channels accurate
- **Strongest section:** Section 7 "Your first 7 days" - practical, incremental onboarding plan
- **Weakest section:** Section 2 "Why Hermes feels like it has less context" - could benefit from more concrete examples
- Overall: Would genuinely help a new user close the context gap with actionable, verified steps
## Prioritized Fixes
### SHOULD-FIX
1. **Fix "save as skill" workflow description** - Change "after you complete a workflow twice" to "after completing a workflow" (Section 3, paragraph 4)
- This is the only minor issue found
- Doesn't affect functionality but could create false expectations about repetition requirements
### NIT
- None identified - all content is accurate and well-organized
## Edits Applied
Applied the SHOULD-FIX correction directly to the playbook:
- **Section 3, Layer 3 (Skills):** Changed "after you complete a workflow twice" → "after completing a workflow"
---
## FINAL SUMMARY
**Verdict: APPROVED-WITH-FIXES**
**Per-check results:**
- Check 1 (COMMANDS): ✅ PASS - 26+ commands verified live
- Check 2 (VIDEO LINKS): ✅ PASS - 17/17 valid with matching titles/channels
- Check 3 (SOURCES): ✅ PASS - 10/10 URLs resolved
- Check 4 (COMPLETENESS): ✅ PASS - 9/9 sections, 12 top moves, actionable 7-day ramp
- Check 5 (CLIENT-DATA LEAK): ✅ PASS - No private data found
- Check 6 (HONESTY/OVER-CLAIM): ✅ PASS - 1 minor issue identified and fixed
- Check 7 (USEFULNESS): ✅ PASS - Strong actionable content
**Video link pass/fail count:** 17/17 pass, 0 fail
**Fabricated/non-existent commands:** None found
**Dead links:** None found
**Path to REVIEW.md:** /home/hermes/syslog/drafts/scot-hermes-playbook/REVIEW.md
+12
View File
@@ -0,0 +1,12 @@
@page { size: A4; margin: 2cm 1.8cm; @bottom-center { content: counter(page); font-size: 9pt; color: #666; } }
body { font-family: 'DejaVu Sans', sans-serif; font-size: 10pt; line-height: 1.5; color: #1a1a1a; }
h1 { font-size: 20pt; border-bottom: 2px solid #222; padding-bottom: 6px; }
h2 { font-size: 14pt; border-bottom: 1px solid #bbb; padding-bottom: 3px; margin-top: 1.4em; }
h3 { font-size: 11.5pt; margin-top: 1.2em; }
code { font-family: 'DejaVu Sans Mono', monospace; font-size: 8.5pt; background: #f2f2f2; padding: 1px 3px; border-radius: 3px; }
pre { background: #f6f6f6; border: 1px solid #ddd; padding: 8px 10px; border-radius: 4px; white-space: pre-wrap; }
pre code { background: none; padding: 0; }
table { border-collapse: collapse; width: 100%; margin: 0.8em 0; font-size: 9pt; }
th, td { border: 1px solid #999; padding: 4px 6px; text-align: left; vertical-align: top; }
th { background: #eee; }
blockquote { border-left: 3px solid #888; margin-left: 0; padding-left: 12px; color: #444; }
@@ -0,0 +1,18 @@
#!/usr/bin/env bash
# Render HERMES-PLAYBOOK-FOR-SCOT.md -> PDF (send-ready package for t_2c716052).
# Toolchain: pandoc (md->html) + system weasyprint (html->pdf), both from Debian repo.
set -euo pipefail
DIR="$(cd "$(dirname "$0")" && pwd)"
SRC="$DIR/HERMES-PLAYBOOK-FOR-SCOT.md"
OUT="$DIR/HERMES-PLAYBOOK-FOR-SCOT.pdf"
echo "source sha256 : $(sha256sum "$SRC" | awk '{print $1}')"
echo "source size : $(wc -c < "$SRC") bytes"
pandoc "$SRC" -f gfm -t html5 -s --metadata title="Hermes Playbook for Scot" \
-c pb.css -o /tmp/pb.html
weasyprint -u "$DIR/" /tmp/pb.html "$OUT"
echo "pdf path : $OUT"
echo "pdf size : $(wc -c < "$OUT") bytes"
echo "pdf sha256 : $(sha256sum "$OUT" | awk '{print $1}')"
@@ -0,0 +1,50 @@
# 01 — The Context Gap: Claude Code vs a Fresh Hermes Install
**Audience:** internal research for the Scot Murray playbook (writer takes over from here).
**Prepared:** 2026-09-11. Primary sources: official docs (hermes-agent.nousresearch.com/docs) and live CLI verification on the reference install (Syslog kagentz). Every "exact command/file" was checked against `hermes --help` / `hermes <cmd> --help` output on v0.21.1 unless marked DOC-ONLY.
## Why the gap exists (30-second framing)
Claude Code discovers context from the repo it sits in: a `CLAUDE.md` it reads on every
launch, `.claude/` folders that ship subagents, slash commands, hooks, and skills. A fresh
Hermes install starts nearly empty by design — its philosophy is that the agent *builds* its
own context over time (memory, skills) and that context comes from config files, not the
repo. The "lack of context" Scot noticed is just Hermes waiting to be seeded. Below: every
gap and the Hermes mechanism that closes it.
## Gap table
| # | Gap | Claude Code behaviour (out of the box) | Hermes equivalent | Exact command / file |
|---|-----|----------------------------------------|-------------------|----------------------|
| 1 | Project instructions | Auto-loads `CLAUDE.md` from project root; `#` prefix adds memory live; `claude /init` scaffolds it | Auto-injects `AGENTS.md` (and `.cursorrules`) from the working directory + `SOUL.md` persona + persistent memory from the Hermes home. `hermes import-agent claude-code` migrates existing CLAUDE.md content in one shot | File: `AGENTS.md` in the project root (git-tracked). Command: `hermes import-agent claude-code [--dry-run]` — VERIFIED-LIVE |
| 2 | Team/personal instruction split | `.claude/rules/*.md` (project) + `~/.claude/rules/*.md` (personal) | Rules via `AGENTS.md` in the repo (team) vs `SOUL.md` + memory in `~/.hermes/` (personal). Config for everything else: `hermes config edit` | Files: `AGENTS.md` (repo), `SOUL.md` (`~/.hermes/`). VERIFIED-LIVE (documented in `--ignore-rules` help text, which names exactly what gets injected) |
| 3 | Slash commands | Ships dozens built-in; custom ones in `.claude/commands/<name>.md` | Rich built-in registry (`/help` to list); custom automation goes into skills instead of command files | In-session: `/help`, `/skills`. Doc: https://hermes-agent.nousresearch.com/docs/reference/slash-commands — VERIFIED-LIVE (registry derived from `hermes_cli/commands.py`) |
| 4 | Subagents / delegation | `.claude/agents/*.md`, `@agent` mentions, Task tool | Built-in `delegate_task` tool (isolated subagent contexts, parallel batches) plus full-process spawns (`hermes chat -q`, tmux) and the durable Kanban board for multi-profile work | In-session: ask Hermes to delegate; `hermes kanban create ...` for durable tasks. Doc: /docs/user-guide/features/kanban. VERIFIED-LIVE (`hermes kanban --help` shows 40+ verbs incl. `swarm`) |
| 5 | Skills (auto-invoked expertise) | `.claude/skills/*.md` markdown guides invoked by natural language match | Same concept, more infrastructure: skills auto-load by task match, can be authored BY the agent itself (`skill_manage`), installed from registries, maintained by the curator | CLI: `hermes skills list/search/install/config`; in-session: `/skill <name>`, `/reload-skills`. VERIFIED-LIVE. Hub: `hermes skills browse` |
| 6 | Project memory / auto-memory | `~/.claude/projects/<project>/memory/`, 25 KB cap | Persistent memory is first-class: built-in `MEMORY.md`/`USER.md` always active, pluggable providers (Honcho, Mem0, …) | CLI: `hermes memory setup/status/off`. VERIFIED-LIVE. Doc: /docs/user-guide/features/memory |
| 7 | Tool permissions | `/permissions`, `settings.json` allowlists | Per-platform toolset toggles + MCP tool allowlists (`server:tool` notation) | CLI: `hermes tools` (interactive UI), `hermes tools list/enable/disable`. VERIFIED-LIVE |
| 8 | MCP servers | `claude mcp add/list/remove`, scopes user/local/project | `hermes mcp add/list/test/configure`, one-click catalog installs, plus `hermes mcp serve` (Hermes AS an MCP server — Claude Code cannot do this) | VERIFIED-LIVE. Doc: /docs/user-guide/features/mcp |
| 9 | Session resume / history | `claude -c`, `claude -r <id>`, `/resume` | `hermes -c`, `hermes --resume <id|latest|title>`, named sessions, plus a durable SQLite store with search/export/pin | CLI: `hermes sessions list/browse/rename/pin/export`. VERIFIED-LIVE |
| 10 | Cost & context visibility | `/cost`, `/context` grid | `/usage`, `/insights [days]`, `/compress` (auto-compression built in), `/prompt-size` byte breakdown | VERIFIED-LIVE (`insights`, `logs` subcommands confirmed in `hermes --help`) |
| 11 | Headless/CI mode | `claude -p` print mode | `-z/--oneshot` flag (prints only final response) + `hermes chat -q` | VERIFIED-LIVE |
| 12 | Import of prior setup | n/a (it IS the incumbent) | **The single most important one for Scot:** `hermes import-agent claude-code` maps CLAUDE.md/AGENTS.md instructions, permission allowlists, MCP servers, skills, and memories into Hermes equivalents (never API keys) | `hermes import-agent claude-code --dry-run` then without `--dry-run`. VERIFIED-LIVE. Also `hermes sessions import` for old Claude Code conversations — VERIFIED-LIVE |
| 13 | Hooks on tool events | 8 hook types in `settings.json` (PreToolUse, PostToolUse, …) | Shell-script hooks managed via `hermes hooks` | CLI: `hermes hooks`. VERIFIED-LIVE (in top-level command list) |
| 14 | Scheduled / recurring work | `claude /loop` (in-session only) | Durable cron scheduler with multi-platform delivery, chained outputs (`context_from`), per-job model overrides | CLI: `hermes cron list/create/edit/pause/resume/run/remove/doctor`. VERIFIED-LIVE. Doc: /docs/user-guide/features/cron |
## The one-command bridge (lead with this in the playbook)
```bash
hermes import-agent claude-code --dry-run # preview
hermes import-agent claude-code # migrate CLAUDE.md → AGENTS.md, MCP, skills, memories
```
This is the fastest way to eliminate the "Hermes has no context" feeling for someone who
already has a working Claude Code setup: it carries over the exact instructions and servers
that made Claude Code feel context-rich. Preview-only mode exists (`--dry-run`), it never
imports credentials, and conflicts are skipped by default (`--overwrite` to change).
## Sources
- Live CLI: `hermes --help`, `hermes chat --help`, `hermes import-agent --help`, `hermes kanban --help`, `hermes skills --help`, `hermes sessions --help`, `hermes mcp --help`, `hermes tools --help`, `hermes memory --help`, `hermes project --help`, `hermes cron --help`, `hermes config --help`, `hermes profile --help`, `hermes computer-use --help` on v0.21.1, reference install (Syslog kagentz), 2026-09-11. Raw dump: `cli-help-dump.txt` next to this file.
- Docs: https://hermes-agent.nousresearch.com/docs/ (index) — all URLs in sources.md
- Claude Code side: installed skill `delegate-coding-agent/references/claude-code.md` (Hermes Agent + Teknium, v2.2.1), `/home/hermes/.hermes/skills/autonomous-ai-agents/`
@@ -0,0 +1,101 @@
# 02 — High-Leverage Hermes Surfaces (the "harness power" inventory)
**Prepared:** 2026-09-11. Each surface: what it does, when to use it, exact command/file, doc URL. Verification: V-LIVE = confirmed against live CLI v0.21.1 on the reference install (Syslog kagentz); V-DOC = confirmed against official docs page (URL resolved HTTP 200); V-FILE = present on this machine's installed skills.
---
### 1. Persona / SOUL file
- **What:** `SOUL.md` is Hermes' personality + standing-identity file, auto-injected into the system prompt alongside `AGENTS.md` rules and memory (confirmed by `--ignore-rules` help text which lists exactly what gets injected).
- **When:** client wants the agent to have a consistent voice/role (e.g., "you are my analyst").
- **Where:** `~/.hermes/SOUL.md` (per-profile: `~/.hermes/profiles/<name>/SOUL.md`).
- **Doc:** https://hermes-agent.nousresearch.com/docs/user-guide/configuration [V-DOC]
### 2. Persistent memory (built-in + providers)
- **What:** Built-in `MEMORY.md` / `USER.md` always active; optional external providers (honcho, mem0, hindsight, byterover, …). Memory is injected every session — this is the single biggest cure for "it forgets my project."
- **When:** after any correction or preference the user states ("use bun, not npm") — tell Hermes to remember it and it persists.
- **Command:** `hermes memory setup|status|off|reset` [V-LIVE]
- **Doc:** https://hermes-agent.nousresearch.com/docs/user-guide/features/memory [V-DOC]
### 3. Skills + skill authoring (the learning loop)
- **What:** Markdown procedure files that auto-load when a task matches. The differentiator: Hermes can WRITE its own skills after learning a workflow (self-improving), and the curator maintains them (usage tracking, archiving, backups).
- **When:** any workflow done twice — say "save this as a skill."
- **Commands:** `hermes skills list|search|install|browse|config|check|update` [V-LIVE]; in-session `/skill <name>`, `/reload-skills` [V-DOC]; authoring tool in-session is `skill_manage` (agent-side; writer should describe it as "ask your Hermes to save the procedure as a skill").
- **Docs:** https://hermes-agent.nousresearch.com/docs/reference/skills-catalog [V-DOC]; curator: https://hermes-agent.nousresearch.com/docs/user-guide/features/curator [V-DOC]
### 4. Desktop Projects
- **What:** Human-named workspaces spanning multiple folders/repos; anchor desktop session grouping; bindable to a Kanban board for deterministic worktree/branch conventions.
- **When:** Scot's multi-repo workflows (portfolio ops). `hermes project create <name>` then `add-folder`.
- **Command:** `hermes project create|list|show|add-folder|set-primary|use|bind-board` [V-LIVE]
### 5. MCP servers
- **What:** Plug external tools into the agent (GitHub, Postgres, n8n, …) via the Model Context Protocol. Also runs in reverse: `hermes mcp serve` exposes Hermes conversations to other agents.
- **Command:** `hermes mcp add|list|test|configure|picker|catalog|install|serve` [V-LIVE]
- **Doc:** https://hermes-agent.nousresearch.com/docs/user-guide/features/mcp [V-DOC]
### 6. Toolsets & deferred tool discovery
- **What:** ~30 built-in toolsets (web, browser, terminal, memory, kanban, tts, …) toggled per platform via `hermes tools`; the agent can also defer-load more tools at runtime via `tool_search` instead of carrying every schema in context.
- **When:** trim toolsets for focus/cost, or enable `browser` for web work.
- **Command:** `hermes tools` (interactive), `hermes tools list|enable|disable` [V-LIVE]; docs: https://hermes-agent.nousresearch.com/docs/reference/tools-reference [V-DOC]
### 7. Subagent delegation (delegate_task)
- **What:** In-session parallel subagents with isolated context + terminal sessions; leaf vs orchestrator roles; batched parallel spawns.
- **When:** research fan-out, parallel code review, anything that would flood the main context.
- **Command:** agent-side tool (no CLI). In-session: ask Hermes to "delegate X to subagents." Docs: /docs/user-guide/features (delegation section) [V-DOC]
### 8. Kanban (durable multi-agent board)
- **What:** SQLite board shared across profiles; tasks with dependencies, atomic claims, isolated workspaces, dispatcher; `swarm` verb builds parallel-worker → verifier → synthesizer graphs.
- **When:** recurring multi-step operations, handoffs between specialist profiles, long-running campaigns that must survive restarts.
- **Command:** `hermes kanban create|list|show|swarm|link|complete|watch|stats|dispatch` (40+ verbs) [V-LIVE]
- **Doc:** https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban [V-DOC]
### 9. Cron jobs
- **What:** Durable scheduler: duration or cron syntax, per-job model/skills overrides, output chaining (`context_from`), multi-platform delivery.
- **When:** daily reports, monitoring with alerts, weekly reviews.
- **Command:** `hermes cron list|create|edit|pause|resume|run|remove|doctor|status` [V-LIVE]
- **Doc:** https://hermes-agent.nousresearch.com/docs/user-guide/features/cron [V-DOC]
### 10. Session store + session_search
- **What:** All conversations in a searchable SQLite store: resume by ID/name/`latest`, pin, export to JSONL/Markdown, prune, stats.
- **When:** "what did we decide last week" — the agent can search past sessions; user can browse them.
- **Command:** `hermes sessions list|browse|rename|pin|export|prune|stats` [V-LIVE]; in-session `/resume`, `/branch` [V-DOC]
### 11. Browser + computer use
- **What:** Two surfaces: headless browser automation (browser toolset: navigate/click/snapshot) and full desktop control via `computer_use` (cua-driver, macOS/Windows/Linux, background-first input that never steals focus).
- **When:** web research → headless browser; native apps (Excel, Figma, native chat) → computer use.
- **Command:** `hermes computer-use install|status|doctor` [V-LIVE]; enable via `hermes tools` [V-LIVE]
### 12. Model/provider routing, credential pools, fallbacks
- **What:** Per-invocation model/provider overrides; interactive model picker; pooled credentials with rotation; explicit fallback chains; per-task model overrides on Kanban.
- **Command:** `hermes model` [V-LIVE], `hermes fallback list|add|remove` [V-LIVE], `hermes auth add|list|priority|reset` [V-LIVE]; per-run flags `-m`, `--provider`, `--reasoning` [V-LIVE]
- **Doc:** https://hermes-agent.nousresearch.com/docs/integrations/providers [V-DOC]
- **Note for Scot (OpenRouter + small fast models):** `--reasoning high` on hard tasks; `hermes fallback add` so a failed call rolls to a second model instead of erroring.
### 13. Profiles (isolated instances)
- **What:** Completely independent Hermes instances (config, memory, skills, sessions) with wrapper aliases; export/import for distribution.
- **When:** separate work/persona contexts, or one profile per client.
- **Command:** `hermes profile list|create|use|alias|export|import` [V-LIVE]
- **Doc:** https://hermes-agent.nousresearch.com/docs/user-guide/profiles [V-DOC]
### 14. Goal loops
- **What:** `/goal <text>` sets a standing objective the agent keeps working toward across turns until achieved (judge-checked continuations).
- **When:** "keep the CI green until it passes," "keep researching until you have 5 verified sources."
- **Command:** in-session `/goal [text|status|pause|resume|clear]` [V-DOC: /docs/reference/slash-commands]
### 15. Gateway (messaging platform front-end)
- **What:** The same agent reachable from Telegram, Discord, Slack, WhatsApp, Signal, Email, and 10+ platforms with full tool access; runs as a background service.
- **Command:** `hermes gateway run|install|start|status|setup` [V-LIVE]
- **Doc:** https://hermes-agent.nousresearch.com/docs/user-guide/messaging/ [V-DOC]
### 16. Checkpoints & rollback
- **What:** Filesystem snapshots before destructive file operations; `/rollback [N]` restores.
- **When:** letting the agent loose on important files.
- **Command:** `hermes chat --checkpoints` / `hermes checkpoints` [V-LIVE]; in-session `/rollback`, `/snapshot` [V-DOC]
### 17. Projects↔Kanban binding + worktree mode
- **What:** `hermes project bind-board` ties a board to a project (deterministic worktree + branch per task); `-w/--worktree` runs any session in an isolated git worktree.
- **When:** parallel coding agents that must not collide.
- **Command:** `hermes project bind-board` [V-LIVE]; `hermes -w` [V-LIVE]
### 18. Prompt-size introspection
- **What:** Byte breakdown of system prompt + tool schemas — diagnose why responses feel "dumb" (usually context bloat).
- **Command:** `hermes prompt-size` [V-LIVE]
@@ -0,0 +1,121 @@
# 03 — Command Cheatsheet (every entry verified)
**Verification method:** each VERIFIED-LIVE entry was confirmed against `hermes --help` or `hermes <cmd> --help` on Hermes Agent v0.21.1 (2026.9.7), reference install (Syslog kagentz), 2026-09-11. Raw output: `cli-help-dump.txt`. DOC-ONLY entries come from the official docs (URL given). Nothing is invented.
## (a) CLI — `hermes ...`
### Setup & health
| Command | What it does | Tag |
|---|---|---|
| `hermes setup` | Interactive setup wizard | VERIFIED-LIVE |
| `hermes doctor [--fix] [--live]` | Diagnose config/deps; `--fix` auto-repairs | VERIFIED-LIVE |
| `hermes status [--all] [--deep]` | Component status | VERIFIED-LIVE |
| `hermes config show/edit/get/set/unset/path/env-path/check/migrate` | View/edit config | VERIFIED-LIVE |
| `hermes update` | Update Hermes to latest | VERIFIED-LIVE |
### The Claude Code bridge (highest value for Scot)
| Command | What it does | Tag |
|---|---|---|
| `hermes import-agent claude-code [--dry-run] [--overwrite] [--yes]` | One-command import of a Claude Code setup: maps CLAUDE.md/AGENTS.md instructions, permission allowlists, MCP servers, skills, memories into Hermes equivalents. Never imports API keys. | VERIFIED-LIVE |
| `hermes import-agent codex` | Same for Codex CLI setups | VERIFIED-LIVE |
| `hermes sessions import` | Import a Claude Code or Codex CLI **session/conversation** into Hermes | VERIFIED-LIVE (subcommand listed in `hermes sessions --help`) |
| `hermes skills trust` | Trust a repo so its project-local skills (`./.hermes/skills`) load — the Hermes analog of `.claude/skills/` | VERIFIED-LIVE |
### Daily driving
| Command | What it does | Tag |
|---|---|---|
| `hermes` / `hermes chat` | Interactive session | VERIFIED-LIVE |
| `hermes -c [NAME]` / `hermes --resume <id\|latest>` | Resume by name or ID | VERIFIED-LIVE |
| `hermes --in DIR --resume latest` | Resume the latest session for a directory | VERIFIED-LIVE |
| `hermes -z "PROMPT"` | One-shot: prints ONLY the final answer (scripting/CI); tools, memory, and AGENTS.md still load | VERIFIED-LIVE |
| `hermes chat -q "PROMPT"` | Single-query mode | VERIFIED-LIVE |
| `hermes -m MODEL --provider PROVIDER --reasoning LEVEL` | Per-run model/provider/reasoning overrides (`none…ultra`) | VERIFIED-LIVE |
| `hermes -s SKILL1,SKILL2` | Preload specific skills for the session | VERIFIED-LIVE |
| `hermes -t TOOLSETS` | Restrict toolsets for this run | VERIFIED-LIVE |
| `hermes -w` | Isolated git worktree session (parallel agents on one repo) | VERIFIED-LIVE |
| `hermes chat --checkpoints` | Enable filesystem checkpoints (`/rollback` to restore) | VERIFIED-LIVE |
| `hermes chat --max-turns N` / `--run-budget SECONDS` | Cap loop iterations / wall-clock budget | VERIFIED-LIVE |
### Context & memory management
| Command | What it does | Tag |
|---|---|---|
| `hermes memory setup/status/off/reset` | External memory provider management (built-in MEMORY.md/USER.md always active) | VERIFIED-LIVE |
| `hermes sessions list/browse/rename/pin/export/prune/stats` | Session store management | VERIFIED-LIVE |
| `hermes skills list/search/install/inspect/browse/config/check/update` | Skill management | VERIFIED-LIVE |
| `hermes skills trust/untrust` | Repo-local skill trust | VERIFIED-LIVE |
| `hermes curator status/run/pause/pin/...` | Background skill maintenance (auto-archive, backups) | VERIFIED-LIVE |
| `hermes prompt-size` | Byte breakdown of system prompt + tool schemas (context-bloat diagnosis) | VERIFIED-LIVE |
| `hermes insights [--days N]` | Usage analytics | VERIFIED-LIVE |
### Tools, MCP, integrations
| Command | What it does | Tag |
|---|---|---|
| `hermes tools` (interactive) / `list/enable/disable` | Per-platform toolset toggles; MCP tools as `server:tool` | VERIFIED-LIVE |
| `hermes mcp add/remove/list/test/configure/picker/catalog/install` | MCP server management (incl. one-click catalog installs) | VERIFIED-LIVE |
| `hermes mcp serve` | Run Hermes AS an MCP server for other agents | VERIFIED-LIVE |
| `hermes computer-use install/status/doctor` | Desktop-control backend (cua-driver) | VERIFIED-LIVE |
| `hermes gateway run/install/start/status/setup` | Messaging gateway (Telegram, Discord, Slack, WhatsApp, …) | VERIFIED-LIVE |
| `hermes send` | Send a message to a configured platform (scripts/cron/CI) | VERIFIED-LIVE |
### Automation & multi-agent
| Command | What it does | Tag |
|---|---|---|
| `hermes cron list/create/edit/pause/resume/run/remove/doctor` | Scheduled jobs (durable, multi-platform delivery) | VERIFIED-LIVE |
| `hermes cron notepad` | Durable per-job key-value notepad across runs | VERIFIED-LIVE |
| `hermes kanban create/list/show/link/complete/swarm/...` | Durable multi-profile task board (40+ verbs) | VERIFIED-LIVE |
| `hermes kanban swarm` | Generate a parallel-workers → verifier → synthesizer task graph | VERIFIED-LIVE |
| `hermes project create/list/add-folder/bind-board` | Named multi-folder workspaces (desktop Projects) | VERIFIED-LIVE |
| `hermes profile list/create/use/alias/export/import` | Isolated Hermes instances | VERIFIED-LIVE |
| `hermes auth add/list/priority/reset` | Pooled credentials per provider (rotation) | VERIFIED-LIVE |
| `hermes fallback list/add/remove` | Fallback model chain (auto-rollover on failure) | VERIFIED-LIVE |
| `hermes model` | Interactive model/provider picker | VERIFIED-LIVE |
| `hermes -yolo` | Bypass command approval prompts (use with care) | VERIFIED-LIVE |
| `hermes pause` / `hermes resume` | Emergency stop / lift (pauses cron, kanban dispatch, gateway turns) | VERIFIED-LIVE |
## (b) In-session slash commands
Source: official slash-commands reference https://hermes-agent.nousresearch.com/docs/reference/slash-commands (DOC-ONLY — slash commands run inside a chat session and were not exercised from this headless research run; the CLI subcommands they map to were verified live). DOC-ONLY.
### Context & session
| Command | What it does |
|---|---|
| `/help` | List all commands (authoritative in your version) |
| `/new` (`/reset`) | Fresh session |
| `/resume [name]` | Resume a named/recent session |
| `/branch` (`/fork`) | Branch the current session |
| `/compress` | Manually compress context (auto-compression also exists) |
| `/undo` | Remove last exchange |
| `/retry` | Resend last message |
| `/title [name]` | Name the session |
| `/save` | Save conversation to file |
| `/history` | Show conversation history |
### Power surfaces
| Command | What it does |
|---|---|
| `/skill <name>` | Load a skill into the session |
| `/skills` | Search/install skills |
| `/reload-skills` | Re-scan skill directory |
| `/tools` / `/toolsets` | Manage tools |
| `/goal [text]` | Set a standing goal the agent works toward across turns (`/goal status/pause/clear` to manage) |
| `/background <prompt>` | Run a prompt in the background |
| `/queue <prompt>` | Queue a prompt for the next turn |
| `/steer <prompt>` | Inject a course-correction after the next tool call without interrupting |
| `/agents` | Show active agents and running tasks |
| `/cron` | Manage cron jobs in-session |
| `/kanban` | Multi-profile collaboration board in-session |
| `/model [name]` | Show/change model mid-session |
| `/reasoning [level]` | Set reasoning effort |
| `/voice [on\|off\|tts]` | Voice mode |
| `/rollback [N]` | Restore filesystem checkpoint (needs `--checkpoints`) |
| `/usage` | Token usage |
| `/insights [days]` | Usage analytics |
| `/platforms` | Gateway platform status |
| `/compact`-equivalent note | Hermes compresses automatically near the context limit; no manual threshold watch needed like Claude Code's `/context` |
### The "work alongside Claude Code" shortlist
1. `hermes import-agent claude-code --dry-run` → migrate the setup (VERIFIED-LIVE)
2. `hermes sessions import` → bring the conversation history over (VERIFIED-LIVE)
3. `hermes skills trust` → load repo-local skills like `.claude/skills/` (VERIFIED-LIVE)
4. `hermes -c` / `hermes --in <repo> --resume latest` → per-directory session continuity (VERIFIED-LIVE)
5. `hermes mcp serve` → expose Hermes to Claude Code as an MCP server (VERIFIED-LIVE) — the reverse direction Claude Code can't do
@@ -0,0 +1,76 @@
# 04 — The Claude Code Bridge: Running Hermes WITH Claude Code
**Prepared:** 2026-09-11. Scot already runs both tools. This file documents the proven integration patterns, citing the installed skills on this host (paths under `/home/hermes/.hermes/skills/`) and official docs.
## Pattern 0 — Import (do this first)
`hermes import-agent claude-code` [VERIFIED-LIVE] maps CLAUDE.md/AGENTS.md instructions,
permission allowlists, MCP servers, skills, and memories into Hermes equivalents. It always
shows a preview, never imports credentials. `hermes sessions import` [VERIFIED-LIVE] pulls
in old Claude Code conversations. After import, Hermes "knows" the projects — the context
gap disappears on day one.
## Pattern 1 — Hermes as orchestrator, Claude Code as worker
Source: installed skill **`autonomous-ai-agents/delegate-coding-agent`** (v1.0.0) + its
reference `references/claude-code.md` (v2.2.1) [V-FILE]. The skill is an official Hermes
skill authored for exactly this.
Two orchestration modes (verbatim from the skill):
- **Print mode (preferred):** `claude -p '<task>' --allowedTools 'Read,Edit' --max-turns 10` —
one-shot, no dialogs, structured JSON output with `session_id`, `num_turns`,
`total_cost_usd`. Ask Hermes: *"delegate this coding task to Claude Code in print mode."*
- **Interactive PTY via tmux:** multi-turn sessions — Hermes starts `tmux new-session`,
sends prompts with `send-keys`, monitors with `capture-pane`. For iterative
refactor → review → fix cycles.
Cross-agent review loop (also from the skill):
```
git diff main...feature | claude -p 'Review this diff for bugs and security issues.' --max-turns 1
```
Hermes runs this, reads the output, and fixes findings itself — Claude Code becomes a
reviewer Hermes coordinates.
Safety rails the skill prescribes: explicit `workdir`, clean git status before launch,
narrow task prompts, `git diff` review, targeted tests before committing.
## Pattern 2 — Parallel workstreams + neutral merge reconciliation
Source: installed skill **`autonomous-ai-agents/merge-reconciler`** [V-FILE].
When Hermes and Claude Code (or two Hermes workers) both edit the same repo and collide:
- Do NOT let either agent resolve the conflict — both are biased toward their own side.
- Spawn a **neutral third agent** with the merge-reconciler skill; it classifies every
conflicted hunk (disjoint-intent / same-question-different-answer / superseded), resolves
under an impartiality contract (touch only conflict markers, surface every design call),
verifies with build/tests, and hands back a summary naming every hunk decision.
- Kanban-native shape: a reconciliation card assigned to a **third profile** with both
workers' cards as parents — parent links carry both sides' completion summaries into the
reconciler's context automatically.
## Pattern 3 — Hermes as MCP server (Claude Code gets Hermes tools)
`hermes mcp serve` [VERIFIED-LIVE] runs Hermes as an MCP server exposing its conversations
and capabilities. Claude Code supports MCP clients (`claude mcp add`), so Claude Code can
consume Hermes as a tool provider — persistent memory, skills, cron — the surfaces Claude
Code lacks. This is the reverse-bridge only Hermes can offer. Docs:
https://hermes-agent.nousresearch.com/docs/user-guide/features/mcp
## Pattern 4 — Import legacy sessions for continuity
`hermes sessions import` [VERIFIED-LIVE] imports a Claude Code session into the Hermes
store; from then on `hermes --resume <id>` / `hermes sessions browse` treat it as native
history. Use when mid-project: the new agent picks up exactly where Claude Code left off.
## Pattern 5 — Desktop GUI automation either side can use
Source: installed skill **`autonomous-ai-agents/computer-use`** (v2.0.0) [V-FILE].
`hermes computer-use install` sets up cua-driver; the `computer_use` toolset drives native
desktop apps background-first (never steals focus/cursor), any-model, cross-platform.
Relevant to the bridge because Claude Code has no desktop automation — if a task needs
Figma/Excel/native apps, that part routes to Hermes while the code routes to Claude Code.
Cmd: `hermes computer-use doctor` for health checks.
## Pattern 6 — The import-agent philosophy in one line
Claude Code holds repo context in `CLAUDE.md`; Hermes holds it in `AGENTS.md` + memory +
skills. `hermes import-agent claude-code` translates the first; the learning loop
("save this as a skill") rebuilds the rest automatically the more Scot uses Hermes.
## Reference paths (for the writer)
- `/home/hermes/.hermes/skills/autonomous-ai-agents/delegate-coding-agent/SKILL.md` and `references/claude-code.md`
- `/home/hermes/.hermes/skills/autonomous-ai-agents/merge-reconciler/SKILL.md`
- `/home/hermes/.hermes/skills/autonomous-ai-agents/computer-use/SKILL.md`
- `hermes import-agent --help` raw output in `cli-help-dump.txt` (lines 554-577)
@@ -0,0 +1,119 @@
# 05 — The Video Watch List (every URL verified 2026-09-11)
**Verification method:** each video was found via YouTube search-results scrape (`videoRenderer` metadata), then confirmed with the YouTube oEmbed endpoint (`curl -s "https://www.youtube.com/oembed?url=<URL>&format=json"`) — every entry below returned HTTP 200 with matching title/author (status PASS). Publish dates, durations, and view counts were read from each watch page's metadata. Raw evidence for all 21 entries: `video-verification.json` in this directory. **21/21 PASS, 0 FAIL.**
## Tier 1 — Hermes-specific, start here
### 1. Learn 95% of Hermes Agent in 31 Minutes
- **Channel:** Sharbel A. | **URL:** https://www.youtube.com/watch?v=Ta2wg6xPaY4 | **Duration:** 31:28 | **Published:** 2026-08-09 | **Views:** ~129k
- **oEmbed:** PASS (title/author match)
- **What it demonstrates:** end-to-end Hermes fundamentals — install, sessions, skills, memory, the learning loop. The most complete single-video orientation found.
- **Watch this when you want** the fastest real overview of the whole harness before touching config.
### 2. Hermes Agent Fundamentals In 29 Minutes
- **Channel:** Tina Huang | **URL:** https://www.youtube.com/watch?v=5_N84t1rUU0 | **Duration:** 29:40 | **Published:** 2026-07-20 | **Views:** ~463k (highest-reach Hermes video found)
- **oEmbed:** PASS
- **What it demonstrates:** conceptual grounding — why Hermes' memory/skills loop differs from one-shot coding agents; practical walkthrough.
- **Watch this when you want** to understand *why* Hermes feels different from Claude Code, not just which buttons to press.
### 3. Every Level of Hermes Agent Explained
- **Channel:** Jack Roberts | **URL:** https://www.youtube.com/watch?v=6GtF_uHbGhw | **Duration:** 25:35 | **Published:** 2026-06-17 | **Views:** ~163k
- **oEmbed:** PASS
- **What it demonstrates:** beginner → advanced ladder of features (memory, skills, automation, multi-agent).
- **Watch this when you want** a map of what to learn next after the basics.
### 4. Hermes Agent Full Tutorial INSTALLATION + USECASES
- **Channel:** CodeHead | **URL:** https://www.youtube.com/watch?v=8GjyOQy19so | **Duration:** 7:47 | **Published:** 2026-05-14 | **Views:** ~64k
- **oEmbed:** PASS
- **What it demonstrates:** install through real use-cases, compact.
- **Watch this when you want** a quick install-to-value demo to share with a colleague.
### 5. Hermes Agent Explained In 5 Minutes
- **Channel:** CodeHead | **URL:** https://www.youtube.com/watch?v=9GpWELm3_XI | **Duration:** 4:53 | **Published:** 2026-05-23 | **Views:** ~251k
- **oEmbed:** PASS
- **What it demonstrates:** 5-minute conceptual pitch of the agent and its learning loop.
- **Watch this when you want** the elevator pitch before committing 30 minutes.
## Tier 2 — Hermes-specific deep dives
### 6. 100 Days With Hermes Agent in 21 Minutes
- **Channel:** Sharbel A. | **URL:** https://www.youtube.com/watch?v=sCa3BtpkziQ | **Duration:** 21:19 | **Published:** 2026-06-17 | **Views:** ~58k
- **oEmbed:** PASS
- **What it demonstrates:** long-horizon usage — what memory/skills accumulation actually looks like after months of daily use.
- **Watch this when you want** to see the payoff of the learning loop over time.
### 7. Hermes Agent - Crash Course for Beginners (AI Agent)
- **Channel:** Adrian Twarog | **URL:** https://www.youtube.com/watch?v=4sAmpcSOVEw | **Duration:** 22:19 | **Published:** 2026-07-21 | **Views:** ~46k
- **oEmbed:** PASS
- **What it demonstrates:** beginner crash course from a well-known dev-YouTube creator.
- **Watch this when you want** a second independent explanation of the basics.
### 8. Hermes Agent: The Ultimate Beginner's Guide
- **Channel:** Metics Media | **URL:** https://www.youtube.com/watch?v=CwPUOVUdApE | **Duration:** 37:08 | **Published:** 2026-04-24 | **Views:** ~119k
- **oEmbed:** PASS
- **What it demonstrates:** long-form beginner guide incl. setup and everyday workflows.
- **Watch this when you want** the most thorough single walkthrough in one sitting.
### 9. Hermes Agent Just Killed OpenClaw (Full Tutorial)
- **Channel:** Leon van Zyl | **URL:** https://www.youtube.com/watch?v=jmtpYUOr7_U | **Duration:** 19:59 | **Published:** 2026-04-28 | **Views:** ~16k
- **oEmbed:** PASS
- **What it demonstrates:** full tutorial framing Hermes against the OpenClaw workflow (MCP config, memory, agents).
- **Watch this when you want** a practitioner's feature-by-feature tutorial.
### 10. Hermes Agent vs OpenClaw
- **Channel:** Sharbel A. | **URL:** https://www.youtube.com/watch?v=zwqhemjHq3E | **Duration:** 15:28 | **Published:** 2026-04-20 | **Views:** ~35k
- **oEmbed:** PASS
- **What it demonstrates:** head-to-head comparison of the two agent harnesses.
- **Watch this when you want** the tradeoffs between Hermes and its main alternative.
### 11. Better than OpenClaw? Testing Hermes Agent w/ Qwen 3 model
- **Channel:** Tonbi's AI Garage | **URL:** https://www.youtube.com/watch?v=8tpuky8HpXw | **Duration:** 15:08 | **Published:** 2026-03-11 | **Views:** ~18k
- **oEmbed:** PASS
- **What it demonstrates:** Hermes driven by an OpenRouter-served open model (Qwen 3) — directly relevant to an OpenRouter-connected install.
- **Watch this when you want** to see how small open models behave inside Hermes.
### 12. Use This To Make The Hermes Agent Basically Free
- **Channel:** AI LABS | **URL:** https://www.youtube.com/watch?v=5d02TYoOzfE | **Duration:** 13:08 | **Published:** 2026-07-01 | **Views:** ~55k
- **oEmbed:** PASS
- **What it demonstrates:** running Hermes on cheap/free model backends.
- **Watch this when you want** to cut inference costs on an OpenRouter account.
### 13. Hermes Agent The 24/7 Self-Evolving AI Agent!
- **Channel:** WorldofAI | **URL:** https://www.youtube.com/watch?v=cu2fgknmemA | **Duration:** 9:15 | **Published:** 2026-04-07 | **Views:** ~47k
- **oEmbed:** PASS
- **What it demonstrates:** always-on operation: gateway, cron, background automation.
- **Watch this when you want** to turn Hermes from a chat window into a 24/7 assistant.
## Tier 3 — Adjacent (origin/philosophy; not tutorials)
### 14. Hermes Co-Founder on Building an AI Agent That Improves Itself | Karan Malhotra
- **Channel:** Peter Yang | **URL:** https://www.youtube.com/watch?v=UWjh5Z4s8jY | **Duration:** 46:45 | **Published:** 2026-08-02 | **Views:** ~37k
- **oEmbed:** PASS
- **What it demonstrates:** interview with Hermes' co-founder on the design philosophy (self-improving agents, skills as memory).
- **Watch this when you want** to understand where the product is going.
### 15. Hermes Agent: Agents that grow with you | Episode #357
- **Channel:** Practical AI | **URL:** https://www.youtube.com/watch?v=UTZhvPXnmwA | **Duration:** 47:34 | **Published:** 2026-05-20 | **Views:** ~1.9k
- **oEmbed:** PASS
- **What it demonstrates:** podcast-depth technical discussion of the agent architecture.
- **Watch this when you want** the engineering story behind the learning loop.
### 16. Did Hermes Agent just kill OpenClaw? (full guide)
- **Channel:** Alex Finn | **URL:** https://www.youtube.com/watch?v=tP6yf22OJdI | **Duration:** 13:55 | **Published:** 2026-03-31 | **Views:** ~132k
- **oEmbed:** PASS
- **What it demonstrates:** guide-style comparison/switch content.
- **Watch this when you want** a switcher's guide perspective.
### 17. Hermes Agent: Why Everyone's Ditching OpenClaw in 2026
- **Channel:** Luke Alexander AI | **URL:** https://www.youtube.com/watch?v=1UgXUjT-QtI | **Duration:** 18:03 | **Published:** 2026-03-26 | **Views:** ~14k
- **oEmbed:** PASS — adjacent, comparison content.
- **Watch this when you want** more comparison context.
## Honesty note (required by task spec)
At least 16 of the 17 entries above are directly Hermes-specific (not merely adjacent); the
"fewer than 5 exist" fallback clause was NOT needed — no padding was necessary. Entries
found in search but excluded as thin/low-signal: `iqN6MVzpJTk` (3.1k views, news-style),
`83nWNRKZTCE` (465 views), `6M2tItdARew` (1.2k views), `P2LIFtrRr2U` (promo-style) — all
also verified PASS and kept in `video-verification.json` as spares. No official Nous
Research YouTube tutorial channel was found in searches; the strongest signal of
Hermes-specific video content is the third-party ecosystem above.
@@ -0,0 +1,577 @@
===== hermes chat --help =====
usage: hermes chat [-h] [-q QUERY | --query-file PATH] [--oneshot]
[--image IMAGE] [-m MODEL] [-t TOOLSETS]
[--reasoning LEVEL] [-s SKILLS] [--provider PROVIDER] [-v]
[-Q] [--resume SESSION_ID] [--no-restore-cwd] [--in DIR]
[--continue [SESSION_NAME]] [--create-if-missing]
[--worktree] [--accept-hooks] [--checkpoints]
[--max-turns N] [--run-budget SECONDS] [--yolo]
[--pass-session-id] [--ignore-user-config] [--ignore-rules]
[--safe-mode] [--source SOURCE] [--tui] [--cli] [--dev]
Start an interactive chat session with Hermes Agent
options:
-h, --help show this help message and exit
-q, --query QUERY Query to run. On a real TTY the prompt seeds an
interactive session (submitted literally as the first
turn); combined with --oneshot or -Q, or on a non-TTY,
it answers and exits.
--query-file PATH Read the single query from a file instead of the
command line ('-' reads stdin). Safe for arbitrary
text: nothing is shell-interpreted, so quotes, $(...),
and backticks are preserved verbatim. Mutually
exclusive with -q.
--oneshot With -q/--query-file: answer the query and exit
(legacy single-query behavior) instead of seeding an
interactive session. Implied on non-TTY stdio and by
-Q/--quiet.
--image IMAGE Optional local image path to attach to a single query
-m, --model MODEL Model to use (e.g., anthropic/claude-sonnet-4)
-t, --toolsets TOOLSETS
Comma-separated toolsets to enable
--reasoning LEVEL Reasoning effort for this session: none, minimal, low,
medium, high, xhigh, max, or ultra. Overrides
agent.reasoning_effort for this run only (same levels
as the /reasoning slash command).
-s, --skills SKILLS Preload one or more skills for the session (repeat
flag or comma-separate)
--provider PROVIDER Inference provider (default: auto). Built-in or a
user-defined name from `providers:` in config.yaml.
-v, --verbose Verbose output
-Q, --quiet Quiet mode for programmatic use: suppress banner,
spinner, and tool previews. Only output the final
response and session info.
--resume, -r SESSION_ID
Resume a previous session by ID (shown on exit), or
'latest' for the most recent session
--no-restore-cwd Don't cd into a resumed session's recorded working
directory.
--in DIR Change into DIR before starting or resuming (scopes '
--resume latest' / -c lookups to DIR's workspace).
--continue, -c [SESSION_NAME]
Resume a session by name, or the most recent if no
name given
--create-if-missing With -c/--continue <name>: if no session matches the
name, create a new session with that title and proceed
(instead of failing with a not-found error).
Programmatic callers that want 'send to this named
thread, making it if needed'.
--worktree, -w Run in an isolated git worktree (for parallel agents
on the same repo)
--accept-hooks Auto-approve any unseen shell hooks declared in
config.yaml without a TTY prompt (see also
HERMES_ACCEPT_HOOKS env var and hooks_auto_accept: in
config.yaml).
--checkpoints Enable filesystem checkpoints before destructive file
operations (use /rollback to restore)
--max-turns N Maximum tool-calling iterations per conversation turn
(default: 500, or agent.max_turns in config)
--run-budget SECONDS Optional wall-clock budget in seconds for each
conversation run. At 80% elapsed the agent gets a one-
time wrap-up notice, and implicit provider stale
timeouts are capped to the remaining budget so one
hung call can't consume the run. Unset = off. Also
configurable as agent.run_budget_seconds in
config.yaml. Intended for one-shot/eval invocations
with a hard ceiling.
--yolo Bypass all dangerous command approval prompts (use at
your own risk)
--pass-session-id Include the session ID in the agent's system prompt
--ignore-user-config Ignore ~/.hermes/config.yaml and fall back to built-in
defaults (credentials in .env are still loaded).
Useful for isolated CI runs, reproduction, and third-
party integrations.
--ignore-rules Skip auto-injection of AGENTS.md, SOUL.md,
.cursorrules, memory, and preloaded skills. Combine
with --ignore-user-config for a fully isolated run.
--safe-mode Troubleshooting mode: disable ALL customizations —
user config, AGENTS.md/memory injection, plugins, and
MCP servers (implies --ignore-user-config and
--ignore-rules). Use to isolate whether a problem
comes from your setup or from Hermes itself.
--source SOURCE Session source tag for filtering (default: cli). Use
'tool' for third-party integrations that should not
appear in user session lists.
--tui Launch the modern TUI instead of the classic REPL
--cli Force the classic prompt_toolkit REPL (overrides
display.interface=tui)
--dev With --tui: run TypeScript sources via tsx (skip dist
build)
===== hermes model --help =====
usage: hermes model [-h] [--refresh] [--portal-url PORTAL_URL]
[--inference-url INFERENCE_URL] [--client-id CLIENT_ID]
[--scope SCOPE] [--no-browser] [--timeout TIMEOUT]
[--ca-bundle CA_BUNDLE] [--insecure]
Interactively select your inference provider and default model
options:
-h, --help show this help message and exit
--refresh Wipe the model picker disk cache and re-fetch every
provider's live /v1/models list.
--portal-url PORTAL_URL
Portal base URL for Nous login (default: production
portal)
--inference-url INFERENCE_URL
Inference API base URL for Nous login (default:
production inference API)
--client-id CLIENT_ID
OAuth client id to use for Nous login (default:
hermes-cli)
--scope SCOPE OAuth scope to request for Nous login
--no-browser Do not attempt to open the browser automatically
during Nous login
--timeout TIMEOUT HTTP request timeout in seconds for Nous login
(default: 15)
--ca-bundle CA_BUNDLE
Path to CA bundle PEM file for Nous TLS verification
--insecure Disable TLS verification for Nous login (testing only)
===== hermes config --help =====
usage: hermes config [-h]
{show,edit,get,set,unset,path,env-path,check,migrate} ...
Manage Hermes Agent configuration
positional arguments:
{show,edit,get,set,unset,path,env-path,check,migrate}
show Show current configuration
edit Open config file in editor
get Print a resolved configuration value
set Set a configuration value
unset Remove a configuration value
path Print config file path
env-path Print .env file path
check Check for missing/outdated config
migrate Update config with new options
options:
-h, --help show this help message and exit
===== hermes cron --help =====
usage: hermes cron [-h] [--accept-hooks]
{list,create,add,edit,pause,resume,run,remove,rm,delete,status,runs,history,incidents,notepad,doctor,tick} ...
Manage scheduled tasks
positional arguments:
{list,create,add,edit,pause,resume,run,remove,rm,delete,status,runs,history,incidents,notepad,doctor,tick}
list List scheduled jobs
create (add) Create a scheduled job
edit Edit an existing scheduled job
pause Pause a scheduled job
resume Resume a paused job
run Run a job on the next scheduler tick
remove (rm, delete)
Remove a scheduled job
status Check if cron scheduler is running
runs (history) Show durable execution attempts
incidents List or acknowledge durable cron failure incidents
notepad Read/write a job's durable notepad (persistent KV
across runs)
doctor Check scheduled jobs for common health issues
tick Run due jobs once and exit
options:
-h, --help show this help message and exit
--accept-hooks Auto-approve unseen shell hooks without a TTY prompt
(equivalent to HERMES_ACCEPT_HOOKS=1 /
hooks_auto_accept: true).
===== hermes kanban --help =====
usage: hermes kanban [-h] [--board <slug>]
{init,boards,create,swarm,list,ls,show,assign,set-model,reclaim,reassign,diagnostics,diag,link,unlink,claim,comment,attach,attachments,attach-rm,complete,edit,block,schedule,unblock,request-review,request-changes,reopen-review,promote,archive,tail,dispatch,daemon,watch,stats,notify-subscribe,notify-list,notify-unsubscribe,log,runs,heartbeat,assignees,context,specify,decompose,gc,repair} ...
Durable SQLite-backed task board shared across Hermes profiles. Tasks are
claimed atomically, can depend on other tasks, and are executed by a named
profile in an isolated workspace. See https://hermes-
agent.nousresearch.com/docs/user-guide/features/kanban or docs/hermes-
kanban-v1-spec.pdf for the full design.
positional arguments:
{init,boards,create,swarm,list,ls,show,assign,set-model,reclaim,reassign,diagnostics,diag,link,unlink,claim,comment,attach,attachments,attach-rm,complete,edit,block,schedule,unblock,request-review,request-changes,reopen-review,promote,archive,tail,dispatch,daemon,watch,stats,notify-subscribe,notify-list,notify-unsubscribe,log,runs,heartbeat,assignees,context,specify,decompose,gc,repair}
init Create kanban.db if missing (idempotent)
boards Manage kanban boards (one board per project /
workstream)
create Create a new task
swarm Create a Kanban Swarm v1 graph (parallel workers →
verifier → synthesizer)
list (ls) List tasks
show Show a task with comments + events
assign Assign or reassign a task
set-model Set or clear a task's model/provider override (takes
effect on the next dispatch)
reclaim Release an active worker claim on a running task
reassign Reassign a task to a different profile, optionally
reclaiming first
diagnostics (diag) List active diagnostics on the current board
link Add a parent->child dependency
unlink Remove a parent->child dependency
claim Atomically claim a ready task (prints resolved
workspace path)
comment Append a comment
attach Attach a local file to a task
attachments List a task's attachments
attach-rm Delete an attachment by id
complete Mark one or more tasks done
edit Edit recovery fields on an already-completed task
block Mark one or more tasks blocked
schedule Park one or more tasks in Scheduled (waiting on time,
not human input)
unblock Return blocked/scheduled tasks to ready, or todo while
parents remain open
request-review Move a task to 'review' (implementation done, awaiting
review) — NOT a block
request-changes Reviewer verdict: return the active review run to its
implementer
reopen-review Send one or more review tasks back for changes (review
-> ready/todo)
promote Manually move one or more todo/blocked tasks to ready
(recovery path)
archive Archive one or more tasks
tail Follow a task's event stream
dispatch One dispatcher pass: reclaim stale, promote ready,
spawn workers
daemon DEPRECATED — dispatcher now runs in the gateway. Use
`hermes gateway start`.
watch Live-stream task_events to the terminal (Ctrl+C to
exit)
stats Per-status + per-assignee counts + oldest-ready age
notify-subscribe Subscribe a gateway source to a task's terminal events
(used by /kanban subscribe in the gateway adapter)
notify-list List notification subscriptions (optionally for a
single task)
notify-unsubscribe Remove a gateway subscription from a task
log Print the worker log for a task (from <kanban-
root>/kanban/logs/)
runs Show attempt history for a task (one row per run:
profile, outcome, elapsed, summary)
heartbeat Emit a heartbeat event for a running task (worker
liveness signal)
assignees List known profiles + per-profile task counts (union
of ~/.hermes/profiles/ and current assignees on the
board)
context Print the full context a worker sees for a task (title
+ body + parent results + comments).
specify Flesh out a triage-column task into a concrete spec
(title + body) and promote it to todo. Uses the
auxiliary LLM configured under
auxiliary.triage_specifier.
decompose Decompose a triage-column task into a graph of child
tasks routed to specialist profiles by description.
Falls back to specify-style single-task promotion when
the task doesn't benefit from fan-out. Uses
auxiliary.kanban_decomposer.
gc Garbage-collect archived-task workspaces, old events,
and old logs
repair Check kanban.db integrity and auto-repair index-only
corruption
options:
-h, --help show this help message and exit
--board <slug> Board slug to operate on. Defaults to the current
board (set via `hermes kanban boards switch <slug>` or
the HERMES_KANBAN_BOARD env var). Use `hermes kanban
boards list` to see all boards.
===== hermes skills --help =====
usage: hermes skills [-h]
{trust,untrust,browse,search,install,inspect,list,check,update,audit,uninstall,reset,list-modified,diff,opt-out,opt-in,repair-official,publish,snapshot,tap,config} ...
Search, install, inspect, audit, configure, and manage skills from skills.sh,
well-known agent skill endpoints, GitHub, ClawHub, and other registries.
positional arguments:
{trust,untrust,browse,search,install,inspect,list,check,update,audit,uninstall,reset,list-modified,diff,opt-out,opt-in,repair-official,publish,snapshot,tap,config}
trust Trust a project so its repo-local skills
(./.hermes/skills, ./.agents/skills) load
untrust Revoke project-skill trust for a repo
browse Browse all available skills (paginated)
search Search skill registries
install Install a skill
inspect Preview a skill without installing
list List installed skills
check Check installed hub skills for updates
update Update installed hub skills
audit Re-scan installed hub skills
uninstall Remove a hub-installed skill
reset Reset a bundled skill — clears 'user-modified'
tracking so updates work again
list-modified List bundled skills you've edited (which `hermes
update` keeps)
diff Show how your copy of a bundled skill differs from the
stock version
opt-out Stop bundled skills from being seeded into this
profile
opt-in Re-enable bundled-skill seeding (undo opt-out)
repair-official Backfill or restore official optional skills from repo
source
publish Publish a skill to a registry
snapshot Export/import skill configurations
tap Manage skill sources
config Interactive skill configuration — enable/disable
individual skills
options:
-h, --help show this help message and exit
===== hermes sessions --help =====
usage: hermes sessions [-h]
{list,export,delete,prune,archive,optimize,clean-markers,optimize-storage,repair,repair-routing,recover,stats,rename,pin,unpin,pinned,retitle-skills,browse,import} ...
View and manage the SQLite session store
positional arguments:
{list,export,delete,prune,archive,optimize,clean-markers,optimize-storage,repair,repair-routing,recover,stats,rename,pin,unpin,pinned,retitle-skills,browse,import}
list List recent sessions
export Export sessions to JSONL, Markdown, or QMD
delete Delete a specific session
prune Delete old sessions (filterable by time window,
source, title, ...)
archive Bulk-archive (soft-hide) sessions matching filters —
no deletion
optimize Reclaim disk space: merge FTS5 segments + VACUUM (no
data change)
clean-markers Permanently clear stale tool-call marker content left
by sessions from before #78148
optimize-storage Migrate the search index to the compact v23 layout
(reclaims disk on large DBs)
repair Repair a malformed state.db schema so hidden sessions
reappear
repair-routing Re-stamp gateway sessions that lost their routing
identity
recover Rebuild canonical session data into a separate clean
database
stats Show session store statistics
rename Set or change a session's title
pin Pin session(s) — durable keep flag, exempt from auto-
archive
unpin Remove the pin (durable keep flag) from session(s)
pinned List pinned sessions
retitle-skills Re-title sessions whose auto-title came from a
/skill's own text
browse Interactive session picker — browse, search, and
resume sessions
import Import a Claude Code or Codex CLI session into Hermes
options:
-h, --help show this help message and exit
===== hermes mcp --help =====
usage: hermes mcp [-h] [--accept-hooks]
{serve,add,remove,rm,list,ls,test,configure,config,login,reauth,picker,catalog,install} ...
Manage MCP server connections and run Hermes as an MCP server. MCP servers
provide additional tools via the Model Context Protocol. Use 'hermes mcp add'
to connect to a new server, or 'hermes mcp serve' to expose Hermes
conversations over MCP.
positional arguments:
{serve,add,remove,rm,list,ls,test,configure,config,login,reauth,picker,catalog,install}
serve Run Hermes as an MCP server (expose conversations to
other agents)
add Add an MCP server (discovery-first install)
remove (rm) Remove an MCP server
list (ls) List configured MCP servers
test Test MCP server connection
configure (config) Toggle tool selection
login Force re-authentication for an OAuth-based MCP server
reauth Re-authenticate one OAuth MCP server, or all of them
(--all)
picker Interactive catalog picker (also the default for
`hermes mcp`)
catalog List Nous-approved MCPs available for one-click
install
install Install a catalog MCP by name (e.g. `hermes mcp
install n8n`)
options:
-h, --help show this help message and exit
--accept-hooks Auto-approve unseen shell hooks without a TTY prompt
(equivalent to HERMES_ACCEPT_HOOKS=1 /
hooks_auto_accept: true).
===== hermes profile --help =====
usage: hermes profile [-h]
{list,use,create,delete,describe,show,alias,rename,export,import,install,update,info} ...
positional arguments:
{list,use,create,delete,describe,show,alias,rename,export,import,install,update,info}
list List all profiles
use Set sticky default profile
create Create a new profile
delete Delete a profile
describe Read or set a profile's description (used by the
kanban orchestrator)
show Show profile details
alias Manage wrapper scripts
rename Rename a profile ('default': sets a display name; id
unchanged)
export Export a profile to archive
import Import a profile from archive
install Install a profile distribution from a git URL or local
directory
update Re-pull a distribution and apply updates (user data
preserved)
info Show a profile's distribution manifest (version,
requirements, source)
options:
-h, --help show this help message and exit
===== hermes memory --help =====
usage: hermes memory [-h] {setup,status,off,reset} ...
Set up and manage external memory provider plugins. Available providers:
honcho, openviking, mem0, hindsight, holographic, retaindb, byterover. Only
one external provider can be active at a time. Built-in memory
(MEMORY.md/USER.md) is always active.
positional arguments:
{setup,status,off,reset}
setup Interactive provider selection and configuration
status Show current memory provider config
off Disable external provider (built-in only)
reset Erase all built-in memory (MEMORY.md and USER.md)
options:
-h, --help show this help message and exit
===== hermes tools --help =====
usage: hermes tools [-h] [--summary] {list,disable,enable,post-setup} ...
Enable, disable, or list tools for CLI, Telegram, Discord, etc. Built-in
toolsets use plain names (e.g. web, memory). MCP tools use server:tool
notation (e.g. github:create_issue). Run 'hermes tools' with no subcommand for
the interactive configuration UI.
positional arguments:
{list,disable,enable,post-setup}
list Show all tools and their enabled/disabled status
disable Disable toolsets or MCP tools
enable Enable toolsets or MCP tools
post-setup Run a provider's post-setup install hook
(npm/pip/binary)
options:
-h, --help show this help message and exit
--summary Print a summary of enabled tools per platform and exit
===== hermes project --help =====
usage: hermes project [-h]
{create,list,ls,show,add-folder,remove-folder,rename,set-primary,use,archive,restore,bind-board} ...
Projects are human-named workspaces that can span multiple folders / repos.
They anchor desktop session grouping and, when bound to a kanban board, give
tasks a deterministic worktree + branch convention. State is per-profile.
positional arguments:
{create,list,ls,show,add-folder,remove-folder,rename,set-primary,use,archive,restore,bind-board}
create Create a new project
list (ls) List projects
show Show a project's details
add-folder Add a folder to a project
remove-folder Remove a folder from a project
rename Rename a project
set-primary Set the primary folder
use Set the active project
archive Archive a project
restore Restore an archived project
bind-board Bind a kanban board to a project
options:
-h, --help show this help message and exit
===== hermes gateway --help =====
usage: hermes gateway [-h] [--accept-hooks]
{run,start,stop,restart,status,install,uninstall,list,setup,migrate-legacy,enroll} ...
Manage the messaging gateway (Telegram, Discord, WhatsApp, Weixin, and more)
positional arguments:
{run,start,stop,restart,status,install,uninstall,list,setup,migrate-legacy,enroll}
run Run gateway in foreground (recommended for WSL,
Docker, Termux)
start Start the installed systemd/launchd background service
stop Stop gateway service
restart Restart gateway service
status Show gateway status
install Install gateway as a systemd/launchd background
service
uninstall Uninstall gateway service
list List all profiles and their gateway status
setup Configure messaging platforms
migrate-legacy Remove legacy hermes.service units from pre-rename
installs
enroll Enroll this gateway with a relay connector (writes
relay auth creds to .env)
options:
-h, --help show this help message and exit
--accept-hooks Auto-approve unseen shell hooks without a TTY prompt
(equivalent to HERMES_ACCEPT_HOOKS=1 /
hooks_auto_accept: true).
===== hermes computer-use --help =====
usage: hermes computer-use [-h] {install,status,doctor,permissions} ...
Install or check the cua-driver binary used by the `computer_use` toolset.
Supported on macOS, Windows, and Linux. Use `hermes computer-use install` to
fetch and run the upstream cua-driver installer. This is equivalent to the
post-setup hook that `hermes tools` runs when you first enable the Computer
Use toolset, and is a stable target for re-running the install if it didn't
fire (e.g. when toggling the toolset on a returning-user setup). Use `hermes
computer-use doctor` to run cua-driver's `health_report` MCP tool and surface
its check matrix (TCC, bundle identity, version, platform support, ...) in
human-readable form.
positional arguments:
{install,status,doctor,permissions}
install Install or repair the cua-driver binary
(macOS/Windows/Linux)
status Print whether cua-driver is installed and on PATH
doctor Run cua-driver `health_report` and surface the check
matrix
permissions Check or grant macOS Accessibility + Screen Recording
(macOS)
options:
-h, --help show this help message and exit
===== hermes doctor --help =====
usage: hermes doctor [-h] [--fix] [--live] [--ack ADVISORY_ID]
Diagnose issues with Hermes Agent setup
options:
-h, --help show this help message and exit
--fix Attempt to fix issues automatically
--live Opt-in: run one bounded, read-only real-call health probe
per configured tool backend
(Firecrawl/FAL/browser/MCP/TTS/STT) after the static
checks. Makes real network calls.
--ack ADVISORY_ID Acknowledge a security advisory by ID and exit. After
ack, the advisory will no longer trigger startup banners.
Run `hermes doctor` first to see active advisories and
their IDs.
===== hermes status --help =====
usage: hermes status [-h] [--all] [--deep]
Display status of Hermes Agent components
options:
-h, --help show this help message and exit
--all Show all details (redacted for sharing)
--deep Run deep checks (may take longer)
===== hermes import-agent --help =====
usage: hermes import-agent [-h] [--source SOURCE] [--dry-run] [--overwrite]
[--yes]
[{claude-code,codex}]
One-command import of another coding agent's setup into Hermes. Maps
CLAUDE.md/AGENTS.md instructions, permission allowlists, MCP servers, skills,
and memories into their Hermes equivalents. Always shows a preview before
making changes. API keys and credentials are never imported — run 'hermes
setup' for those.
positional arguments:
{claude-code,codex} Which agent to import from (default: auto-detect
~/.claude or ~/.codex)
options:
-h, --help show this help message and exit
--source SOURCE Path to the agent's config directory (default:
~/.claude or ~/.codex)
--dry-run Preview only — stop after showing what would be
imported
--overwrite Overwrite existing Hermes items on name conflicts
(default: skip)
--yes, -y Skip confirmation prompts
@@ -0,0 +1,33 @@
import json, subprocess, re
data = json.load(open("/home/hermes/syslog/drafts/scot-hermes-playbook/research/video-verification.json"))
have = {r["videoId"] for r in data}
extra = []
for v in ["8GjyOQy19so", "zwqhemjHq3E"]:
if v in have:
continue
url = f"https://www.youtube.com/watch?v={v}"
oe = subprocess.run(["curl", "-s", f"https://www.youtube.com/oembed?url={url}&format=json"],
capture_output=True, text=True, timeout=30)
try:
oej = json.loads(oe.stdout)
o = {"status": "PASS", "title": oej.get("title"), "author": oej.get("author_name")}
except Exception:
o = {"status": "FAIL", "raw": oe.stdout[:200]}
wp = subprocess.run(["curl", "-s", "-L", url,
"-H", "User-Agent: Mozilla/5.0 (Windows NT 10.0) Chrome/124.0",
"-H", "Accept-Language: en-US"], capture_output=True, text=True, timeout=30)
html = wp.stdout
pub = re.search(r'"publishDate":"([\d\-T:Z]+)"', html)
dur = re.search(r'"lengthSeconds":"(\d+)"', html)
views = re.search(r'"viewCount":"(\d+)"', html)
rec = {"videoId": v, "oembed": o,
"publishDate": pub.group(1) if pub else None,
"lengthSeconds": int(dur.group(1)) if dur else None,
"views": int(views.group(1)) if views else None}
extra.append(rec)
print(json.dumps(rec))
data.extend(extra)
json.dump(data, open("/home/hermes/syslog/drafts/scot-hermes-playbook/research/video-verification.json", "w"), indent=2)
print("total videos in ledger:", len(data))
@@ -0,0 +1,38 @@
import json, subprocess, re, sys
vids = ["9GpWELm3_XI","iqN6MVzpJTk","Ta2wg6xPaY4","8tpuky8HpXw","tP6yf22OJdI",
"UWjh5Z4s8jY","5d02TYoOzfE","83nWNRKZTCE","6M2tItdARew","5_N84t1rUU0",
"UTZhvPXnmwA","1UgXUjT-QtI","6GtF_uHbGhw","CwPUOVUdApE","sCa3BtpkziQ",
"jmtpYUOr7_U","cu2fgknmemA","4sAmpcSOVEw","P2LIFtrRr2U"]
results = []
for v in vids:
url = f"https://www.youtube.com/watch?v={v}"
oe = subprocess.run(["curl","-s",f"https://www.youtube.com/oembed?url={url}&format=json"],
capture_output=True, text=True, timeout=30)
try:
oej = json.loads(oe.stdout)
oembed = {"status":"PASS","title":oej.get("title"),"author":oej.get("author_name")}
except Exception:
oembed = {"status":"FAIL","raw":oe.stdout[:200]}
# watch page for date + duration
wp = subprocess.run(["curl","-s","-L",url,"-H","User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/124.0",
"-H","Accept-Language: en-US,en;q=0.9"], capture_output=True, text=True, timeout=30)
html = wp.stdout
pub = re.search(r'"publishDate":"([\d\-T:Z]+)"', html)
upd = re.search(r'"uploadDate":"([\d\-T:Z]+)"', html)
dur = re.search(r'"lengthSeconds":"(\d+)"', html)
views = re.search(r'"viewCount":"(\d+)"', html)
results.append({
"videoId": v, "oembed": oembed,
"publishDate": pub.group(1) if pub else None,
"uploadDate": upd.group(1) if upd else None,
"lengthSeconds": int(dur.group(1)) if dur else None,
"views": int(views.group(1)) if views else None,
"watchpage_bytes": len(html),
})
print(json.dumps(results[-1]))
with open("/home/hermes/syslog/drafts/scot-hermes-playbook/research/video-verification.json","w") as f:
json.dump(results, f, indent=2)
print("saved video-verification.json")
@@ -0,0 +1,271 @@
[
{
"videoId": "9GpWELm3_XI",
"oembed": {
"status": "PASS",
"title": "Hermes Agent Explained In 5 Minutes",
"author": "CodeHead"
},
"publishDate": "2026-05-23T08:00:27-07:00",
"uploadDate": "2026-05-23T08:00:27-07:00",
"lengthSeconds": 293,
"views": 251275,
"watchpage_bytes": 1419048
},
{
"videoId": "iqN6MVzpJTk",
"oembed": {
"status": "PASS",
"title": "Hermes Agent by Nous Research: The Open-Source Agent Model Everyone Is Switching To",
"author": "Praveen Govindaraj"
},
"publishDate": "2026-02-26T21:56:54-08:00",
"uploadDate": "2026-02-26T21:56:54-08:00",
"lengthSeconds": 193,
"views": 3140,
"watchpage_bytes": 1305120
},
{
"videoId": "Ta2wg6xPaY4",
"oembed": {
"status": "PASS",
"title": "Learn 95% of Hermes Agent in 31 Minutes",
"author": "Sharbel A."
},
"publishDate": "2026-08-09T07:00:19-07:00",
"uploadDate": "2026-08-09T07:00:19-07:00",
"lengthSeconds": 1888,
"views": 129313,
"watchpage_bytes": 1468130
},
{
"videoId": "8tpuky8HpXw",
"oembed": {
"status": "PASS",
"title": "Better than OpenClaw? Testing Hermes Agent w/ Qwen 3 model",
"author": "Tonbi's AI Garage"
},
"publishDate": "2026-03-11T07:00:14-07:00",
"uploadDate": "2026-03-11T07:00:14-07:00",
"lengthSeconds": 907,
"views": 18167,
"watchpage_bytes": 1326925
},
{
"videoId": "tP6yf22OJdI",
"oembed": {
"status": "PASS",
"title": "Did Hermes Agent just kill OpenClaw? (full guide)",
"author": "Alex Finn"
},
"publishDate": "2026-03-31T06:15:10-07:00",
"uploadDate": "2026-03-31T06:15:10-07:00",
"lengthSeconds": 835,
"views": 132128,
"watchpage_bytes": 1390199
},
{
"videoId": "UWjh5Z4s8jY",
"oembed": {
"status": "PASS",
"title": "Hermes Co-Founder on Building an AI Agent That Improves Itself | Karan Malhotra",
"author": "Peter Yang"
},
"publishDate": "2026-08-02T06:00:12-07:00",
"uploadDate": "2026-08-02T06:00:12-07:00",
"lengthSeconds": 2804,
"views": 36613,
"watchpage_bytes": 1386664
},
{
"videoId": "5d02TYoOzfE",
"oembed": {
"status": "PASS",
"title": "Use This To Make The Hermes Agent Basically Free",
"author": "AI LABS"
},
"publishDate": "2026-07-01T07:00:26-07:00",
"uploadDate": "2026-07-01T07:00:26-07:00",
"lengthSeconds": 788,
"views": 54858,
"watchpage_bytes": 1412833
},
{
"videoId": "83nWNRKZTCE",
"oembed": {
"status": "PASS",
"title": "Meet the AI Agent That Grows With You Hermes Agent by Nous Research",
"author": "Eddy Says Hi #EddySaysHi"
},
"publishDate": "2026-03-21T13:00:09-07:00",
"uploadDate": "2026-03-21T13:00:09-07:00",
"lengthSeconds": 366,
"views": 465,
"watchpage_bytes": 1254068
},
{
"videoId": "6M2tItdARew",
"oembed": {
"status": "PASS",
"title": "The AI Agent That Never Forgets: Meet Hermes Agent by Nous Research",
"author": "Siggi"
},
"publishDate": "2026-03-11T13:53:17-07:00",
"uploadDate": "2026-03-11T13:53:17-07:00",
"lengthSeconds": 371,
"views": 1199,
"watchpage_bytes": 1267812
},
{
"videoId": "5_N84t1rUU0",
"oembed": {
"status": "PASS",
"title": "Hermes Agent Fundamentals In 29 Minutes",
"author": "Tina Huang"
},
"publishDate": "2026-07-20T09:37:02-07:00",
"uploadDate": "2026-07-20T09:37:02-07:00",
"lengthSeconds": 1780,
"views": 462764,
"watchpage_bytes": 1561142
},
{
"videoId": "UTZhvPXnmwA",
"oembed": {
"status": "PASS",
"title": "Hermes Agent: Agents that grow with you |Episode #357|",
"author": "Practical AI"
},
"publishDate": "2026-05-20T15:00:17-07:00",
"uploadDate": "2026-05-20T15:00:17-07:00",
"lengthSeconds": 2853,
"views": 1888,
"watchpage_bytes": 1329765
},
{
"videoId": "1UgXUjT-QtI",
"oembed": {
"status": "PASS",
"title": "Hermes Agent: Why Everyone's Ditching OpenClaw in 2026",
"author": "Luke Alexander AI"
},
"publishDate": "2026-03-26T07:10:29-07:00",
"uploadDate": "2026-03-26T07:10:29-07:00",
"lengthSeconds": 1083,
"views": 13625,
"watchpage_bytes": 1321848
},
{
"videoId": "6GtF_uHbGhw",
"oembed": {
"status": "PASS",
"title": "Every Level of Hermes Agent Explained",
"author": "Jack Roberts"
},
"publishDate": "2026-06-17T12:27:42-07:00",
"uploadDate": "2026-06-17T12:27:42-07:00",
"lengthSeconds": 1535,
"views": 162977,
"watchpage_bytes": 1577121
},
{
"videoId": "CwPUOVUdApE",
"oembed": {
"status": "PASS",
"title": "Hermes Agent: The Ultimate Beginner\u2019s Guide",
"author": "Metics Media"
},
"publishDate": "2026-04-24T06:59:04-07:00",
"uploadDate": "2026-04-24T06:59:04-07:00",
"lengthSeconds": 2228,
"views": 118612,
"watchpage_bytes": 1637578
},
{
"videoId": "sCa3BtpkziQ",
"oembed": {
"status": "PASS",
"title": "100 Days With Hermes Agent in 21 Minutes",
"author": "Sharbel A."
},
"publishDate": "2026-06-17T07:47:26-07:00",
"uploadDate": "2026-06-17T07:47:26-07:00",
"lengthSeconds": 1279,
"views": 57581,
"watchpage_bytes": 1454230
},
{
"videoId": "jmtpYUOr7_U",
"oembed": {
"status": "PASS",
"title": "Hermes Agent Just Killed OpenClaw (Full Tutorial)",
"author": "Leon van Zyl"
},
"publishDate": "2026-04-28T04:19:38-07:00",
"uploadDate": "2026-04-28T04:19:38-07:00",
"lengthSeconds": 1199,
"views": 15988,
"watchpage_bytes": 1510488
},
{
"videoId": "cu2fgknmemA",
"oembed": {
"status": "PASS",
"title": "Hermes Agent The 24/7 Self-Evolving AI Agent!",
"author": "WorldofAI"
},
"publishDate": "2026-04-07T00:01:34-07:00",
"uploadDate": "2026-04-07T00:01:34-07:00",
"lengthSeconds": 555,
"views": 46889,
"watchpage_bytes": 1545788
},
{
"videoId": "4sAmpcSOVEw",
"oembed": {
"status": "PASS",
"title": "Hermes Agent - Crash Course for Beginners (AI Agent)",
"author": "Adrian Twarog"
},
"publishDate": "2026-07-21T01:20:41-07:00",
"uploadDate": "2026-07-21T01:20:41-07:00",
"lengthSeconds": 1338,
"views": 46484,
"watchpage_bytes": 1585548
},
{
"videoId": "P2LIFtrRr2U",
"oembed": {
"status": "PASS",
"title": "Hermes Agent: New FREE OpenClaw Alternative!",
"author": "Julian Goldie SEO"
},
"publishDate": "2026-03-09T14:00:32-07:00",
"uploadDate": "2026-03-09T14:00:32-07:00",
"lengthSeconds": 735,
"views": 9975,
"watchpage_bytes": 1353286
},
{
"videoId": "8GjyOQy19so",
"oembed": {
"status": "PASS",
"title": "Hermes Agent Full Tutorial INSTALLATION + USECASES",
"author": "CodeHead"
},
"publishDate": "2026-05-14T08:00:23-07:00",
"lengthSeconds": 467,
"views": 64285
},
{
"videoId": "zwqhemjHq3E",
"oembed": {
"status": "PASS",
"title": "Hermes Agent vs OpenClaw",
"author": "Sharbel A."
},
"publishDate": "2026-04-20T07:14:00-07:00",
"lengthSeconds": 928,
"views": 35360
}
]
@@ -0,0 +1,10 @@
import re, sys
html = open(sys.argv[1], encoding='utf-8', errors='ignore').read()
ids = re.findall(r'"videoRenderer":\{"videoId":"([\w-]{11})"', html)
print("videoRenderer hits:", len(set(ids)))
for vid in dict.fromkeys(ids):
m = re.search(r'"videoId":"%s".{0,3000}?"title":\{"runs":\[\{"text":"(.*?)"\}' % vid, html, re.S)
ch = re.search(r'"videoId":"%s".{0,6000}?"ownerText":\{"runs":\[\{"text":"(.*?)"' % vid, html, re.S)
dur = re.search(r'"videoId":"%s".{0,4000}?"lengthText":\{"accessibility".{0,400}?"simpleText":"(.*?)"' % vid, html, re.S)
print((vid, m.group(1) if m else "?", ch.group(1) if ch else "?", dur.group(1) if dur else "?"))
@@ -0,0 +1,42 @@
# 06 — Sources
Access date for ALL entries: **2026-09-11** (via citation ledger `sources.py`; doc URLs additionally confirmed HTTP 200 by curl -L).
## Official docs (hermes-agent.nousresearch.com)
| # | URL | Supported |
|---|-----|-----------|
| 1 | https://hermes-agent.nousresearch.com/docs | Docs index; overall feature map |
| 3 | https://hermes-agent.nousresearch.com/docs/user-guide/configuration | Config sections, SOUL.md, checkpoints |
| 4 | https://hermes-agent.nousresearch.com/docs/reference/slash-commands | Slash command registry (03) |
| 5 | https://hermes-agent.nousresearch.com/docs/reference/tools-reference | Toolset inventory (02 §6) |
| 6 | https://hermes-agent.nousresearch.com/docs/user-guide/features/cron | Cron surface (02 §9) |
| 7 | https://hermes-agent.nousresearch.com/docs/user-guide/features/kanban | Kanban surface (02 §8) |
| 8 | https://hermes-agent.nousresearch.com/docs/user-guide/features/mcp | MCP surface + `hermes mcp serve` (02 §5, 04 P3) |
| 9 | https://hermes-agent.nousresearch.com/docs/user-guide/features/memory | Memory surface (02 §2) |
| 10 | https://hermes-agent.nousresearch.com/docs/user-guide/profiles | Profiles (02 §13) |
| 11 | https://hermes-agent.nousresearch.com/docs/integrations/providers | Model/provider routing (02 §12) |
| 12 | https://hermes-agent.nousresearch.com/docs/user-guide/messaging/ | Gateway platforms (02 §15) |
| 13 | https://hermes-agent.nousresearch.com/docs/user-guide/features/curator | Skill maintenance (02 §3) |
| 14 | https://hermes-agent.nousresearch.com/docs/reference/cli-commands | CLI command index cross-check (03) |
| 15 | https://hermes-agent.nousresearch.com/docs/reference/skills-catalog | Skills catalog (02 §3) |
## GitHub
| # | URL | Supported |
|---|-----|-----------|
| 2 | https://github.com/nousresearch/hermes-agent | Repo identity, learning-loop description (01, 02) |
## Local primary sources (not web URLs; verified on this host)
- Live CLI help output, Hermes Agent v0.21.1 (2026.9.7), reference install (Syslog kagentz): `hermes --help` + `hermes {chat,model,config,cron,kanban,skills,sessions,mcp,profile,memory,tools,project,gateway,computer-use,doctor,status,import-agent} --help` → raw dump `cli-help-dump.txt` (577 lines). Basis for all VERIFIED-LIVE tags in 01/02/03/04.
- `/home/hermes/.hermes/skills/autonomous-ai-agents/delegate-coding-agent/SKILL.md` (v1.0.0) + `references/claude-code.md` (v2.2.1) → 04 Patterns 1-2, 01 Claude Code column.
- `/home/hermes/.hermes/skills/autonomous-ai-agents/merge-reconciler/SKILL.md` → 04 Pattern 2.
- `/home/hermes/.hermes/skills/autonomous-ai-agents/computer-use/SKILL.md` (v2.0.0) → 04 Pattern 5, 02 §11.
## YouTube verification
- YouTube search results pages (scraped 2026-09-11): `https://www.youtube.com/results?search_query=hermes+agent+nous+research` and `...nous+research+hermes+agent+official`
- oEmbed endpoint per video: `https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v=<ID>&format=json` — all 21 checked IDs returned PASS (HTTP 200, title/author match). Raw evidence incl. publishDate/lengthSeconds/viewCount per watch page: `video-verification.json`.
- 17 listed in 05-videos.md + 4 spares; 21/21 pass, 0 fail.
## Explicit gaps (could not close)
1. No official Nous Research-produced tutorial video was found — video list is third-party ecosystem content (disclosed in 05).
2. Slash commands were verified DOC-ONLY (https://hermes-agent.nousresearch.com/docs/reference/slash-commands); they require an interactive session to exercise, which this headless run does not have. CLI equivalents were verified live.
3. `hermes-agent.nousresearch.com/docs/developer-guide/` returned 404 — developer docs live in-repo (`AGENTS.md` in the GitHub repo), not as a docs site section.
+29 -33
View File
@@ -19,7 +19,7 @@ triggers:
- on model add/remove
- on GPU health degradation
- on agent key rotation
- on router restart (roster must be loaded)
- on harness container restart (LiteLLM reloads its model list)
---
## Maintains
@@ -36,46 +36,42 @@ triggers:
- prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
- port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding
## Fleet Topology (Current — July 2026)
## Fleet Topology (Current — 2026-09-11, router decommissioned)
```
┌──────────────────────────────────────────────────────────────────┐
│ CT 116 (192.168.68.116) — Inference Harness Host │
│ │
│ nginx:80 (entrypoint) │
│ ├─ /v1/* → harness-litellm:4000 (API requests) │
│ ├─ /v1/* → harness-litellm:4000 (API requests) │
│ ├─ /admin/* → harness-litellm:4000 (admin endpoints) │
│ ├─ /dashboard/ → harness-dashboard:3000 (harness UI) │
│ ├─ /litellm/* → harness-litellm:4000 (LiteLLM UI + API) │
│ ├─ /litellm/* → harness-litellm:4000 (LiteLLM UI + API) │
│ ├─ /health/* → harness-litellm:4000 (health probes) │
│ └─ /gpu/* → 192.168.68.24:9100 (fleet monitor) │
│ │
│ Containers: │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ LiteLLM │ │ Router │ │Dashboard │ │ Grafana │ │
│ │ :4000 │ │ :9000 │ │ :3000 │ │ :3000 │ │
│ │ keys+sync│ │deprecated│ │ harness │ │ Prometheus│ │
│ │ fallback │ │not in │ │ UI │ │ data src │ │
│ └──────────┘ └───┬──────┘ └──────────┘ └──────────┘ │
│ │ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │PostgreSQL│ │ Redis │ │Prometheus│ │
│ │ :5432 │ │ :6379 │ │ :9090 │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└────────────────────┼─────────────────────────────────────────────┘
│
┌───────────────┼───────────────┬──────────────────┐
│ │ │ │
┌────▼─────┐ ┌──────▼──────┐ ┌────▼──────┐ ┌───────▼──────┐
│ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │
│ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │
│ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │
│ 128K ctx │ │ 128K ctx │ │ 128K ctx │ │ Watchdog │
│ qwen3.6 │ │ Qwen3.5-9B │ │ qwen3.5 │ │ Prometheus │
│ 27B-code │ │ :8080 │ │ -35B-udq4 │ │ exporter │
│ :8080 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │
│ :9400 │ └─────────────┘ └───────────┘ └──────────────┘
└──────────┘
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ LiteLLM │ │Dashboard │ │ Grafana │ │
│ │ :4000 │ │ :3000 │ │ :3001 │ │
│ │ keys+sync│ │ harness │ │Prometheus│ │
│ │ fallback │ │ UI │ │ data src │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │PostgreSQL│ │ Redis │ │Prometheus│ │
│ │ :5432 │ │ :6379 │ │ :9090 │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└───────┼──────────────────────────────────────────────────────────┘
│
┌─┴─────────────┬───────────────┬───────────────┐
│ │ │ │
┌─────▼──────┐ ┌─────▼──────┐ ┌─────▼──────┐ ┌─────▼──────┐
│ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │
│ RTX 3090 │ │ RTX 5070 │ │ Strix Halo │ │ GPU Monitor│
│ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │
│ :8080 │ │ :9400 exp │ │ :9400 exp │ │ :9401 │
└────────────┘ └────────────┘ └────────────┘ └────────────┘
```
## Stable Role-Based Aliases (Introduced 2026-07-15)
@@ -105,7 +101,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
|-------|-----|--------|---------|---------|
| Qwen3.8-27B-Uncensored-Q4_K_M | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path.
Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) was decommissioned 2026-09-11 and is NOT in the inference path.
### Direct Model Endpoints
@@ -153,7 +149,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
6. Cleanup model files (optional)
### heal
1. Check all GPUs via router internal `:9000/health/unified`
1. Check all GPUs via gpu-monitor `{{gpu_dashboard_url}}/gpu-data`
2. Check LiteLLM health via nginx `:80/litellm/health/liveliness`
3. Reset stuck circuit breakers if idle (Redis)
4. Restart dead llama-server instances via SSH
@@ -161,7 +157,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
6. Verify GPU monitor server is running on pi (:9100)
7. Verify watchdog is running on pi
8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
9. Reload roster via `POST :9000/admin/roster/reload` if available
9. Router roster reload — REMOVED 2026-09-11 (router decommissioned; LiteLLM fallbacks handle routing)
### sync-keys
1. List all agent keys in LiteLLM DB via `GET /key/list`
@@ -258,7 +254,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
Benchmarks from 2026-07-17. Strix Halo model: qwen3.6-35B-udq4. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
All 3 GPUs now at 128K context (2026-07-17, reduced from 256K for stability).
Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
Benchmarks run through LiteLLM proxy (192.168.68.116:4000) every 5 minutes.
Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
History stored at `/root/data/toks-history.json` with 7-day rolling window.
+17 -17
View File
@@ -2,7 +2,7 @@
kind: responsibility
name: gpu-monitor
description: >
Comprehensive GPU fleet monitor — polls every subsystem (sidecars, router,
Comprehensive GPU fleet monitor — polls every subsystem (sidecars,
LiteLLM, Strix Halo, dashboard) every 15s, renders a live HTML dashboard,
checks alert thresholds, and exposes a JSON API for downstream consumers.
agent: abiba
@@ -46,19 +46,19 @@ and `http://192.168.68.110/health` answer `000`. Never use a bare-port-80 probe
as a GPU liveness signal: on 2026-09-09 that produced three false
`DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000` rounds while
`http://192.168.68.8:8080/health` and `http://192.168.68.110:8080/health`
answered `200`. Port 80 is valid only on the router (.116), never on a GPU host.
answered `200`. Port 80 is valid only on the harness host (.116), never on a GPU host.
### Subsystems Polled
| Subsystem | Endpoint | Frequency | Metrics |
|-----------|----------|-----------|---------|
| GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) — `301` → `/gpu/gpu-data` is **alive** |
| GPU Status (all, via fleet API) | `http://192.168.68.116/gpu/gpu-data` | 15s | models, CB, scores, GPU status from gpu-monitor on .24:9100 |
| GPU .8 (RTX 3090) health | `http://192.168.68.8:8080/health` | 15s | direct liveness fallback — **:8080 ONLY, never bare port 80** |
| GPU .110 (RTX 5070) health | `http://192.168.68.110:8080/health` | 15s | direct liveness fallback — **:8080 ONLY, never bare port 80** |
| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (`301` → `/gpu/gpu-data` = alive) |
| Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
| Fleet (unified) | `http://192.168.68.116/health/unified` | 15s | nginx `301` → `/gpu/gpu-data` served by gpu-monitor = alive (router decommissioned 2026-09-11) |
| Harness (basic) | `http://192.168.68.116/health` | 15s | nginx → LiteLLM `/health/liveliness` |
| LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | Strix Halo status via router — cannot poll .15:8080 directly (firewalled to .116 only) |
| Strix Halo | `http://192.168.68.116/health/unified` (nginx → fleet API) | 15s | Strix Halo status via gpu-monitor — cannot poll .15:8080 directly (firewalled to .116 only) |
| Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |
### Alert Delivery
@@ -75,8 +75,8 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and
### Liveness rule (scoped)
The any-HTTP-response rule applies ONLY to redirect/auth-gated liveness
endpoints, where any HTTP answer proves a listener is up. Applied here: the
router's `/health/unified` answers `301 Moved Permanently` → `/gpu/gpu-data`
endpoints, where any HTTP answer proves a listener is up. Applied here: nginx's
`/health/unified` answers `301 Moved Permanently` → `/gpu/gpu-data`
(the same payload) and LiteLLM's `/litellm/health` answers `301` →
`/litellm/health/liveliness`. For those endpoints a probe is **ALIVE** on
**ANY** HTTP status — `3xx` redirects and `401`/`403` auth challenges included —
@@ -84,8 +84,8 @@ and **DOWN = connection refused (`000`) or timeout only**. Same scoped rule as
zulip-health (Tanko) and infrastructure-monitoring.
Probes whose success condition is specifically a bare `200` are NOT covered by
the any-HTTP rule. On those — the GPU `:8080/health` endpoints, the router
`/health`, and the dashboard — an unexpected status (`401`/`403`, `5xx`, or
the any-HTTP rule. On those — the GPU `:8080/health` endpoints, nginx `/health`,
and the dashboard — an unexpected status (`401`/`403`, `5xx`, or
anything other than the expected `200`) is an **ALERT**, not "alive".
| Metric | Warning | Critical |
@@ -93,7 +93,7 @@ anything other than the expected `200`) is an **ALERT**, not "alive".
| GPU Temp | >80°C | >90°C |
| VRAM Usage | >90% | >95% |
| GPU Util | >95% | >98% |
| Sidecar Unreachable | — | info (sidecars not deployed — use router /health/unified) |
| Sidecar Unreachable | — | info (sidecars not deployed — use gpu-monitor /gpu-data) |
| Model Down | — | critical (circuit breaker open) |
### JSON API Response Schema (/gpu-data)
@@ -174,13 +174,13 @@ Open `http://localhost:9100/` in browser — Live HTML dashboard
pkill -f gpu-monitor-server.py
python3 /root/scripts/gpu-monitor-server.py &
```
Or via PM2: `pm2 restart gpu-monitor`
Managed by systemd (verified 2026-09-11): `systemctl restart gpu-monitor`
### check-router
The router health is accessed through nginx on port 80 on the **router**
(.116) — NOT port 9000 directly, and NOT bare port 80 on a GPU host.
`curl http://192.168.68.116/health/unified` — Router unified health via nginx proxy; answers `301` → `/gpu/gpu-data` (same payload) = alive
`curl http://192.168.68.116:9000/health/unified` — ❌ WILL FAIL (port bound to 127.0.0.1 only)
### check-fleet
Fleet health is accessed through nginx on port 80 on the harness host
(.116) — NOT port 9000 (router decommissioned 2026-09-11), and NOT bare
port 80 on a GPU host.
`curl http://192.168.68.116/health/unified` — nginx answers `301` → `/gpu/gpu-data` (fleet monitor payload) = alive
`curl http://192.168.68.8/health` — ❌ NEVER USE (GPU host, no port-80 listener → false `000`/DEGRADED)
## Configuration Files
+5 -5
View File
@@ -9,7 +9,7 @@ description: >
(v2.1.0) with active remediation rules, Prometheus metrics consumption,
VRAM trend analysis, and predictive alerting.
UPDATED 2026-07-18: Model assignments synced to 2026-07-17 swaps.
Router (port 9000) references replaced with direct GPU routing.
Router (port 9000) DECOMMISSIONED 2026-09-11; references replaced with direct GPU routing.
Benchmark baselines refreshed to live values.
Prometheus exporters removed — not deployed; fall back to direct sidecar probes.
Stable role-based aliases (strix-moe, gpu-dense, gpu-light) from gpu-fleet.
@@ -51,7 +51,7 @@ depends_on:
| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |
Key notes:
- All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
- All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) was decommissioned 2026-09-11 and is NOT in the inference path.
- Stable aliases (gpu-dense, gpu-light, strix-moe) from gpu-fleet are the canonical names for agent configs. Model-specific names still work but are deprecated.
- RTX 5070 tok/s is ~145 for Qwen3.5-9B — gpu-light is the fastest endpoint. Route vision/web/light work there first. NOTE: Qwen3.5-9B is multimodal (image+text), NOT text-only like gemma-4-12b was.
- Strix Halo is 62.9 tok/s (89% of 70.5 baseline) — below optimal but stable. Check for competing workloads.
@@ -104,10 +104,10 @@ Key notes:
### Rule 5: Circuit Breaker Stuck Open
- **Detect**: Circuit breaker open >10 minutes with GPU reporting healthy
- **Note**: Router (port 9000) is deprecated. If circuit breakers are reported by gpu-monitor, they come from LiteLLM's internal tracking, not the old router.
- **Note**: Router (port 9000) was decommissioned 2026-09-11. If circuit breakers are reported by gpu-monitor, they come from LiteLLM's internal tracking, not the old router.
- **Fix**:
1. Verify GPU /health returns 200 on direct port (:8080)
2. If GPU healthy, alert but do NOT reset via router API (deprecated)
2. If GPU healthy, alert but do NOT reset via router API (decommissioned 2026-09-11)
3. Check LiteLLM health directly: http://192.168.68.116/litellm/health/liveliness
4. Restart LiteLLM container on CT 116 if circuit breakers are stuck
- **Verify**: LiteLLM returns healthy, circuit breaker clears within 60s
@@ -276,7 +276,7 @@ Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchab
2. **Model restart**: ✅ Only if >50% failure rate over 60s + 30s grace period. Not on single stuck request.
3. **Strix direct access**: ✅ Open firewall .15:8080 → .24 for direct health probe + restart.
4. **VRAM thresholds**: Tiered — **300MB/h** (RTX 3090), **300MB/h** (RTX 5070), 200MB/h (Strix). Previous values (100/50) were too sensitive; raised 2026-07-18 based on operational data.
5. **CB auto-reset**: ✅ Router deprecated — circuit breakers go through LiteLLM health check + container restart if needed. No per-GPU auto-reset.
5. **CB auto-reset**: ✅ Router decommissioned 2026-09-11 — circuit breakers go through LiteLLM health check + container restart if needed. No per-GPU auto-reset.
6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Current baselines live in gpu-monitor.
7. **Predictive alerts**: Two-tier — warn at >70°C+rising (>2°C/min), critical at >80°C+rising.
8. **Prometheus**: ❌ Not deployed. Use direct sidecar probes (:8080/health) and gpu-monitor API. Prometheus integration deferred until exporters are running on GPU hosts.
+1 -1
View File
@@ -39,7 +39,7 @@ runtime via `infisical run --` wrapper. Plaintext keys removed from this baselin
```
Infisical vault → infisical run -- hermes gateway → LITELLM_API_KEY (runtime)
↓
Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → GPU (llama-server)
└── Key DB (Postgres)
```
+3 -7
View File
@@ -199,15 +199,14 @@ description: >
### Ecosystem B: CT 116 syslog-api (192.168.68.116)
8 containers in inference-harness stack:
11 containers in inference-harness stack (verified live 2026-09-11; LiteLLM upgraded 1.90.0-rc.1 -> 1.99.1):
| Container | Image | Port | Role |
|-----------|-------|------|------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | API proxy, key mgmt, fallbacks |
| harness-router | inference-harness-router | :9000 (127.0.0.1) | GPU routing, slot booking, CB |
| harness-litellm | ghcr.io/docker.litellm.ai/berriai/litellm:1.99.1 | :4000→:4000 | API proxy, key mgmt, fallbacks |
| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/ |
| harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) |
| harness-redis | redis:7-alpine | :6379 | Router slots, circuit breakers |
| harness-redis | redis:7-alpine | :6379 | LiteLLM cache + rate-limit state |
| harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI |
| harness-grafana | grafana/grafana | :3000→:3001 (direct LAN, not behind nginx) | GPU + Proxmox + Docker dashboards |
| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 6 jobs |
@@ -588,9 +587,6 @@ curl -s -H "Authorization: Bearer $(infisical secrets get LITELLM_MASTER_KEY --p
# Storage check
ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
# Router roster reload (if needed)
curl -s -X POST http://192.168.68.116:9000/admin/roster/reload \
-H "Authorization: Bearer sk-admin-ee09fffd04978b61a1569ac670c68814"
# Restart stuck GPU (saturation watchdog alternative)
ssh root@192.168.68.8 "systemctl restart llama-server"
+2 -2
View File
@@ -173,7 +173,7 @@ curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
# Expected: {"status":"ok","version":"..."}
# LiteLLM metrics (Prometheus endpoint)
curl -s http://192.168.68.116:4001/metrics | head -20
curl -s http://192.168.68.116:4000/metrics | head -20
# Expected: Prometheus-formatted metrics output
```
@@ -242,5 +242,5 @@ curl -s http://192.168.68.116:9090/api/v1/targets
curl -s http://192.168.68.116:3001/api/health
# LiteLLM metrics (already live)
curl -s http://192.168.68.116:4001/metrics | head -20
curl -s http://192.168.68.116:4000/metrics | head -20
```
+20 -7
View File
@@ -3,15 +3,16 @@ kind: responsibility
name: infrastructure-update
description: >
Autonomous system-wide update contract covering all 5 Proxmox nodes,
15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
15+ containers/VMs, and 5 Docker ecosystems (docker-vm .7, CT 116 .116,
CT 117, hwpve .11, NetBird VPS 72.61.0.17). Updates apt packages,
Docker images, and container stacks in safe waves with health checks
and automatic rollback on failure.
agent: abiba
triggers:
- on "infra update" command
- weekly (Sunday 03:00 EDT) via cron
- weekly (Sunday 03:00 America/New_York) via Agent Zero scheduler task "weekly-fleet-docker-update" (qSOOVzsU) — implemented 2026-09-08
- on security advisory relay from Mumuni
version: 1.2.0
version: 1.3.0
---
## Maintains
@@ -80,7 +81,13 @@ Before ANY update wave:
| VM 109 (.7) | Home stack (Pulse, Stirling PDF) — JDownloader moved to CT 118 LXC 2026-08-01 | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
| CT 117 (storepve) | Zulip | `pct exec 117 -- bash -c 'cd /opt/zulip && docker compose pull && docker compose up -d'` (from storepve; compose recreates on zulip_default network) |
| CT 117 (storepve) | Jitsi | `pct exec 117 -- bash -c 'cd /opt/jitsi && docker compose pull && docker compose up -d'` (from storepve) |
| hwpve (.11) | Authentik (server, worker, postgres) | `ssh root@192.168.68.11 'cd /root && docker compose pull && docker compose up -d'` |
| NetBird VPS (72.61.0.17) | NetBird (server, dashboard, proxy, traefik, crowdsec) | `ssh root@72.61.0.17 'cd /root && docker compose pull && docker compose up -d'` |
| VM 109 (.7) | Trove test | `cd /opt/trove-test && docker compose pull && docker compose up -d` |
| VM 109 (.7) | docker-stats | `cd /opt/docker-stats && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Monitoring (Grafana, Prometheus, Alertmanager, PVE exporter) | `cd /opt/monitoring && docker compose pull && docker compose up -d` |
**Verify after Wave 3:**
- All containers healthy: `docker ps` on each host
@@ -88,8 +95,12 @@ Before ANY update wave:
- MCP integration test: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools (23 RA-H OS + 67 GitHub)
- Zulip test: send test message to #agent-hub
- Dashboard loading: `curl localhost:3001/` (via CT 116)
- Firecrawl test: `curl :3002/`
- Firecrawl test: `curl -X POST http://192.168.68.7:3002/v1/search -H 'Content-Type: application/json' -d '{"query":"health","limit":1}'` → `"success":true` (GET `/` returns 200)
- Authentik test: `curl http://192.168.68.11:9000/` → 302 redirect to login
- NetBird test: `curl -s -o /dev/null -w '%{http_code}' https://netbird.sysloggh.net/` → 200
- harness-litellm cold start: allow 3-5 min after recreate — reports unhealthy and :4000 refuses connections while loading config/DB, then recovers to 200 on its own (verified 2026-09-08)
- SearXNG test: `curl :8888`
- Digest-pin sweep: `grep -rn '@sha256:' /opt/*/docker-compose.y*` on every host — digest-pinned images are INVISIBLE to `docker compose pull` (the pin re-pulls the same digest forever, so new releases never appear). Flag every pin in the run report and propose un-pinning to a floating tag with user approval before editing. Found 2026-09-10: audiobookshelf was digest-pinned at 2.34.0 (container created 2026-07-18) and silently missed by every sweep; dockhand stack was also pinned (stack removed 2026-09-10, unused). After un-pinning audiobookshelf to :latest it updated to 2.36.0 and verified HTTP 200.
## Wave 4: Proxmox Kernel Reboot
@@ -158,6 +169,8 @@ Before Wave 1, snapshot these files:
/opt/search-stack/searxng/docker-compose.yml (VM 109 .7)
/opt/home_stack/docker-compose.yml (VM 109 .7)
/opt/audiobookshelf/docker-compose.yml (VM 109 .7)
/root/compose.yml (hwpve .11 — Authentik server/worker/postgres)
/root/docker-compose.yml (NetBird VPS — netbird server/dashboard/proxy, traefik, crowdsec)
/root/.pi/agent/extensions/config.yaml (CT 100 .24)
/etc/systemd/system/strix-server.service (amdpve .15 — strix-moe)
/etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110)
@@ -193,7 +206,7 @@ mcp_servers:
| Key | MCP Access |
|-----|-----------|
| Master key | ✅ Full — 90 tools (vault-injected) |
| Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.90.0-rc.1 |
| Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.99.1 |
### Known Limitations
- Per-key MCP server grants not functional — only master key has access
@@ -220,7 +233,7 @@ When LiteLLM is upgraded to a version supporting per-key MCP grants:
- [ ] All 5 PVE nodes updated, no reboot-loop
- [ ] All VMs/CTs running post-update
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117 + hwpve .11 + NetBird VPS)
- [ ] LiteLLM inference passing (syslog-auto test)
- [ ] Zulip server + all 3 agents connected
- [ ] GPU fleet at full capacity (3/3)
+12 -13
View File
@@ -12,8 +12,8 @@ note: >
description: >
Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
Router (harness-router :9000) is DEPRECATED — container still runs but
is not in the request path. GPU monitoring via Prometheus/Grafana and
Router (harness-router :9000) was DECOMMISSIONED 2026-09-11 (container,
image and config removed). GPU monitoring via Prometheus/Grafana and
fleet dashboard (gpu-monitor :9100).
Designed as a reusable contract for any Syslog agent.
@@ -42,8 +42,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
│
Grafana :3001
harness-router :9000 — DEPRECATED, container still runs but
NOT in request path. nginx routes /v1 → LiteLLM directly.
harness-router :9000 — DECOMMISSIONED 2026-09-11 (container, image
and config removed). nginx routes /v1 → LiteLLM directly.
Router slot booking + circuit breakers replaced by
LiteLLM native fallbacks + timeouts.
```
@@ -100,8 +100,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
| Container | Image | Port | Health Check |
|-----------|-------|------|-------------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
| harness-litellm | ghcr.io/docker.litellm.ai/berriai/litellm:1.99.1 | :4000→:4000 | /health/liveliness |
| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
| harness-redis | redis:7-alpine | :6379 | PING |
@@ -121,14 +120,14 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
4. **Check backend container health**:
- SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
- Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
- SSH to {{backend_host}} → `docker ps` → verify 11 containers healthy
- Critical: harness-litellm, harness-nginx, harness-postgres
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus,
harness-alertmanager, harness-zulip-bridge, harness-docker-stats, harness-pve-exporter
5. **Check router roster loaded**:
- GET http://{{backend_host}}:9000/health → expect 200
- GET http://{{backend_host}}:9000/health/unified → expect 3 models
- If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload
5. **Check GPU fleet health via gpu-monitor** (router decommissioned 2026-09-11):
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
- nginx `/health/unified` is now a `301` redirect to `/gpu/gpu-data` (same payload)
6. **Check GPU fleet health** (via fleet dashboard):
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
+12 -11
View File
@@ -20,7 +20,7 @@ note: >
Last verified: 2026-07-12
description: >
LiteLLM inference stack health monitoring + self-healing. Verifies the full
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
nginx → LiteLLM → GPU chain, 11 containers on CT 116, 3 GPU hosts, model
inference, and agent keys. Applies remediation rules for common failures.
Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
---
@@ -41,8 +41,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
│
Grafana :3001
harness-router :9000 — DEPRECATED, container still runs but
NOT in request path. nginx routes /v1 → LiteLLM directly.
harness-router :9000 — DECOMMISSIONED 2026-09-11 (container, image
and config removed). nginx routes /v1 → LiteLLM directly.
Router slot booking + circuit breakers replaced by
LiteLLM native fallbacks + timeouts.
```
@@ -98,8 +98,7 @@ Preferred implementation: uncap shared pool, add capped alias for crew-only.
| Container | Image | Port | Health Check |
|-----------|-------|------|-------------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
| harness-litellm | ghcr.io/docker.litellm.ai/berriai/litellm:1.99.1 | :4000→:4000 | /health/liveliness |
| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
| harness-redis | redis:7-alpine | :6379 | PING |
@@ -113,7 +112,7 @@ Preferred implementation: uncap shared pool, add capped alias for crew-only.
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v4 (2026-09-10) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Abiba is pi-only since the harness purge, so its Hermes config/wrapper/gateway legs are skipped rather than reported as faults; koby is **report-only** (captain's 2026-08-17 ruling) — its findings go to the `--json` `report_only` array and are never counted as fleet failures or repaired, and its CT 111 liveness is probed on storepve (.6). Covers: LiteLLM keys, GPU ports, agent gateways, CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Every run/report carries the absolute execution path (`script=` + `cwd=`). Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v4 (2026-09-10) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Abiba is pi-only since the harness purge, so its Hermes config/wrapper/gateway legs are skipped rather than reported as faults; koby is **report-only** (captain's 2026-08-17 ruling) — its findings go to the `--json` `report_only` array and are never counted as fleet failures or repaired, and its CT 111 liveness is probed on storepve (.6). Covers: LiteLLM keys, GPU ports, agent gateways, CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Every run/report carries the absolute execution path (`script=` + `cwd=`). The current fleet roster is owned by the script changelog (`scripts/agent-health-check.py`); mumuni is no longer probed from this host. Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
## Maintains
@@ -146,10 +145,11 @@ Run this first on every cycle. Results feed into remediation rules below.
- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
### 3. Check backend container health
- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
- SSH to {{backend_host}} → `docker ps` → verify 11 containers healthy
- Critical: harness-litellm, harness-nginx, harness-postgres
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
- Deprecated but running: harness-router (not in path, reference only)
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus,
harness-alertmanager, harness-zulip-bridge, harness-docker-stats, harness-pve-exporter
- Decommissioned 2026-09-11: harness-router (container, image and config removed)
### 4. Check GPU fleet health (via fleet dashboard)
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
@@ -215,8 +215,9 @@ Fix → generate keys in LiteLLM via /key/generate → update /etc/environment o
Escalate → if SSH access unavailable, send Zulip DM
### Rule 9: Stale Active Counter in Redis — DEPRECATED
Router no longer in path so Redis active counters are unused. Rule retained
for reference but inactive. If Redis issues occur, check harness-redis container.
Router no longer in path so router active-slot counters are unused. Rule retained
for reference but inactive. `harness-redis` now serves only LiteLLM cache and
rate-limit state; check the container if cache errors appear.
---
---
+28 -6
View File
@@ -6,7 +6,7 @@ name: memory-fixer
description: >
Auto-fix low-hanging fruit in the RA-H OS knowledge graph. No judgment calls — only deterministic Level 1 operations.
Escalate anything that needs Kwame's input. Executes confirmed Kwame decisions to completion (state + updated_at).
version: 2.0.0
version: 2.1.0
---
---
@@ -69,13 +69,15 @@ FROM nodes
WHERE json_extract(metadata, '$.namespace') IS NULL;
```
### 3. Staleness Review Tagging
### 3. Staleness Review Tagging (refresh-suggested nodes only)
Using the type-based windows from the memory-monitor contract, tag nodes stale beyond their window. **Only process a maximum of 10 nodes per run** to avoid overwhelming Kwame. Prioritize infrastructure first, then dynamic, then ephemeral.
**Archive-suggested nodes are NO LONGER tagged — they are archived outright (see Level 1 fix 4).** Tagging with `[REVIEW: refresh]` applies only to living nodes (infrastructure, deployment, system, system-health, business, philosophy, research, learning, investigation, analysis, project, agent, registry, policy).
**Exclusion Rules:**
- Nodes with `state` = `review_pending`, `deprecated`, `archived`, or `not_processed` are NOT processed
- Nodes whose `description` already starts with `[REVIEW:` are NOT re-processed
- Nodes whose `description` already starts with `[REVIEW:` or `[ARCHIVED]` are NOT re-processed
```sql
SELECT id, title, json_extract(metadata, '$.type') as node_type,
@@ -104,9 +106,28 @@ LIMIT 10;
For each identified node, call `updateNode(id, { description: "[REVIEW: action] " + originalDescription })`.
### 4. Stale-Node Archiving (Level 1 — standing Kwame directive, 2026-09-11)
**Kwame's standing directive: stale nodes CAN be archived by the fixer. No per-batch escalation, no `[REVIEW: archive]` tagging — archive them.**
For every node whose suggested action is `archive` (i.e. its type is NOT one of the living types in fix 3), archive it in a **single** `updateNode` call:
```python
updateNode(id, {
"description": "[ARCHIVED] " + originalDescriptionWithoutReviewTag,
"metadata": {"state": "archived"}
})
```
- `state` transitions **DO work through `updateNode`** (`archived`, and back to `active`). The former "state only accepts processed/not_processed, use SSH" claim was wrong — verified 2026-09-11 by archiving 7 nodes (#61, #373, #388, #465, #475, #526, #1476) over the bridge with `updated_at` auto-bumping. **SSH to the bridge host is a fallback, not a requirement**, and it is blocked from kagentz anyway.
- Pass `description` and `metadata` in the **same** call, and always keep the `updates` object nested: `{"id": N, "updates": {…}}`.
- Archiving is non-destructive: the node stays in the graph, marked `state: archived` + `[ARCHIVED] ` prefix. **Living nodes (refresh-suggested) are NEVER archived** without a specific Kwame decision — they are the cluster/agent/business canon.
**Archive candidates are identified by the fix 3 query's `suggested_action = 'archive'` branch** (the `ELSE 'archive'` case: anything not an infrastructure/skill/documentation/strategic/audit type).
## Level 2 Escalations (Kwame Decision Required)
1. **Stale nodes** flagged with `[REVIEW: …]` — Archive, refresh, or keep?
1. **Refresh-suggested stale nodes** flagged with `[REVIEW: refresh]` — refresh or keep? (Archive-suggested nodes are auto-archived under fix 4 and are not escalated.)
2. **Duplicate Nodes** (same title or >70% title overlap) — Merge or keep?
3. **Orphan Nodes >90 days old** — Archive or connect?
@@ -144,7 +165,7 @@ Reply with:
The fixer reads Kwame's previous response and **executes the decision to completion** — it must not leave a node in review-pending forever. Tagging alone is NOT enough; each confirmed decision must also update `state` and `updated_at` so the node drops out of the stale window on the next run.
> ⚠️ `updateNode` cannot set `state` to non-standard values (restricted to `processed`/`not_processed`) and cannot add metadata keys. For state transitions and `updated_at` bumps, use **direct SSH + SQLite** on the bridge host:
> ⚠️ **Corrected 2026-09-11:** `updateNode` DOES accept `state` changes — `{"updates": {"description": …, "metadata": {"state": "archived"}}}` works over the bridge, and `updated_at` bumps automatically. The old "use direct SSH + SQLite for state transitions" instruction was based on a wrong assumption; SSH is a fallback only (and is blocked from kagentz). Use one `updateNode` call for both the tag and the state.
> ```bash
> ssh root@192.168.68.65 "sqlite3 /root/.local/share/RA-H/db/rah.sqlite \"UPDATE nodes SET metadata = json_set(metadata, '$.state', '<state>'), updated_at = datetime('now') WHERE id = <id>;\""
> ```
@@ -172,8 +193,9 @@ The result must be 0 rows when all decisions are executed. Report what was done.
## Checks
- **State integrity:** archived nodes have `state: archived` + `[ARCHIVED]` prefix; kept nodes are `state: active` without a `[REVIEW:]` tag.
- **Auto-archive applied:** no node should ever be left tagged `[REVIEW: archive]` — that tag is retired. Any `[REVIEW: archive]` found means fix 4 was skipped; archive it and report.
- **No review-pending forever:** after executing Kwame's decisions, `[REVIEW:%` node count must be 0.
- **Timestamps:** every executed decision bumps `updated_at`, so the node exits the stale window on the next run.
- **Timestamps:** every executed decision (and every auto-archive) bumps `updated_at`, so the node exits the stale window on the next run.
## Logging
Every Level 1 fix logged to `~/.hermes/logs/memory-fixer/YYYY-MM-DD.md`
+8 -2
View File
@@ -17,8 +17,9 @@ Changelog:
v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114),
abiba (.24).
Fleet roster: tanko (.122), koby (.129), koonimo (.114), abiba (.24).
(v2 also carried a mumuni probe; see v5 — mumuni is no longer probed: she
moved to her own container, kagentz CT 105 / .14, and is monitored there.)
v3 (2026-09-08): GPU unit repoint verified live (.8 llama-chat-api.service,
.110 llama-server.service, .15 strix-server.service) — .8 was probing a stale
llama-server unit that reads inactive, producing false UNREACHABLE legs.
@@ -44,6 +45,11 @@ Changelog:
separate from `failures`. Every run prints absolute execution provenance
(script + cwd) in the header, in the cron ALERT line, and in --json output so
a stale-consumer report is distinguishable from a fault at read time.
v5 (2026-09-10): roster correction only, no behavior change. mumuni was removed
from the AGENTS dict when she moved off this host onto her own container
(kagentz CT 105 on minipve, .14, dedicated `hermes` user) and is monitored
from her side. This script must not probe mumuni or .24 — the v2 changelog
roster line was the last reference still placing her at .24 / CT100.
"""
import subprocess, json, sys, os, time, re, io, contextlib
+67 -60
View File
@@ -15,7 +15,7 @@ import smtplib, json, subprocess, os, sys, datetime, re
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
PVE = "https://minipve.sysloggh.net"
PVE = "https://192.168.68.12:8006"
AUTH = "Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
# ── Shared credentials —─
@@ -38,10 +38,16 @@ TIME_STR = NOW.strftime("%Y-%m-%d %H:%M UTC")
# ── Helpers ──
def pve_get(path):
cmd = f'curl -sfk --connect-timeout 10 "{PVE}{path}" -H "{AUTH}"'
"""Fetch PVE API data. Returns list on success, None on error (to distinguish from empty list)."""
cmd = f'curl -sk --connect-timeout 10 "{PVE}{path}" -H "{AUTH}"'
try:
return json.loads(subprocess.check_output(cmd, shell=True))["data"]
except: return []
r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=12)
if r.returncode != 0:
return None
data = json.loads(r.stdout)
return data.get("data", [])
except:
return None
def ssh(host, cmd):
try:
@@ -97,21 +103,33 @@ def collect():
# ── Proxmox Nodes ──
nodes = pve_get("/api2/json/nodes")
report["nodes"] = {n["node"]: {
"cpu_pct": round(n.get('cpu',0)*100, 1),
"ram": f"{n.get('mem',0)//1024//1024}/{n.get('maxmem',0)//1024//1024}MB",
"ram_pct": round(n.get('mem',0)/n.get('maxmem',1)*100, 0),
"disk": f"{n.get('disk',0)//1024//1024//1024}/{n.get('maxdisk',0)//1024//1024//1024}GB",
"disk_pct": round(n.get('disk',0)/n.get('maxdisk',1)*100, 0),
"uptime_h": n.get('uptime',0)//3600,
"status": n["status"]
} for n in nodes}
report["node_count"] = len(nodes)
report["nodes_online"] = sum(1 for n in nodes if n["status"] == "online")
if nodes is None:
report["nodes"] = {}
report["node_count"] = 0
report["nodes_online"] = 0
report["pve_probe_status"] = "unreachable"
else:
report["nodes"] = {n["node"]: {
"cpu_pct": round(n.get('cpu',0)*100, 1),
"ram": f"{n.get('mem',0)//1024//1024}/{n.get('maxmem',0)//1024//1024}MB",
"ram_pct": round(n.get('mem',0)/n.get('maxmem',1)*100, 0),
"disk": f"{n.get('disk',0)//1024//1024//1024}/{n.get('maxdisk',0)//1024//1024//1024}GB",
"disk_pct": round(n.get('disk',0)/n.get('maxdisk',1)*100, 0),
"uptime_h": n.get('uptime',0)//3600,
"status": n["status"]
} for n in nodes}
report["node_count"] = len(nodes)
report["nodes_online"] = sum(1 for n in nodes if n["status"] == "online")
report["pve_probe_status"] = "ok"
# ── VMs/CTs ──
resources = pve_get("/api2/json/cluster/resources")
vms = [r for r in resources if r.get("type") in ("qemu","lxc")]
if resources is None:
vms = []
report["resources_probe_status"] = "unreachable"
else:
vms = [r for r in resources if r.get("type") in ("qemu","lxc")]
report["resources_probe_status"] = "ok"
report["total_vms"] = len(vms)
report["running_vms"] = sum(1 for v in vms if v.get("status") == "running")
stopped = [v for v in vms if v.get("status") != "running"]
@@ -183,7 +201,7 @@ def collect():
("Authentik", "https://auth.sysloggh.net"),
("Zulip", "https://chat.sysloggh.net"),
("Pulse", "https://pulse.sysloggh.net"),
("Proxmox", "https://minipve.sysloggh.net"),
("Proxmox", "https://192.168.68.12:8006"),
("SearXNG", "http://192.168.68.7:8888"),
("Firecrawl", "http://192.168.68.7:3002/health"),
]
@@ -195,10 +213,11 @@ def collect():
# ── LiteLLM Specific Checks (from litellm-health prose contract) ──
report["litellm"] = {"checks": []}
# Check 1: LiteLLM aggregate health endpoint (router binds to 127.0.0.1, check via SSH)
health_unified = ssh(LITELLM_BACKEND, "curl -sf http://127.0.0.1:9000/health/unified -o /dev/null -w '%{http_code}' 2>/dev/null")
# Check 1: Fleet health via nginx /health/unified (301 -> /gpu/gpu-data served by gpu-monitor on .24:9100).
# Router (:9000) was decommissioned 2026-09-11; probing it was a guaranteed daily failure.
health_unified = ssh(LITELLM_BACKEND, "curl -s -o /dev/null -w '%{http_code}' --max-time 8 http://127.0.0.1/health/unified 2>/dev/null")
report["litellm"]["health_unified"] = health_unified or "000"
report["litellm"]["checks"].append({"name": "unified-health", "status": "pass" if health_unified == "200" else "fail", "code": health_unified or "000"})
report["litellm"]["checks"].append({"name": "fleet-health-via-nginx", "status": "pass" if health_unified in ("200", "301") else "fail", "code": health_unified or "000"})
# Check 2: Nginx-proxied internal endpoints
for path, name in [("/litellm/ui/", "nginx-ui"), ("/litellm/docs", "nginx-docs")]:
@@ -206,8 +225,11 @@ def collect():
report["litellm"]["checks"].append({"name": name, "status": "pass" if code == "200" else "fail", "code": code})
# Check 3: Docker container health for LiteLLM stack
expected_containers = ["harness-litellm", "harness-nginx", "harness-router",
"harness-postgres", "harness-redis", "harness-dashboard"]
expected_containers = ["harness-litellm", "harness-nginx", "harness-postgres",
"harness-redis", "harness-dashboard", "harness-grafana",
"harness-prometheus", "harness-alertmanager",
"harness-zulip-bridge", "harness-docker-stats",
"harness-pve-exporter"]
actual_names = [c["name"] for c in containers2]
report["litellm"]["expected_containers"] = expected_containers
report["litellm"]["missing_containers"] = [e for e in expected_containers if e not in actual_names]
@@ -247,11 +269,13 @@ def collect():
zulip_health = json.loads(health_body) if health_body else {}
except:
zulip_health = {}
report["zulip_ext"]["connected"] = zulip_health.get("connected", False)
report["zulip_ext"]["queue_id"] = zulip_health.get("queue_id")
report["zulip_ext"]["last_error"] = zulip_health.get("last_error")
report["zulip_ext"]["messages_processed"] = zulip_health.get("messages_processed", 0)
report["zulip_ext"]["retry_count"] = zulip_health.get("retry_count", 0)
# Live state is nested under 'zulip' key
zulip_state = zulip_health.get("zulip", {})
report["zulip_ext"]["connected"] = zulip_state.get("connected", False)
report["zulip_ext"]["queue_id"] = zulip_state.get("queue_id")
report["zulip_ext"]["last_error"] = zulip_state.get("last_error")
report["zulip_ext"]["messages_processed"] = zulip_state.get("messages_processed", 0)
report["zulip_ext"]["skipped"] = zulip_state.get("skipped", 0)
# Phase 2: PM2 process check
pm2_raw = subprocess.check_output(
@@ -289,8 +313,8 @@ def collect():
# Abiba (pi)
report["agents"]["abiba"] = {
"platform": "pi", "ct": 100, "ip": "192.168.68.24",
"zulip_connected": zulip_health.get("connected", False),
"zulip_processed": zulip_health.get("messages_processed", 0),
"zulip_connected": zulip_state.get("connected", False),
"zulip_processed": zulip_state.get("messages_processed", 0),
"pm2_status": pm2.get("status", "unknown"),
"pm2_restarts": pm2.get("restarts", "?"),
"pm2_uptime": pm2.get("uptime", "?"),
@@ -308,26 +332,13 @@ def collect():
"updated_at": "",
}
# Mumuni (CT 100, IP 192.168.68.24)
mumuni_state = ssh("192.168.68.24", "cat ~/.hermes/gateway_state.json 2>/dev/null")
mumuni_data = {}
try:
mumuni_data = json.loads(mumuni_state) if mumuni_state else {}
except:
mumuni_data = {}
mumuni_platforms = mumuni_data.get("platforms", {})
report["agents"]["mumuni"] = {
"platform": "hermes", "ct": 100, "ip": "192.168.68.24",
"gateway_state": mumuni_data.get("gateway_state", "unknown"),
"telegram_state": mumuni_platforms.get("telegram", {}).get("state", "unknown"),
"zulip_state": mumuni_platforms.get("zulip", {}).get("state", "not_installed"),
"email_state": mumuni_platforms.get("email", {}).get("state", "unknown"),
"hermes_version": "",
}
# Get Hermes version
ver = ssh("192.168.68.24", "hermes --version 2>/dev/null | head -1")
if ver:
report["agents"]["mumuni"]["hermes_version"] = ver.split("·")[0].replace("Hermes Agent ","").strip()
# Mumuni is deliberately absent from this digest: captain ruling 2026-09-10.
# She moved off this host onto her own container (kagentz CT 105 on minipve,
# 192.168.68.14, dedicated `hermes` user) and is monitored from her side. The
# former probe ssh'd to 192.168.68.24 for the decommissioned deployment's
# ~/.hermes/gateway_state.json, always read "unknown", and published a false
# "mumuni:unknown" line in the agent table and the gateway-unknown issue
# count of every digest. Do NOT re-add an .24 / gateway_state probe.
return report
@@ -419,7 +430,7 @@ th {{ color: #8b949e; font-weight: normal; }}
<div class="alert {'good' if not issues else 'bad' if any('🔴' in i for i in issues) else 'warn'}">
<p style="margin:0;font-size:16px"><b>{status}</b></p>
<p style="margin:4px 0 0 0;font-size:13px">
{r['node_count']} PVE nodes · {r['total_vms']} VMs/CTs · {r['running_vms']} running ·
Proxmox: {r.get('pve_probe_status', 'ok')} ({r['nodes_online']}/{r['node_count']}) · {r['total_vms']} VMs/CTs · {r['running_vms']} running ·
{r['docker_vm']['total'] + r['docker_syslog']['total'] + r['docker_netbird']['total']} containers ·
{len(r['endpoints'])} endpoints · {len(r.get('agents',{}))} agents
</p>
@@ -435,8 +446,10 @@ th {{ color: #8b949e; font-weight: normal; }}
# ── Quick Stats ──
html += '<div class="card"><h2>📊 Quick Stats</h2><div class="grid">'
pve_status_label = "unreachable" if r.get('pve_probe_status') == 'unreachable' else f"{r['nodes_online']}/{r['node_count']}"
pve_status_color = "red" if r.get('pve_probe_status') == 'unreachable' or r['nodes_online'] != r['node_count'] else "green"
stats = [
("PVE Nodes", f"{r['nodes_online']}/{r['node_count']}", "green" if r['nodes_online'] == r['node_count'] else "red"),
("PVE Nodes", pve_status_label, pve_status_color),
("VMs/CTs", f"{r['running_vms']}/{r['total_vms']}", "green" if r['running_vms'] == r['total_vms'] else "red"),
("Containers", f"{r['docker_vm']['running']}/{r['docker_vm']['total']}", "green" if r['docker_vm']['running'] == r['docker_vm']['total'] else "yellow"),
("LiteLLM Ctrs", f"{r['docker_syslog']['running']}/{r['docker_syslog']['total']}", "green" if r['docker_syslog']['running'] == r['docker_syslog']['total'] else "red"),
@@ -539,12 +552,6 @@ th {{ color: #8b949e; font-weight: normal; }}
zulip_state = "✅" if agent.get("zulip_state") == "connected" else ("❌" if agent.get("zulip_state") == "disconnected" else "⬜")
gateway = agent.get("gateway_state", "?")
processed = "DSH"
elif name == "mumuni":
zulip_state = "⬜" if agent.get("zulip_state") == "not_installed" else ("✅" if agent.get("zulip_state") == "connected" else "⬜")
gateway = agent.get("gateway_state", "?")
tg = "✅" if agent.get("telegram_state") == "connected" else "❌"
ver = agent.get("hermes_version", "")
processed = f"TG:{tg} v{ver}"
else:
zulip_state = "⬜"
gateway = agent.get("gateway_state", "?")
@@ -698,6 +705,6 @@ if __name__ == "__main__":
print(f" Zulip Ext: {'✅' if report.get('zulip_ext',{}).get('connected') else '❌'}")
print(f" LiteLLM: {sum(1 for c in report.get('litellm',{}).get('checks',[]) if c['status']=='pass')}/{len(report.get('litellm',{}).get('checks',[]))} checks pass")
agent_parts = []
for k,v in report.get('agents',{}).items():
agent_parts.append(f"{k}:{v.get('gateway_state',v.get('pm2_status','?'))}")
print(f" Agents: {', '.join(agent_parts)}")
for k,v in report.get('agents',{}).items():
agent_parts.append(f"{k}:{v.get('gateway_state',v.get('pm2_status','?'))}")
print(f" Agents: {', '.join(agent_parts)}")
+1 -2
View File
@@ -5,6 +5,7 @@
# Field positions (awk -F'│'): $7=pid $8=uptime $9=restarts $10=status
TELEGRAM_BOT_TOKEN="$(grep TELEGRAM_BOT_TOKEN /root/.pi/agent/extensions/telegram/.env 2>/dev/null | cut -d= -f2 || echo '')"
LOG="/root/pm2-self-heal.log"
TELEGRAM_CHAT_ID="5822977936"
notify_tg() {
@@ -16,8 +17,6 @@ notify_tg() {
-d "text=${msg}" \
-d "parse_mode=HTML" > /dev/null 2>&1 || true
}
ALERTS="${ALERTS}$msg"
}
# Log-only mode: replaced by prose contract pm2-self-heal.prose.md
# Only alerts Telegram on actual failure (status != online)
+6 -4
View File
@@ -66,11 +66,13 @@ The infrastructure-control.prose.md contract is the canonical reference for the
2. NO .19 IP — Zulip is CT 117 on storepve.
3. NO CT 122/123 — Tanko=CT 112, Mumuni=CT 100 (inside Abiba). No CT 114 anywhere.
4. Strix Halo :8080 is FIREWALLED to .116 only — cannot be probed from abiba (.24).
5. abiba-zulip PM2 process is DECOMMISSIONED (2026-07-04) — abiba uses Telegram only.
5. abiba-zulip PM2 process is ONLINE (verified 2026-09-11) — this rule was stale.
**Docker on CT 116 (8 containers):**
harness-litellm, harness-router, harness-nginx, harness-postgres,
harness-redis, harness-dashboard, harness-grafana, harness-prometheus
**Docker on CT 116 (11 containers, verified 2026-09-11):**
harness-litellm, harness-nginx, harness-postgres, harness-redis,
harness-dashboard, harness-grafana, harness-prometheus, harness-alertmanager,
harness-zulip-bridge, harness-docker-stats, harness-pve-exporter
(harness-router was decommissioned 2026-09-11)
## DIFF TO REVIEW
+14 -19
View File
@@ -1,7 +1,10 @@
#!/bin/bash
# /root/scripts/zulip-monitor.sh — Zulip Mesh Health Monitor
# Implements zulip-health.prose.md v2
# Implements zulip-health.prose.md v3
# Runs every 15 min via cron. Alerts: Zulip private DM to the owner plus a stream post to #agent-hub on topic 'zulip-health'.
# Legs: global Zulip server, Platform A pi/Abiba (the Zulip bridge), Platform B
# Tanko (DSH), Platform C Agent Zero (kagentz). The former Platform B Hermes
# agent leg is retired — see the note after the Tanko leg.
set -euo pipefail
ZULIP_SITE="https://chat.sysloggh.net"
@@ -21,7 +24,8 @@ notify() {
# Zulip DM to owner
local content="${severity} Zulip Monitor: ${msg}"
local form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
local form
form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-d "${form}" > /dev/null 2>&1 || true
@@ -141,23 +145,14 @@ else
esac
fi
# ── Platform B: Hermes (Mumuni) ──
MUMUNI_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.24 \
"cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}")
MUMUNI_ZULIP=$(echo "$MUMUNI_STATE" | python3 -c "
import sys,json
d=json.load(sys.stdin)
p=d.get('platforms',{}).get('zulip',{})
print(p.get('state','unknown'))
" 2>/dev/null)
if [ "$MUMUNI_ZULIP" != "connected" ]; then
notify "🔴" "Mumuni (Hermes) Zulip state: $MUMUNI_ZULIP"
ISSUES=$((ISSUES + 1))
echo " Mumuni: ❌ state=$MUMUNI_ZULIP" >> "$LOG"
else
echo " Mumuni: ✅ Zulip connected" >> "$LOG"
fi
# ── Removed: the former "Platform B: Hermes" agent leg ──
# Captain ruling 2026-09-10: that agent moved off this host onto her own
# container (kagentz CT 105 on minipve, dedicated `hermes` user) and is now
# monitored on her side — see the out-of-scope note in zulip-health.prose.md.
# The old leg ssh'd to her former CT 100 deployment and read its Hermes gateway
# state, which reported "unknown" on every run and posted a false 🔴 DM plus an
# #agent-hub stream alert. Do NOT re-add a probe for her: this monitor must
# never contact her former host.
# ── Platform C: Agent Zero (kagentz) ──
AZ_A2A=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \
+138
View File
@@ -0,0 +1,138 @@
"""
Regression tests for daily-infra-report.py fixes (PR #64).
Tests:
(a) Asserts the nested zulip read feeds the agent-card fields
(b) Asserts an unreachable pve_get renders labelled-unreachable, not "0/0"
"""
import json
import subprocess
import sys
from pathlib import Path
from unittest.mock import patch, MagicMock
# Add scripts to path
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
import importlib.util
def load_script():
"""Load the daily-infra-report script as a module."""
script_path = Path(__file__).parent.parent / "scripts" / "daily-infra-report.py"
spec = importlib.util.spec_from_file_location("daily_infra_report", script_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_nested_zulip_read_feeds_agent_card():
"""Test that Zulip state is read from the nested 'zulip' key and feeds agent-card fields."""
# Mock the http_get_body response with nested structure
mock_health_response = json.dumps({
"status": "ok",
"platform": "pi",
"agent": "abiba",
"zulip": {
"connected": True,
"queue_id": "test-queue-id",
"messages_processed": 42,
"skipped": 5,
"last_error": None
}
})
# Import and patch
report_mod = load_script()
with patch.object(report_mod, 'http_get_body', return_value=mock_health_response):
# Simulate the collect() function's Zulip section
zulip_health = json.loads(report_mod.http_get_body("http://localhost:9200/health"))
zulip_state = zulip_health.get("zulip", {})
# Assert the nested key is read correctly
assert zulip_state.get("connected") == True, "Zulip connected should be True from nested key"
assert zulip_state.get("messages_processed") == 42, "messages_processed should be 42 from nested key"
assert zulip_state.get("queue_id") == "test-queue-id", "queue_id should be read from nested key"
# Simulate the agent card field population
agent_card = {
"zulip_connected": zulip_state.get("connected", False),
"zulip_processed": zulip_state.get("messages_processed", 0),
}
assert agent_card["zulip_connected"] == True, "Agent card should show Zulip connected"
assert agent_card["zulip_processed"] == 42, "Agent card should show 42 processed messages"
def test_unreachable_pve_get_renders_labelled_unreachable():
"""Test that an unreachable PVE API renders 'unreachable' instead of '0/0'."""
# Import and patch
report_mod = load_script()
# Test pve_get returns None on error
with patch.object(report_mod.subprocess, 'run') as mock_run:
mock_run.return_value.returncode = 7 # Connection failure
result = report_mod.pve_get("/api2/json/nodes")
assert result is None, "pve_get should return None on connection failure"
# Test the render logic
report = {
"nodes": {},
"node_count": 0,
"nodes_online": 0,
"pve_probe_status": "unreachable",
"total_vms": 0,
"running_vms": 0,
}
# The render should show "unreachable" not "0/0"
pve_status_label = "unreachable" if report.get('pve_probe_status') == 'unreachable' else f"{report['nodes_online']}/{report['node_count']}"
assert pve_status_label == "unreachable", "PVE status should show 'unreachable' when probe fails, not '0/0'"
def test_unreachable_resources_renders_labelled_unreachable():
"""Test that unreachable resources probe renders 'unreachable' instead of '0/0'."""
report_mod = load_script()
# Test resources probe returns None
with patch.object(report_mod.subprocess, 'run') as mock_run:
mock_run.return_value.returncode = 7
result = report_mod.pve_get("/api2/json/cluster/resources")
assert result is None, "pve_get for resources should return None on connection failure"
# Test the render logic
report = {
"resources_probe_status": "unreachable",
"total_vms": 0,
"running_vms": 0,
}
resources_label = "unreachable" if report.get('resources_probe_status') == 'unreachable' else f"{report['running_vms']}/{report['total_vms']}"
assert resources_label == "unreachable", "Resources status should show 'unreachable' when probe fails, not '0/0'"
if __name__ == "__main__":
print("Running tests...")
try:
test_nested_zulip_read_feeds_agent_card()
print("✓ test_nested_zulip_read_feeds_agent_card passed")
except AssertionError as e:
print(f"✗ test_nested_zulip_read_feeds_agent_card failed: {e}")
sys.exit(1)
try:
test_unreachable_pve_get_renders_labelled_unreachable()
print("✓ test_unreachable_pve_get_renders_labelled_unreachable passed")
except AssertionError as e:
print(f"✗ test_unreachable_pve_get_renders_labelled_unreachable failed: {e}")
sys.exit(1)
try:
test_unreachable_resources_renders_labelled_unreachable()
print("✓ test_unreachable_resources_renders_labelled_unreachable passed")
except AssertionError as e:
print(f"✗ test_unreachable_resources_renders_labelled_unreachable failed: {e}")
sys.exit(1)
print("All tests passed!")
+352
View File
@@ -0,0 +1,352 @@
"""Regression tests for the 2026-09-10 retirement of the Mumuni monitoring leg.
WHY THIS FILE EXISTS: captain ruling 2026-09-10 — Mumuni moved off this host
onto her own container (kagentz CT 105 on minipve, 192.168.68.14, dedicated
`hermes` user) and is monitored from her side. The monitor nevertheless kept
ssh'ing to root@192.168.68.24 for `~/.hermes/gateway_state.json` on the
decommissioned deployment, read "unknown" on every run, and posted a false 🔴
"Mumuni (Hermes) Zulip state: unknown" DM + #agent-hub stream alert to the
captain. The daily infra digest published a matching `mumuni:unknown` row.
CONTRACT UNDER TEST:
* `scripts/zulip-monitor.sh` carries NO Mumuni probe and NO 192.168.68.24
reference; it never ssh'es .24, and even on a failing run it emits no Mumuni
notify (stdout alert, Zulip payload, or log line).
* The Abiba (pi — the Zulip bridge), Tanko (DSH) and Agent Zero (kagentz) legs
still work: deleting the Mumuni leg must not have gutted the rest.
* `scripts/daily-infra-report.py` no longer probes .24 for a Hermes gateway
state and no longer emits a `mumuni` agent entry.
* `scripts/agent-health-check.py`'s AGENTS roster has no mumuni entry. This is
a pin, not a behavior change — verify the probe was already gone.
* `zulip-health.prose.md` retires the Mumuni-only steps and says explicitly
that Mumuni is not monitored from this host.
HOW: behavioral execution plus one named deliverable-text contract. The sandbox
copies the shipped monitor verbatim and rewrites only its LOG constant, then
runs it with stub ssh/curl on PATH; the ssh stub records every host it is asked
to reach, so "never probes .24" and "no Mumuni notify" are asserted from
observed behavior. The daily digest is pinned by importing it and exercising
collect() and build_html() directly. The single source-text assertion is the
deliverable-text contract the captain acceptance names for the shipped monitor.
Usage: python3 -m pytest tests/test_mumuni_monitor_removal.py
"""
from __future__ import annotations
import importlib.util
import os
import pathlib
import stat
import subprocess
import pytest
ROOT = pathlib.Path(__file__).resolve().parents[1]
ZULIP_MONITOR = ROOT / "scripts" / "zulip-monitor.sh"
DAILY_REPORT = ROOT / "scripts" / "daily-infra-report.py"
AHC = ROOT / "scripts" / "agent-health-check.py"
HEALTH_CONTRACT = ROOT / "zulip-health.prose.md"
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
MUMUNI_IP = "192.168.68.24" # Mumuni's old (decommissioned) deployment
TANKO_VANTAGE = "192.168.68.15" # amdpve — Tanko CT 112 via pct exec
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
# ── scripts/zulip-monitor.sh: deliverable-text contract ─────────────
def test_zulip_monitor_deliverable_text_contract():
"""Owned deliverable-text contract for scripts/zulip-monitor.sh.
Captain acceptance requires the shipped monitor to contain no Mumuni probe
identifier and no 192.168.68.24 literal. Behavioral proof that the monitor
never contacts that host and never emits a Mumuni notify lives in the
sandbox tests below; this only pins the named text contract.
"""
text = ZULIP_MONITOR.read_text()
assert "mumuni" not in text.lower()
assert MUMUNI_IP not in text
# ── scripts/zulip-monitor.sh: behavioral sandbox ─────────────────────
SSH_STUB = r"""#!/usr/bin/env bash
# Stub ssh: record the target host, then answer by host + remote command.
printf '%s\n' "$*" >> "$RECORD_DIR/ssh.calls"
host=""
for a in "$@"; do
case "$a" in
*@192.168.*) host="${a##*@}" ;;
esac
done
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
cmd="${*: -1}"
case "$host" in
192.168.68.15)
case "$cmd" in
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
*curl*) printf '%s' "$TANKO_HTTP" ;;
esac ;;
192.168.68.14)
case "$cmd" in
*agent.json*) printf '%s' "$AZ_A2A" ;;
*"ps aux"*) printf '%s\n' "$AZ_PS" ;;
esac ;;
*)
printf 'UNEXPECTED-SSH-HOST %s\n' "$host" >> "$RECORD_DIR/unexpected-ssh" ;;
esac
exit 0
"""
CURL_STUB = r"""#!/usr/bin/env bash
# Stub curl: serve the Abiba health fixture and the Zulip server 200, and
# record every call (including notify) payloads.
printf '%s\n' "$*" >> "$RECORD_DIR/curl.calls"
case "$*" in
*:9200/health*)
case " $* " in
*" -w "*) printf '%s' "$PI_HTTP" ;; # -w '%{http_code}' probe
*) printf '%s' "$PI_BODY" ;; # body probe
esac ;;
*server_settings*)
printf '%s' "$SERVER_HTTP" ;;
esac
exit 0
"""
def _write_exec(path: pathlib.Path, body: str) -> None:
path.write_text(body)
path.chmod(path.stat().st_mode
| stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200",
az_a2a='{"name":"kagentz"}',
az_ps="root 111 0.1 0.2 /opt/venv-a0/bin/python3 -u adapter.py"):
"""Run the shipped monitor in a sandbox; return (proc, record_dir, log_path).
Only the LOG constant is rewritten (to keep the run inside the worktree).
Everything else — legs, labels, notify logic — is the shipped script.
"""
sandbox = tmp_path / "sandbox"
bindir = sandbox / "bin"
record = sandbox / "record"
bindir.mkdir(parents=True)
record.mkdir()
_write_exec(bindir / "ssh", SSH_STUB)
_write_exec(bindir / "curl", CURL_STUB)
source = ZULIP_MONITOR.read_text()
log_line = 'LOG="/root/zulip-health-monitor.log"'
assert log_line in source, "LOG constant moved — update the sandbox harness"
log_path = sandbox / "zulip-health-monitor.log"
script = sandbox / "zulip-monitor.sh"
script.write_text(source.replace(log_line, f'LOG="{log_path}"'))
env = dict(os.environ)
env.update({
"PATH": f"{bindir}:{env['PATH']}",
"RECORD_DIR": str(record),
"TANKO_SVC": tanko_svc,
"TANKO_HTTP": tanko_http,
"AZ_A2A": az_a2a,
"AZ_PS": az_ps,
"PI_HTTP": "200",
"PI_BODY": CONNECTED_FIXTURE.read_text(),
"SERVER_HTTP": "200",
})
proc = subprocess.run(["bash", str(script)], cwd=sandbox, env=env,
capture_output=True, text=True)
return proc, record, log_path
def test_healthy_run_is_quiet_and_never_reaches_mumuni(tmp_path):
proc, record, log_path = _run_monitor(tmp_path)
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# Every retained leg actually ran and passed.
assert "Server: ✅ HTTP 200" in log
assert "Abiba: ✅ Connected" in log
assert "Tanko: ✅ service=active http=200" in log
assert "kagentz: ✅ A2A alive" in log
assert "kagentz: ✅ Adapter running" in log
assert "Result: ✅ All healthy" in log
# A healthy run emits no notify at all — and certainly no Mumuni one.
assert proc.stdout == ""
assert "Mumuni" not in log
assert "🔴" not in log
# Observed behavior: .24 is never resolved, only Tanko's vantage and the
# Agent Zero host are contacted.
hosts = record.joinpath("ssh.hosts").read_text().split()
assert MUMUNI_IP not in hosts
assert set(hosts) == {TANKO_VANTAGE, AGENT_ZERO_HOST}
assert not record.joinpath("unexpected-ssh").exists()
def test_failing_run_alerts_on_tanko_but_never_on_mumuni(tmp_path):
# Failure path: exercises notify() end to end so "no Mumuni notify" is
# proven on the alert path, not only on the quiet healthy path.
proc, record, log_path = _run_monitor(tmp_path, tanko_svc="inactive",
tanko_http="000")
assert proc.returncode == 0, proc.stderr
alerts = proc.stdout
assert "Tanko (DSH dsh-web) service state: inactive" in alerts
assert "1 issue(s) found" in alerts
# No Mumuni text in stdout, the log, or any Zulip DM/stream payload.
assert "Mumuni" not in alerts
assert "Mumuni" not in log_path.read_text()
assert MUMUNI_IP not in alerts + log_path.read_text()
payloads = record.joinpath("curl.calls").read_text()
assert "Mumuni" not in payloads
assert MUMUNI_IP not in payloads
# The rest of the monitor still ran alongside the failing Tanko leg.
log = log_path.read_text()
assert "Abiba: ✅ Connected" in log
assert "kagentz: ✅ A2A alive" in log
assert "Result: 🔴 1 issue(s) found" in log
# ── scripts/daily-infra-report.py: behavioral digest checks ──────────
@pytest.fixture(scope="module")
def daily():
spec = importlib.util.spec_from_file_location("daily_infra_report", DAILY_REPORT)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
DAILY_AGENTS = {
"abiba": {
"platform": "pi", "ct": 100, "ip": MUMUNI_IP,
"zulip_connected": True, "zulip_processed": 5,
"pm2_status": "online", "pm2_restarts": "0", "pm2_uptime": "1h",
},
"tanko": {
"platform": "dsh", "ct": 112, "ip": "192.168.68.122",
"gateway_state": "n/a (DSH)", "zulip_state": "connected",
"telegram_state": "unknown", "gateway_pid": None, "updated_at": "",
},
}
def _fabricated_report(agents):
return {
"nodes": {},
"node_count": 1,
"nodes_online": 1,
"total_vms": 0,
"running_vms": 0,
"stopped_vms": [],
"vms_by_node": {n: [] for n in
["amdpve", "minipve", "storepve", "acerpve", "ocupve"]},
"storage": [],
"docker_vm": {"total": 0, "running": 0, "unhealthy": [],
"containers": [], "reclaimable": "", "disk_used": "1%"},
"docker_syslog": {"total": 0, "running": 0, "containers": []},
"docker_netbird": {"total": 0, "running": 0, "containers": []},
"endpoints": [],
"litellm": {"checks": []},
"nfs": [],
"zulip_ext": {
"connected": True, "queue_id": "queue", "last_error": None,
"messages_processed": 0, "retry_count": 0, "pm2": {},
"pm2_healthy": True, "bot_skipped_15min": 0, "finalized_1h": 0,
"failed_finalize_1h": 0, "finalize_fail_pct": 0,
"server_status": "200",
},
"agents": agents,
}
def _agent_status_card(html):
start = html.index("🤖 Agent Status")
end = html.index("💬 Zulip Extension")
return html[start:end]
def test_daily_report_renders_only_abiba_and_tanko_agents(daily):
"""build_html() over a Mumuni-free agent set must render no Mumuni row and
no Mumuni gateway-unknown issue, while abiba and tanko rows still render."""
html = daily.build_html(_fabricated_report(dict(DAILY_AGENTS)))
card = _agent_status_card(html)
assert "mumuni" not in card.lower()
assert "abiba" in card
assert "tanko" in card
assert "mumuni" not in html.lower()
def test_daily_report_collect_never_probes_mumuni(monkeypatch, daily):
"""collect() with ssh stubbed must add no mumuni agent and must never ssh
its decommissioned .24 host."""
probed = []
class _NoSubprocess:
@staticmethod
def check_output(*args, **kwargs):
return b""
def fake_ssh(host, cmd):
probed.append(host)
return ""
monkeypatch.setattr(daily, "pve_get", lambda path: [])
monkeypatch.setattr(daily, "ssh_jerome", lambda host, cmd: "")
monkeypatch.setattr(daily, "ssh", fake_ssh)
monkeypatch.setattr(daily, "http_get",
lambda url, auth=None, timeout=10: "200")
monkeypatch.setattr(daily, "http_get_body",
lambda url, auth=None, timeout=10: "")
monkeypatch.setattr(daily, "count_in_log", lambda *a, **k: 0)
monkeypatch.setattr(daily, "subprocess", _NoSubprocess)
report = daily.collect()
assert "mumuni" not in report["agents"]
assert MUMUNI_IP not in probed
# ── scripts/agent-health-check.py: roster pin ───────────────────────
@pytest.fixture(scope="module")
def ahc():
spec = importlib.util.spec_from_file_location("agent_health_check_roster", AHC)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_agent_health_roster_has_no_mumuni_entry(ahc):
assert "mumuni" not in ahc.AGENTS
# ── zulip-health.prose.md: contract reconciliation ──────────────────
def test_health_contract_retires_mumuni_only_steps():
text = HEALTH_CONTRACT.read_text()
assert MUMUNI_IP not in text
for step in ("**B4:", "**B5:", "**B6:"):
assert step not in text
def test_health_contract_states_mumuni_is_not_monitored_from_this_host():
text = HEALTH_CONTRACT.read_text()
assert "Mumuni is NOT monitored from this host" in text
assert "monitored on her side" in text
assert "her own container" in text
def test_health_contract_keeps_tanko_agent_zero_and_bridge_steps():
text = HEALTH_CONTRACT.read_text()
for marker in ("**B1:", "**B2:", "**B3:", "Step 4: Platform C",
"Step 2: Platform A", "Step 1: Zulip Server Liveness"):
assert marker in text, marker
+25 -43
View File
@@ -1,9 +1,9 @@
---
kind: responsibility
name: zulip-health
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Tanko on DSH / Mumuni on Hermes), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (pi/Abiba Zulip bridge), Platform B (Tanko on DSH), and Platform C (Agent Zero Docker). Verifies bot registration, DM delivery, and cross-platform connectivity. Mumuni is no longer monitored from this host — she runs on her own container (kagentz CT 105 on minipve, .14) and is monitored on her side.
title: Zulip Mesh Health Monitor — Multi-Platform
version: 3.0.0
version: 3.1.0
runtime_contract: 2
agent: abiba
report_only_agents:
@@ -12,13 +12,23 @@ report_only_agents:
# Zulip Mesh Health Monitor
Monitors ALL Zulip-connected agents across platforms (pi, Hermes, DSH, Agent Zero).
Runs every 15 minutes in the background. Also triggers on session start.
Monitors the Zulip-connected agents under this host's operational control (pi,
DSH, Agent Zero). Runs every 15 minutes in the background. Also triggers on
session start.
> **Mumuni is NOT monitored from this host (captain ruling 2026-09-10).** She
> moved off this host onto her own container — kagentz CT 105 on minipve
> (192.168.68.14), running a dedicated `hermes` user — and is monitored on her
> side. No step in this contract, and no leg of `scripts/zulip-monitor.sh`, may
> ssh to her old deployment, read her `~/.hermes/gateway_state.json`, or alert on
> her state. The former Platform-B-for-Mumuni steps (gateway process, heartbeat,
> response delivery) are retired: they always read "unknown" against the
> decommissioned deployment and produced a false 🔴 alert on every run.
## Requires
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); Mumuni (192.168.68.14, kagentz CT105 on minipve); and Agent Zero Docker host (192.168.68.14)
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
- **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -81,13 +91,13 @@ Log as "unreachable" — don't treat as critical unless it persists for 3+ conse
## Streaming Support (2026-07-05)
Zulip agents now support progressive message editing during agent generation.
When a Zulip agent (Tanko on DSH, Mumuni on Hermes) processes a message, the response is
When a Zulip agent under this monitor's scope (Tanko on DSH) processes a message, the response is
streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
- Adapter implements `edit_message()` using `_api_patch()` helper
- Gateway stream consumer progressively edits the Zulip message
- User sees real-time agent thinking instead of waiting for full response
- Verified: Tanko (CT 112) and Mumuni (kagentz CT 105) both have streaming active
- Verified: Tanko (CT 112) has streaming active; Mumuni's (kagentz CT 105) is verified on her own host, not from here
### Verification
```bash
@@ -183,7 +193,10 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
| Crash loop >10/h | Alert user |
### Step 3: Platform B — Tanko (DSH on amdpve CT 112) & Mumuni (Hermes)
### Step 3: Platform B — Tanko (DSH on amdpve CT 112)
Mumuni is out of scope for this host (see the note above): she runs on her own
container and is monitored on her side.
Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
@@ -240,44 +253,13 @@ timeout only. Never expect a bare `200` — the public URL terminates in the
token-gated authentik chain. Statuses outside the healthy set are
logged/reported as a warning — reported, never healed on.
**B4: Gateway Process** (Hermes agent Mumuni only — Tanko runs no Hermes gateway)
```bash
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
```
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more
than one `gateway run` process is found, the gateway has a collision (typically
one `--force` and one `--replace` process). Kill the newer/duplicate process,
then restart the remaining gateway per-agent (parameterized 2026-08-09, captain
ruling). Check the gateway log for "Gateway running with 2 platform(s)" (not 1)
to confirm Zulip reloaded.
**B5: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
```bash
ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
```
Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
Silence > 300s → warning. Silence > 600s → critical.
**B6: Response Delivery** (Hermes agent Mumuni only)
```bash
ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
```
> 50% fail rate → critical.
**Platform B Actions**
| Condition | Action |
|-----------|--------|
| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` (Mumuni) / restart Tanko via DSH service |
| No heartbeat in 10min | Same as above |
| `Failed to finalize` > 50% | Check PATCH API, Zulip server |
| Response empty/short | Check A2A endpoint / LiteLLM model |
| `dsh-web` service not `active` | Restart Tanko via DSH service |
| HTTP `:3080` connection refused/timeout (`000`) | Same as above |
| HTTP status outside the expected set | Log/report as a warning — reported, never healed on |
### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
@@ -333,7 +315,7 @@ Expected: task ID with "working" status. Poll for completion with `tasks/get`. I
Check each agent's log for excessive bot-to-bot chatter:
- Abiba: `Skipped.*bot msgs` count
- Tanko/Mumuni: Repeated DM exchanges between bots
- Tanko: Repeated DM exchanges between bots
- kagentz: Adapter log for bot DMs being processed
If any bot processes >50 bot-originated messages in 15min → warning.