Commit Graph
29 Commits
Author SHA1 Message Date
root 88e9243ce4 fix(alignment): clarify tanko's live LiteLLM proxy status in health-check description 2026-09-28 12:38:42 +00:00
root 9a789ab76d fix(hermes): separate policy observations from fault findings
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s
Contract design defect: 'uses a non-harness provider' (POLICY) and
'cannot authenticate' (FAULT) were printed as the same violation class.
A policy observation must never be phrased as if the agent were broken.

Changes:
1. Added 'Violation Classification' section to all three contracts
2. Separated POLICY (observation only) from FAULT (requires request-level evidence)
3. Rules:
   - Do NOT infer runtime credential resolution from config text alone
   - Require request-level evidence before calling a FAULT: observed auth failure
     or absence of successful calls
   - If calls are succeeding, output is 'POLICY: uses <provider> directly; calls
     succeeding' - not a violation
   - State what you OBSERVED, not what the field implies

Files changed (3):
- hermes-key-enforcement.prose.md
- hermes-config-template.prose.md
- hermes-agent-baseline.prose.md
2026-09-14 12:32:06 +00:00
root 4e34b7a2a2 fix(hermes): wire all three contracts to reachability helper
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
The helper was correct but dead code - nothing called it. This commit:
1. Makes the helper runnable standalone: scripts/hermes-reachability-check.sh <host> <pattern> <path>
2. Wires all THREE contracts to it:
   - hermes-key-enforcement.prose.md
   - hermes-config-template.prose.md
   - hermes-agent-baseline.prose.md
3. Each contract now explicitly instructs to run the helper and interpret the three outcomes
4. States that the bug this replaces was deriving reachability from the remote grep's exit code

Files changed (4):
- scripts/hermes-reachability-check.sh (standalone mode added)
- hermes-key-enforcement.prose.md (reachability section added)
- hermes-config-template.prose.md (reachability section added)
- hermes-agent-baseline.prose.md (reachability section added)
2026-09-14 12:01:32 +00:00
root bd0065bb31 no-mistakes(review): Fix monitor-key scope, Strix context, retired delegation names 2026-09-12 22:02:58 +00:00
root 8210fd905c fix: align contracts to 4-name LiteLLM registry (2026-09-12)
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
- Remove retired names (qwen3.6-27B-code, qwen3.6-35B-udq4) from live alias claims
- Update Strix Halo model to Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf (strix-moe, 256K ctx)
- Fix litellm-health step 7 probe to gpu-vision (monitor key scoped)
- Move qwen3.6-27B-code/35B-udq4 from raw-but-live to non-resolving in audit
- Fold in pm2-self-heal: remove spoton-service (live PM2 set is 4/4)
- Update hermes templates, key enforcement, timeout tables to live names
2026-09-12 21:39:54 +00:00
root de1428b4ae no-mistakes(review): Harden GC gate key aliases, fail closed, fix baseline 2026-09-12 18:42:36 +00:00
root 9e581ab203 no-mistakes(review): Complete retired-alias field coverage; fix misleading example labels 2026-09-12 16:26:14 +00:00
root 9cac3589cf no-mistakes(review): Fix compression example alias; make retired aliases fail audit 2026-09-12 16:19:43 +00:00
abiba 221f9f79f3 fix(audit): reject retired aliases; sweep gpu-light/gemma-4-12b to gpu-vision
audit-hermes-config.py Rule 8 required auxiliary.vision.model and
auxiliary.web_extract.model to equal the retired 'gpu-light', so a config
adopting the live canonical 'gpu-vision' FAILED our own audit - the audit was
enforcing a dead alias (400 Invalid model name). Rule 8 now requires
gpu-vision; retired names gpu-light/crew-auto join the raw-name rejection set;
the guidance message names the live aliases.

Sweep of the remaining references: gpu-self-heal stops canonicalizing
gpu-light; hermes-config-template, hermes-agent-baseline, hermes-key-enforcement,
inference-optimization, litellm-client-timeouts and gpu-fleet now use the live
gpu-vision alias. Where a file restated model/rpm/weight/fallback state it now
points at CT 116 /opt/inference-harness/litellm_config.yaml instead of
duplicating it. koby's .129 config is report-only and recorded, not edited.

Adds tests/test_audit_hermes_config_alias.py: executes the audit CLI and asserts
gpu-vision passes while gpu-light and gemma-4-12b fail.
2026-09-12 16:11:46 +00:00
agent-zero c26255f5ff fix(router): decommission legacy GPU router across contracts and fleet scripts
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Router container, image, and config are purged on CT 116 (verified: no
container, no image, inference-harness-router:latest removed, :9000 free,
11 containers healthy, 7 models, live syslog-auto completion OK). Updates:

- gpu-fleet.prose.md: topology diagram rebuilt without the router tier
- gpu-monitor / gpu-self-heal / litellm-health / litellm-self-heal: router steps dropped
- infrastructure-control.prose.md: container inventory + litellm row corrected
- scripts/daily-infra-report.py, scripts/prose-ai-review.sh: host-scoped checks

Verified: 40/40 anchors matched uniquely, 0 U+FFFD across changed files,
daily-infra-report.py compiles, no stray harness-router references remain.
2026-09-11 14:20:18 -04:00
root 2dfc3e1530 Merged PR #50: fix/gpu-dense-docs
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-08-30 12:16:39 +00:00
root 79af0ae7a3 Merged PR #52: fix/tanko-runtime 2026-08-30 12:16:28 +00:00
mumuni-bot e898048baf fix: repoint Mumuni refs from CT100/.24 to kagentz CT105/.14 post-migration
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Mumuni migrated off Abiba CT100 (192.168.68.24, purged) to dedicated
CT kagentz (192.168.68.14) on minipve. Verified live on kagentz:
- Gateway: systemd unit hermes-gateway.service (User=hermes), active
- Hermes home: /home/hermes/.hermes
- gateway_state.json present at ~/.hermes/gateway_state.json

Updates Mumuni rows/references in 9 contracts. Abiba-owned refs
(.24 pi agent, GPU monitor, infrastructure-control CRITICAL file)
and historical runs/ logs intentionally left unchanged.

Per relay #738 follow-ups. Refs #735-#738.
2026-08-29 21:41:07 +00:00
root 994ae1b7ac fix: standardize base URLs to authenticated /litellm/v1 path in remaining contracts
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
- Update hermes-agent-baseline.prose.md: change /v1 → /litellm/v1 in all base_url references
- Update hermes-key-enforcement.prose.md: change workaround section base_url to /litellm/v1
- Ensures all contracts consistently use the authenticated LiteLLM path
- DeepSeek harness exemption remains documented for external providers
2026-08-28 15:32:24 +00:00
kagentz-bot 44f7008302 docs: reflect hwepve removal from Tabiri cluster (5 nodes + standalone London node)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
- Remove hwepve from cluster member lists/diagrams in infrastructure-control,
  proxmox-monitor, infrastructure-update, litellm-health, mumuni-delegation
- CTs 100 (abiba) and 105 (kagentz) moved to minipve; CT 114 (mumuni) no longer
  exists (Mumuni runs inside Abiba CT 100)
- Add standalone London role + NetBird routing peer description for hwepve
- Update scripts: agent-health-check.py, pct-run.sh, prose-ai-review.sh
- Update disk-gc CT access table, hermes baselines/restore host references
2026-08-15 20:03:02 -04:00
root b3193e5e1b fix(agent-health): 14 stale-reference fixes (Mumuni CT100/.24, Koby .129, Koonimo CT113, wrapper paths)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-01 14:10:52 +00:00
root 2ace79fcab no-mistakes(document): Update 5→6 node references and fix CT ID contradictions
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-07-24 21:36:38 +00:00
root 767ab25128 merge: resolve PR #26 conflict — keep qwen model ref + add hwepve node
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-07-23 18:10:18 +00:00
root 1c44bf1259 contract updates: ornith decommissioned, 256K→128K context, Mumuni Discord disabled
Change 1: Strix Halo — ornith decommissioned
- gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout
- inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4
- gpu-monitor: ornith status → Strix Halo status
- infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4
- infrastructure-update: ornith→strix-moe via router
- proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe)

Change 2: GPU context 256K→128K fleet-wide
- hermes-agent-baseline: frontmatter description updated
- litellm-health: GPU Fleet Topology table 256K→128K
- litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert
- inference-optimization: compress threshold 256K→128K compact at 85K
- gpu-fleet: instability note updated

Change 3: Mumuni Discord platform disabled
- gpu-fleet: Mumuni platforms: removed discord
2026-07-23 09:00:42 +00:00
root 9f0e04f22c fix: update contracts for hwepve 6th node + Mumuni migration (lxc/114 on hwepve)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
- proxmox-monitor: 5→6 nodes, add hwepve (.4, Huawei Matebook 16), note Mumuni (lxc/114) placement
- zulip-health: add dual-gateway collision detection to B2, note Mumuni on hwepve
- hermes-agent-baseline: fix Mumuni node minipve→hwepve
- hermes-zulip-restore: add hwepve Proxmox column for Mumuni
- mumuni-delegation: fix CT 118/storepve/.6 → lxc/114/hwepve/.123, 5→6 nodes
- pm2-self-heal: verified — no Mumuni references, no changes needed

Post-migration discoveries:
- Mumuni container migrated from minipve to hwepve (6th node, 192.168.68.4)
- IP preserved (192.168.68.123), hostname 'mumuni' unchanged
- Dual gateway collision (PID 1442 --replace + PID 2924 --force) resolved by
  container restart at 10:07; gateway restarted clean at 13:06 with fresh Zulip
  queue (errors=0, reconnects=0)
2026-07-20 17:08:58 +00:00
root 17a77e6b3f fix: fleet config issues from 2026-07-18 relay review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- hermes-agent-baseline: add gpu-dense and gpu-light to models list
- hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add
  vault empty-key guard documentation in Rule 13
- litellm-api-keys: fix pgrep pattern in auditable check
- scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper

Addresses issues found during relay inbox resolution session:
1. pgrep -f 'hermes_cli.main gateway run' matches both the real python
   gateway and the infisical bash wrapper, causing false health readings
2. infisical vault stores empty key silently — no guard/monitoring
3. gpu-dense/gpu-light stable aliases missing from baseline config
2026-07-18 17:39:46 +00:00
root 8215b84f88 merge: resolve conflicts with master (delegation delete + gpu-fleet)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
2026-07-16 21:22:34 +00:00
root b7e23e2592 vault: Infisical cleanup + contract sync (WAL #1316)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
- litellm-api-keys: vault audit, tanko/koby/koonimo migration, key rotation log
- gpu-fleet: ornith-1.0-35b→strix-moe (remaining refs)
- hermes-agent-baseline: 256K all GPUs, CT IPs updated, Shumba retired
- delegation-prose-contract: removed (renamed to mumuni-delegation)
- contract-registry.yaml + cron-prompts-review.md: from feat/contract-registry
2026-07-16 21:17:09 +00:00
Abiba dc572889f8 contracts: sync to ground truth — ornith-1.0-35b→strix-moe, 256K all GPUs, real LiteLLM timeouts
Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts:
- AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist
- All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1)
- LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s)
- Added LiteLLM model surface + key scoping to litellm-self-heal
- Patched health-check script path ref

Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal,
zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline,
delegation-prose-contract, mumuni-delegation-prose-contract
2026-07-16 17:02:44 +00:00
root 79d4a73895 docs: lessons learned from 2026-07-12 session
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
2026-07-12 22:49:39 +00:00
root 776df418ca gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3)
- .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache
- .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV
- .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm
- LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s
- Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma
- Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
2026-07-08 17:25:42 +00:00
jerome f6e8791352 fix(ci): change kind from reference to template in hermes-agent-baseline.prose.md
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
The validate job was failing because 'reference' is not an allowed kind.
Changed to 'template' since this document serves as a canonical baseline
template for restoring agent configurations.
2026-07-07 20:57:48 -04:00
root 5aa93117de docs: capture all session changes — port conflict, health check, streaming
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped
Updated contracts to reflect all infrastructure changes made 2026-07-05/06:

gpu-fleet.prose.md:
  - Port conflict detection on all 3 GPU wrappers
  - Ghost process root cause documented (.8 stale pid 25836)

infrastructure-control.prose.md:
  - Section 7: Agent Health Check (consolidated, non-disruptive)
  - Disabled scripts documented (zulip-watchdog, zulip-monitor)

hermes-agent-baseline.prose.md:
  - Health verification section + GPU port conflict detection
  - Change log updated

zulip-health.prose.md:
  - Streaming support section (edit_message + PATCH API)
2026-07-06 02:46:41 +00:00
root 2ea9cac23a docs: hermes-agent-baseline — canonical good-state reference
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped
Single source of truth for all 4 Hermes agents. Captures:
- Agent map (CT, IP, key, alias)
- Key architecture and config patterns
- Mandatory api_key workaround with explanation
- Known bug: auxiliary_client ignores api_key_env
- Audit procedure (full audit, leak check, key verification)
- Systemd patterns
- Attachment cache locations
2026-07-05 23:37:56 +00:00