- New 'Operating Doctrine (read first)' section with the .117 rule, safe-mutate wrapper usage, field trust taxonomy, and contract-native verify-before-fix flow
- Added hermes-key-enforcement to templates table
- Added stirling-pdf-agent-access to functions table
- Noted VERIFY-BEFORE-USE labels on infrastructure-control
- Added FIELD TRUST convention to frontmatter (live-state vs policy fields)
- Marked credentials/IPs with VERIFY-BEFORE-USE labels
- Fixed 3 stale Zulip IP references (.117 → .19) that caused the 2026-07-03 outage
- Added verify-before-mutate skill reference
- See knowledge graph node #604 for post-mortem
- hermes-key-enforcement.prose.md: NEW contract enforcing api_key_env for all harness/LiteLLM providers, exempting external providers
- hermes-config-template.prose.md: Updated with full fix log (15 total fixes across Koby/Koonimo/Mumuni/Tanko), model:auto detection rule, key-to-agent mapping
- infrastructure-control.prose.md: Added Stirling-PDF service details (port 8989, credentials, API key, Swagger URL) replacing Bento-PDF
- REMOVED hardcoded Agent Keys table (keys became stale after LiteLLM downgrade).
- ADDED Key Management section: verify loop, rotate + deploy procedure, key
deployment rules (plaintext in /etc/environment only, api_key_env pattern),
current aliases (2026-07-02 rotation).
- LiteLLM DB is the single source of truth for keys; contracts never store plaintext.
- Mumuni sub-agent profiles documented (6, all inherit auth from main config).
- infrastructure-control: add Section 5.3 (Network Verification & Routing) with
dual-path checks, NetBird ingress health, DNS drift detection, and routing
regression alerts. Add Section 7 (Configuration Doctrine — IP-First) mandating
LAN IPs for all configs/agents, URLs reserved for human browser access only.
Also enrich access matrix (Mumuni, Koonimo, LiteLLM Admin, Grafana entries),
reachability matrix, and CT 116 nginx routing + Prometheus targets.
- build-zulip-plugin: fold Success Criteria into Maintains postconditions.
- pi-approval-architecture: new reference doc — pi approval model vs Hermes,
architectural limits, /approve and /deny stub commands.
- zulip-approval-fix: new responsibility — documents the Zulip HTML/prefix bug
that broke /approve and /deny, and the one-line fix applied.
Runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min,
pushing Tctl to 98°C (9°C past critical) and throttling 70->29 t/s.
Server-side -n 8192 hard cap now bounds generation. Documented connection
source (all from .116/LiteLLM) and the thermal constraint (fanless APU).
Revert the /grafana/ nginx sub-path I wrongly added — it broke the existing
http://192.168.68.116:3001/d/gpu-fleet URL. Restored direct 0.0.0.0:3001 LAN
access. Contract is now the source of truth: no GF_SERVER_SERVE_FROM_SUB_PATH,
no nginx /grafana/ route. gpu-fleet + 3 new dashboards all at :3001.
- gpu-fleet: documented parallel=2/batch=4096 optimizations for RTX 3090/5070
- gpu-fleet: added Strix Halo ROCm status (installed, blocked by HSA runtime on Debian 13)
- gpu-fleet: documented alert migration from DM → #agent-hub stream topics
- gpu-monitor: added Alert Delivery section with stream topic conventions
- All GPU alerts now go to #agent-hub for cross-agent visibility
- Rate limiting during reconnection (429): honor retry-after headers
- Placeholder edit fails silently: fallback to sending as new message
- Both fixes deployed to pi Zulip extension
- Polls GPU sidecars every 15s for temp, VRAM, power, utilization
- Creates responsive HTML dashboard at /root/dashboard/gpu-fleet.html
- Served via nginx at /gpu/ on CT 116
- PM2-managed gpu-monitor server on port 9100
- Tracks Strix Halo llama-server via SSH
- Exposes JSON API at /gpu-data for integrations
Documents lessons from building pi Zulip extension and Hermes Zulip plugin:
1. Queue registration content-type
2. Missing bot_user_id for @mentions
3. Zero stream subscriptions
4. Stale errors never cleared
5. Stuck detection too aggressive
6. Env var name mismatch
7. Poll interval too aggressive
8. A2A port conflict
Also: fixed Hermes adapter to clear stale errors on successful poll
Root cause: abiba-zulip extension had STUCK_THRESHOLD_MS=30min.
After 30min of inactivity (overnight), it reported stuck=True, which
triggered the health monitor to restart it — cycling restarts up to 18.
Permanent fixes:
1. STUCK_THRESHOLD_MS raised 30min → 4 hours in extension index.js
2. PM2 restart counter reset: delete+re-add abiba-zulip (was 18, now 0)
3. pm2-self-heal.sh alert threshold documented at 30 (implementation already)
4. Prose contract updated with historical note and threshold rationale
- Added --no-color flag to all PM2 commands
- Self-process guard: abiba-zulip is READ-ONLY, never restarted
- Corrected column indices (PID=6, restarts=8, status=9)
- Added pm2-self-heal.sh shell script implementation
Monitors whether Abiba Bot responds to @mentions in stream topics.
Self-test: sends a test mention, checks for response within 30s.
Documents 4 known failure modes with remediation rules.
Follows same self-improving pattern as litellm-self-heal.
Maps how OpenProse contracts connect to Gitea, containers, remote
agents, and infrastructure. Documents the access matrix, what works
today, and what gaps remain for full enforcement.
Self-improving contract that generates, validates, and iteratively
improves Hermes platform plugins. Tracks generations in knowledge
graph, learns from past successes and failures, and can update
its own rules as new patterns emerge.
Responsibility contract that checks abiba-zulip and abiba-telegram
every 5 minutes. Restarts stopped/errored processes, escalates if
restart count exceeds 5/hour. Logs to knowledge graph and alerts
owner via Zulip DM.
root
jerome