The Gitea Actions runner cannot resolve the internal hostname
git.sysloggh.net during checkout. Using the direct IP address
resolves the DNS resolution issue in CI pipeline steps.
Single non-disruptive script replacing 7 scattered Zulip health checks.
Checks every 10 min via cron, never restarts anything:
- LiteLLM key validation for all 4 Hermes agents
- GPU port conflict / ghost process detection
- Agent gateway liveness + Zulip streaming status
- Recent gateway error count
Port conflict detection added to all 3 GPU wrappers:
- .8 (qwen): wrapper detects ghost on port 8080 before starting
- .110 (gemma): same pattern
- .15 (ornith): port-cleanup.sh replaces blanket pkill -x llama-server
_resolve_task_provider_model() in agent/auxiliary_client.py ignores
api_key_env for auxiliary tasks (vision, compression). The custom
provider path resolves it, but aux tasks take a different code path.
Workaround: set api_key directly alongside api_key_env.
Applied to Tanko (vision + compression).
Tanko: old key sk-620a05e95a... was from docker-compose router config,
not LiteLLM key DB. Deleted + regenerated via LiteLLM API:
sk-CggiHWlamQyShxWC3Hx6uw (alias: tanko, budget: $100)
Mumuni: verified via pct-run 114 — key is sk-VrqCNlwUgzoNGOpikJ7nwQ
(was previously listed as sk-XY2aUfvy2... in contracts)
hermes-key-enforcement.prose.md:
- Add IP column to verified agents table
- Fix Tanko status: was using master key via systemd drop-in (not compliant)
- Add detection query step 2: check systemd drop-ins for master key leaks
- Add step 3: verify running process env against dedicated key
- Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)
gpu-fleet.prose.md:
- Add CT ID column to agent table for cross-reference
- Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
- Update Abiba LiteLLM key to match actual models.json key
- Set Koonimo IP to unknown (was .114 — incorrect per user)
- Fix pi-specific paths .24 → .65
hermes-config-template.prose.md:
- Set Koonimo IP to unknown
infrastructure-control.prose.md:
- Set Koonimo IP to unknown
- Fix Koonimo CT reference
Test results:
- Test #1: Recursive grep finds sub-profile violations old query missed (2/2 vs 1/2)
- Test #2: safe-mutate step correctly positioned in Violation Response, YAML valid
- Replaced actions/checkout@v4 with native git clone (no Node.js needed)
- Fixed FAILED counter: increment instead of resetting to 1
- Fixed exit condition: -gt 0 instead of -eq 1 (caught >1 violation)
- Test-violations.prose.md still present — pipeline should now catch it
- Removed actions/checkout@v4 (requires Node.js, unavailable on runner)
- Gitea act runner provides repo checkout automatically
- Added 'enforcement' to valid kind list for hermes-key-enforcement
- All steps are now pure shell — zero external dependencies
Previously the pipeline only triggered on pull_request events.
Direct pushes to master bypassed all validation (auth, lint, ai-review).
Now push to master also triggers the pipeline.
NEW: docs/AUTHORING-GUIDE.md — canonical style guide for writing contracts
Patterns borrowed from Anthropic's frontend-design skill:
- Ground it in the subject: verify live state before writing
- Two-pass process: draft → critique → revise → ship
- Restraint: one contract, one concern; cut one thing before shipping
- Self-critique: would an agent trust this at 3am with 30s of context?
Core rules:
- Specificity over generality (lists rot, vagueness is invisible)
- Live-state fields in tables, not prose (grep-able)
- Descriptions answer: what, to what, why care?
- Templates for function and responsibility contracts
- CI enforces structure; author enforces quality
Linked from AGENTS.md for agent discovery.
- New 'Operating Doctrine (read first)' section with the .117 rule, safe-mutate wrapper usage, field trust taxonomy, and contract-native verify-before-fix flow
- Added hermes-key-enforcement to templates table
- Added stirling-pdf-agent-access to functions table
- Noted VERIFY-BEFORE-USE labels on infrastructure-control
- Added FIELD TRUST convention to frontmatter (live-state vs policy fields)
- Marked credentials/IPs with VERIFY-BEFORE-USE labels
- Fixed 3 stale Zulip IP references (.117 → .19) that caused the 2026-07-03 outage
- Added verify-before-mutate skill reference
- See knowledge graph node #604 for post-mortem
- hermes-key-enforcement.prose.md: NEW contract enforcing api_key_env for all harness/LiteLLM providers, exempting external providers
- hermes-config-template.prose.md: Updated with full fix log (15 total fixes across Koby/Koonimo/Mumuni/Tanko), model:auto detection rule, key-to-agent mapping
- infrastructure-control.prose.md: Added Stirling-PDF service details (port 8989, credentials, API key, Swagger URL) replacing Bento-PDF
- REMOVED hardcoded Agent Keys table (keys became stale after LiteLLM downgrade).
- ADDED Key Management section: verify loop, rotate + deploy procedure, key
deployment rules (plaintext in /etc/environment only, api_key_env pattern),
current aliases (2026-07-02 rotation).
- LiteLLM DB is the single source of truth for keys; contracts never store plaintext.
- Mumuni sub-agent profiles documented (6, all inherit auth from main config).
- infrastructure-control: add Section 5.3 (Network Verification & Routing) with
dual-path checks, NetBird ingress health, DNS drift detection, and routing
regression alerts. Add Section 7 (Configuration Doctrine — IP-First) mandating
LAN IPs for all configs/agents, URLs reserved for human browser access only.
Also enrich access matrix (Mumuni, Koonimo, LiteLLM Admin, Grafana entries),
reachability matrix, and CT 116 nginx routing + Prometheus targets.
- build-zulip-plugin: fold Success Criteria into Maintains postconditions.
- pi-approval-architecture: new reference doc — pi approval model vs Hermes,
architectural limits, /approve and /deny stub commands.
- zulip-approval-fix: new responsibility — documents the Zulip HTML/prefix bug
that broke /approve and /deny, and the one-line fix applied.
Agent Zero
jerome