Commit Graph
93 Commits
Author SHA1 Message Date
root 776df418ca gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3)
- .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache
- .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV
- .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm
- LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s
- Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma
- Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
2026-07-08 17:25:42 +00:00
jerome f6e8791352 fix(ci): change kind from reference to template in hermes-agent-baseline.prose.md
The validate job was failing because 'reference' is not an allowed kind.
Changed to 'template' since this document serves as a canonical baseline
template for restoring agent configurations.
2026-07-07 20:57:48 -04:00
root b1ef4cbe9b docs: self-update contract — Gen 6 postconditions, status clarification, state persistence
- Added stream_subscription_check, dm_response_time_offline_verifiable, known_issues_tracked postconditions
- Added 'Status' section clarifying active for Hermes agents only (not retired with pi Zulip)
- Added 'State Persistence' section with KG node schema for build-zulip-plugin:state
- Execution step 4: always run offline tests (simulate_dm_latency, selftest)
- Execution step 5: update KG state node after each generation
2026-07-07 07:18:45 +00:00
kagentz-bot 41ef546d21 Merge pull request 'fix: use IP 192.168.68.17:3000 instead of git.sysloggh.net for CI checkout' (#4) from fix/ci-checkout-ip into master 2026-07-06 22:56:48 +00:00
Agent Zero 8f89603c31 fix: use IP 192.168.68.17:3000 instead of git.sysloggh.net hostname for CI checkout
The Gitea Actions runner cannot resolve the internal hostname
git.sysloggh.net during checkout. Using the direct IP address
resolves the DNS resolution issue in CI pipeline steps.
2026-07-06 18:54:12 -04:00
Agent Zero f1a031b682 fix: Firecrawl health endpoint is / not /health 2026-07-06 18:21:30 -04:00
Agent Zero f14b2fc04e infra-update contract: v1.1.0 - fix CT→VM refs, remove deprecated stacks 2026-07-06 18:05:16 -04:00
Agent Zero 09ffbf10aa infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths 2026-07-06 18:05:16 -04:00
root 5aa93117de docs: capture all session changes — port conflict, health check, streaming
Updated contracts to reflect all infrastructure changes made 2026-07-05/06:

gpu-fleet.prose.md:
  - Port conflict detection on all 3 GPU wrappers
  - Ghost process root cause documented (.8 stale pid 25836)

infrastructure-control.prose.md:
  - Section 7: Agent Health Check (consolidated, non-disruptive)
  - Disabled scripts documented (zulip-watchdog, zulip-monitor)

hermes-agent-baseline.prose.md:
  - Health verification section + GPU port conflict detection
  - Change log updated

zulip-health.prose.md:
  - Streaming support section (edit_message + PATCH API)
2026-07-06 02:46:41 +00:00
root 79855ea9e1 feat: consolidated agent health check — key validation + GPU port conflict + streaming
Single non-disruptive script replacing 7 scattered Zulip health checks.
Checks every 10 min via cron, never restarts anything:
- LiteLLM key validation for all 4 Hermes agents
- GPU port conflict / ghost process detection
- Agent gateway liveness + Zulip streaming status
- Recent gateway error count

Port conflict detection added to all 3 GPU wrappers:
- .8 (qwen): wrapper detects ghost on port 8080 before starting
- .110 (gemma): same pattern
- .15 (ornith): port-cleanup.sh replaces blanket pkill -x llama-server
2026-07-06 02:23:54 +00:00
root 2ea9cac23a docs: hermes-agent-baseline — canonical good-state reference
Single source of truth for all 4 Hermes agents. Captures:
- Agent map (CT, IP, key, alias)
- Key architecture and config patterns
- Mandatory api_key workaround with explanation
- Known bug: auxiliary_client ignores api_key_env
- Audit procedure (full audit, leak check, key verification)
- Systemd patterns
- Attachment cache locations
2026-07-05 23:37:56 +00:00
root 35e864bdb7 audit: all 4 Hermes agents verified — no master key, api_key workaround applied
Agent audit (2026-07-05):
  Tanko    CT 112: sk-CggiHWlamQyShxWC3Hx6uw  — vision+compression patched
  Mumuni   CT 114: sk-VrqCNlwUgzoNGOpikJ7nwQ  — vision patched
  Tdunna   CT 111: sk-6sbCNjz2T6lTVDBdlNHXsA  — vision+compression patched
  Baggy    CT 113: sk-krnw_zGBwvvL5b7l2t-s-A  — vision+compression patched

All 4 agents: zero master key references, dedicated LiteLLM keys,
api_key workaround applied to bypass auxiliary_client bug.
2026-07-05 23:36:00 +00:00
root 53d4290886 docs: document auxiliary_client api_key_env bug + workaround
_resolve_task_provider_model() in agent/auxiliary_client.py ignores
api_key_env for auxiliary tasks (vision, compression). The custom
provider path resolves it, but aux tasks take a different code path.

Workaround: set api_key directly alongside api_key_env.
Applied to Tanko (vision + compression).
2026-07-05 23:17:37 +00:00
root c8dcd8b63d fix: Tanko key regenerated, Mumuni key verified from /etc/environment
Tanko: old key sk-620a05e95a... was from docker-compose router config,
not LiteLLM key DB. Deleted + regenerated via LiteLLM API:
  sk-CggiHWlamQyShxWC3Hx6uw  (alias: tanko, budget: $100)

Mumuni: verified via pct-run 114 — key is sk-VrqCNlwUgzoNGOpikJ7nwQ
(was previously listed as sk-XY2aUfvy2... in contracts)
2026-07-05 20:18:05 +00:00
root 5c90eb8f6c rename: Koonimo→Baggy, Koby→Tdunna — match hostnames
Standardize agent names across all contracts and pct-run.sh.
No functional changes — same CT IDs, same keys, same nodes.
2026-07-05 20:04:22 +00:00
root b65116b847 feat: pct-run.sh — run commands in CTs by ID, no IPs needed
Adds /scripts/pct-run.sh that maps CT IDs to PVE nodes and runs
pct exec directly. No hardcoded IPs — Proxmox is the source of truth.

Updates infrastructure-control with CT access table using pct-run.

Usage examples:
  pct-run 112 cat /etc/hostname       # Tanko
  pct-run 114 grep LITELLM /etc/environment  # Mumuni key check
  pct-run 116 docker ps               # LiteLLM containers
2026-07-05 20:02:20 +00:00
root 0db95a4d1c fix: correct IPs, CT IDs, and key status across all contracts
hermes-key-enforcement.prose.md:
  - Add IP column to verified agents table
  - Fix Tanko status: was using master key via systemd drop-in (not compliant)
  - Add detection query step 2: check systemd drop-ins for master key leaks
  - Add step 3: verify running process env against dedicated key
  - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)

gpu-fleet.prose.md:
  - Add CT ID column to agent table for cross-reference
  - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
  - Update Abiba LiteLLM key to match actual models.json key
  - Set Koonimo IP to unknown (was .114 — incorrect per user)
  - Fix pi-specific paths .24 → .65

hermes-config-template.prose.md:
  - Set Koonimo IP to unknown

infrastructure-control.prose.md:
  - Set Koonimo IP to unknown
  - Fix Koonimo CT reference
2026-07-05 19:58:14 +00:00
root 2d2259035a Merge branch 'fix/mumuni-review-suggestions': tested changes
- Recursive detection query catches sub-profile violations (verified)
- safe-mutate cross-reference added to Violation Response (verified)
2026-07-05 08:50:02 +00:00
root 7ce6cad01f fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference
Test results:
- Test #1: Recursive grep finds sub-profile violations old query missed (2/2 vs 1/2)
- Test #2: safe-mutate step correctly positioned in Violation Response, YAML valid
2026-07-05 08:49:58 +00:00
root 0147ac2abb Revert "fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference"
This reverts commit c4ab31e56b.
2026-07-05 08:49:23 +00:00
root c4ab31e56b fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference 2026-07-05 08:46:32 +00:00
root 9cf6304e23 fix: remove hardcoded LiteLLM defaults — rely on Gitea secrets 2026-07-04 23:33:34 +00:00
root 91ec7ea3d2 Merge PR #1: docs: update key enforcement — all agents named keys, systemd fix, CI pipeline 2026-07-04 23:32:19 +00:00
root e59f59d776 fix: ai-review gracefully skips when LiteLLM secrets not configured 2026-07-04 23:30:25 +00:00
root 889f4e4dd1 docs: update key enforcement — all agents named keys, systemd fix, CI pipeline
- All 4 Hermes agents now have dedicated named LiteLLM keys (tanko, mumuni, koby, koonimo)
- Documented systemd drop-in pattern fix (litellm-key.conf)
- Added one-step rotation procedure
- Added CI pipeline and branch protection documentation
- Removed old mumuni keys, generated fresh keys for koby/koonimo
2026-07-04 23:28:59 +00:00
root cd7d038f91 ci: verify branch protection — abiba-bot push should work 2026-07-04 23:15:56 +00:00
root b3ab3a6850 fix: remove all Gitea expression syntax from ai-review script 2026-07-04 23:09:29 +00:00
root 105844cf7b fix: ai-review script — handle Gitea env vars + push event diffs 2026-07-04 23:08:19 +00:00
root 52566ff36e fix: remove test file, add missing descriptions to zulip-health + memory-audit-maintenance 2026-07-04 23:06:40 +00:00
root 36da6cb1f7 fix: use http://git.sysloggh.net for checkout (no TLS inside network) 2026-07-04 23:00:55 +00:00
root 3dd2698ef8 fix: add native git checkout + fix validator increment/exit bugs
- Replaced actions/checkout@v4 with native git clone (no Node.js needed)
- Fixed FAILED counter: increment instead of resetting to 1
- Fixed exit condition: -gt 0 instead of -eq 1 (caught >1 violation)
- Test-violations.prose.md still present — pipeline should now catch it
2026-07-04 23:00:03 +00:00
root 89af42fa4c test: intentional violations — should fail pipeline 2026-07-04 22:58:15 +00:00
root b669d1becf fix: remove Node.js dependency from CI pipeline + add enforcement kind
- Removed actions/checkout@v4 (requires Node.js, unavailable on runner)
- Gitea act runner provides repo checkout automatically
- Added 'enforcement' to valid kind list for hermes-key-enforcement
- All steps are now pure shell — zero external dependencies
2026-07-04 22:54:42 +00:00
root ae5fb4d335 ci: verify pipeline with Node.js on runner CT 110 2026-07-04 22:53:25 +00:00
root d27892f7a5 ci: trigger pipeline — Node.js installed on runner CT 110 2026-07-04 22:52:37 +00:00
root 031a8253c7 fix: add push trigger for master branch to CI pipeline
Previously the pipeline only triggered on pull_request events.
Direct pushes to master bypassed all validation (auth, lint, ai-review).
Now push to master also triggers the pipeline.
2026-07-04 22:50:02 +00:00
root 895858c60a feat: LiteLLM API key management contract + longevity policy
- New contract: litellm-api-keys.prose.md — create/rotate/verify named API keys
- Renamed tanko key alias from tanko-jul2026 to tanko (bare name)
- Added Key Longevity Policy: permanent keys (duration:null), event-driven rotation
- Added default_key_generate_params to litellm config (permanent + 00 budget)
- Updated hermes-key-enforcement verified agents table for 2026-07-04
2026-07-04 22:48:48 +00:00
root 4eec96b851 docs: link authoring guide from AGENTS.md 2026-07-04 22:31:44 +00:00
root 436f583882 docs: add prose contract authoring guide (Anthropic-inspired)
NEW: docs/AUTHORING-GUIDE.md — canonical style guide for writing contracts

Patterns borrowed from Anthropic's frontend-design skill:
- Ground it in the subject: verify live state before writing
- Two-pass process: draft → critique → revise → ship
- Restraint: one contract, one concern; cut one thing before shipping
- Self-critique: would an agent trust this at 3am with 30s of context?

Core rules:
- Specificity over generality (lists rot, vagueness is invisible)
- Live-state fields in tables, not prose (grep-able)
- Descriptions answer: what, to what, why care?
- Templates for function and responsibility contracts
- CI enforces structure; author enforces quality

Linked from AGENTS.md for agent discovery.
2026-07-04 22:31:33 +00:00
root 57605c6f16 docs: add AGENTS.md + authorization enforcement gates
NEW: AGENTS.md — complete agent workflow documentation
- Step-by-step PR workflow for all agents
- Authorization matrix (who can change which contracts)
- Emergency bypass procedure
- Quick start commands

NEW: scripts/prose-auth-check.sh — authorization enforcement
- Blocks unauthorized agents from changing CRITICAL contracts
- infrastructure-control, proxmox-monitor: abiba only
- hermes-config-template: abiba, mumuni, tanko
- zulip-health: abiba, mumuni
- All scripts: abiba only
- Fails CI if unauthorized changes detected

UPDATED: .gitea/workflows/pr-pipeline.yaml
- Added Stage 0: auth check (runs first)
- Added gate job that confirms all 4 checks passed
- Expanded trigger paths to include scripts/*.sh

UPDATED: branch protection — now requires 4 contexts:
  pr-pipeline / auth, validate, lint, ai-review
2026-07-04 22:25:05 +00:00
root 263c4080f3 ci: add automated PR validation pipeline + fix regressions
NEW: Gitea Actions CI pipeline (.gitea/workflows/pr-pipeline.yaml)
- Stage 1 (validate): YAML frontmatter validation for all .prose.md files
- Stage 2 (lint): Structural checks + regression detection
  * /grafana/ nginx route (reverted 2026-07-02)
  * Stale CT IDs (122→112, no 123)
  * Verified: .19/.122/.123 = correct bridge IPs
- Stage 3 (ai-review): LiteLLM-powered diff review against ground truth
- Stage 4 (auto-merge): Gate — safe to merge when all green

NEW: scripts/prose-lint.sh — contract structure + regression checker
NEW: scripts/prose-ai-review.sh — AI review via syslog-auto model

FIXES FOUND BY LINT:
- gpu-fleet.prose.md: removed /grafana/ from nginx routing diagram
- gpu-fleet.prose.md: removed /grafana/ from config file description
- infrastructure-control.prose.md: CT 122→112 in topology diagram
- scripts/prose-lint.sh: updated to not flag .19 (verified correct IP)
2026-07-04 22:13:51 +00:00
root dd985f0e49 pm2-self-heal: migrate alerts from Zulip to Telegram
- abiba-zulip PM2 process decommissioned (2026-07-04)
- Replace Zulip stream alerts (agent-hub > alerts-pm2) with Telegram DM
- Remove stale abiba-zulip monitoring block (process no longer exists)
- Retain abiba-telegram auto-restart (critical — sole communication bridge)
- Token sourced from extension .env, not hardcoded
2026-07-04 22:07:35 +00:00
root 0d0215e9dd prose: annotate infrastructure-monitoring with current deployment status
- Core stack (Prometheus + Grafana + pve/node/docker exporters) already
  deployed by proxmox-monitor contract (Jul 2)
- GPU exporters (.8/.110/.15) NOT yet live — gpu-exporter crash-loops,
  sidecars never deployed
- Strike nginx /monitoring/ proxy suggestion (sub-path pattern already
  tried and reverted for /grafana/ per proxmox-monitor)
- Add cross-reference: this contract defines target state, proxmox-monitor
  is the as-built
2026-07-04 21:55:43 +00:00
jerome 0a3b62a598 hermes-config-template: update to v2.1.0 — max_tokens, auxiliary consistency, compression threshold
- Add model.max_tokens: 4096 (thermal safety — Rule 6)
- Add context_length: 262144 to template
- Add Rule 7: Auxiliary model consistency (gemma-4-12b + base_url + api_key_env)
- Add Rule 8: Compression threshold for 256K models (0.65, not 0.25)
- Fix compression section: threshold 0.65, max_context_window 262144, protect_last_n 40
- Remove stray api_key_env from compression block (not propagated)
- Update agent Keys table to show key aliases instead of redacted values
- Add base_url and timeout to all auxiliary service configs
- Add auxiliary: compression: section matching main compression config
- Update version 2.0.0 → 2.1.0
2026-07-04 17:54:43 -04:00
root 253e19f680 prose: review, fix, and consolidate infrastructure contracts
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00
Abiba 3077fda0b7 README: surface verify-before-mutate doctrine, field trust levels, verify-before-fix pattern
- New 'Operating Doctrine (read first)' section with the .117 rule, safe-mutate wrapper usage, field trust taxonomy, and contract-native verify-before-fix flow
- Added hermes-key-enforcement to templates table
- Added stirling-pdf-agent-access to functions table
- Noted VERIFY-BEFORE-USE labels on infrastructure-control
2026-07-03 20:54:42 +00:00
Abiba d31ae0c25a Verify-before-mutate doctrine: annotate live-state fields, fix stale Zulip IP (.117→.19)
- Added FIELD TRUST convention to frontmatter (live-state vs policy fields)
- Marked credentials/IPs with VERIFY-BEFORE-USE labels
- Fixed 3 stale Zulip IP references (.117 → .19) that caused the 2026-07-03 outage
- Added verify-before-mutate skill reference
- See knowledge graph node #604 for post-mortem
2026-07-03 20:10:45 +00:00
Abiba ae143d7a88 Add Stirling-PDF agent API access contract + skill. OAuth2 configured but disabled (needs license). 2026-07-03 19:10:22 +00:00
Abiba 8dfbad5415 Remove stale Bento-PDF reference from migration note 2026-07-03 18:20:08 +00:00
Abiba c1bdbbc491 2026-07-03: Hermes key enforcement contract + Stirling-PDF docs + config standardization
- hermes-key-enforcement.prose.md: NEW contract enforcing api_key_env for all harness/LiteLLM providers, exempting external providers
- hermes-config-template.prose.md: Updated with full fix log (15 total fixes across Koby/Koonimo/Mumuni/Tanko), model:auto detection rule, key-to-agent mapping
- infrastructure-control.prose.md: Added Stirling-PDF service details (port 8989, credentials, API key, Swagger URL) replacing Bento-PDF
2026-07-03 18:19:41 +00:00