Commit Graph
46 Commits
Author SHA1 Message Date
root 053d30ce7c feat: consolidate litellm-health into litellm-self-heal
- litellm-self-heal: merged health check as §Health Check section,
  includes full architecture diagram (v4.0.0), GPU topology, timeout
  tables, container list, and execution flow. Eliminates duplication
  between the two contracts.
- litellm-health: marked DEPRECATED, retained for reference
- README.md: updated with consolidated contract, deprecation notice,
  disk-gc-threat-response, and infrastructure-monitoring status
2026-07-09 06:01:48 +00:00
root a2c5a14c74 fix: LiteLLM contracts — align with 2026-07-08 architecture changes
- litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly),
  GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s),
  Strix Halo: CPU→Vulkan, context sizes + parallel slots added
- litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED
  (router not in path), GPU topology synced with gpu-fleet
- litellm-api-keys: reference gpu-fleet as source of truth for keys,
  removed qwen3.6-35B-A3B from model list (never deployed)
2026-07-09 05:37:16 +00:00
root 97977f320c fix: contract improvements from 2026-07-09 run log review
- pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs
- disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope,
  remove jitsi (decommissioned), update access matrix with KVM VM section
- infrastructure-control: docker-vm container count 11→16, add Trove agents stack
- infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)
2026-07-09 05:32:00 +00:00
root 15e8998289 disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs 2026-07-09 05:20:42 +00:00
root 25c6c76fb3 disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs 2026-07-09 05:17:05 +00:00
root 4819247e39 Add zulip-oidc-redirect-fix prose contract
Fixes Zulip OIDC authentication where redirect_uri used internal IP
(192.168.68.19) instead of public domain (chat.sysloggh.net) after
server restart. Monkey-patches social_core.strategy.BaseStrategy
to use ROOT_DOMAIN_URI for all redirect URI construction.
2026-07-09 00:02:18 +00:00
root 1dc4155d9a fix(hermes-zulip-plugin): default to master branch, add gateway restart
- Switched default branch from feat/zulip-streaming to master
- Added Step 5: gateway restart with smoke check after plugin install
- Plugin changes require restart to take effect for a valid test
- Outstanding feature branch merges should be addressed post-install
2026-07-08 17:51:19 +00:00
root 09efbdf4e5 feat: add hermes-zulip-plugin (install & repair) and relocate hermes-zulip-restore to repo
- hermes-zulip-plugin.prose.md: installs latest zulip-platform plugin from
  canonical repo, verifies _strip_html fix, sends relay success signal.
  Distinct from restore — plugin-layer only, no env/gateway/connection checks.
- hermes-zulip-restore.prose.md: relocated from /root/ into prose-contracts repo.
- Removed duplicate zulip-platform-verification from /root/vault/contracts/.
2026-07-08 17:47:23 +00:00
root 776df418ca gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3)
- .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache
- .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV
- .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm
- LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s
- Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma
- Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
2026-07-08 17:25:42 +00:00
root b1ef4cbe9b docs: self-update contract — Gen 6 postconditions, status clarification, state persistence
- Added stream_subscription_check, dm_response_time_offline_verifiable, known_issues_tracked postconditions
- Added 'Status' section clarifying active for Hermes agents only (not retired with pi Zulip)
- Added 'State Persistence' section with KG node schema for build-zulip-plugin:state
- Execution step 4: always run offline tests (simulate_dm_latency, selftest)
- Execution step 5: update KG state node after each generation
2026-07-07 07:18:45 +00:00
root 5aa93117de docs: capture all session changes — port conflict, health check, streaming
Updated contracts to reflect all infrastructure changes made 2026-07-05/06:

gpu-fleet.prose.md:
  - Port conflict detection on all 3 GPU wrappers
  - Ghost process root cause documented (.8 stale pid 25836)

infrastructure-control.prose.md:
  - Section 7: Agent Health Check (consolidated, non-disruptive)
  - Disabled scripts documented (zulip-watchdog, zulip-monitor)

hermes-agent-baseline.prose.md:
  - Health verification section + GPU port conflict detection
  - Change log updated

zulip-health.prose.md:
  - Streaming support section (edit_message + PATCH API)
2026-07-06 02:46:41 +00:00
root 79855ea9e1 feat: consolidated agent health check — key validation + GPU port conflict + streaming
Single non-disruptive script replacing 7 scattered Zulip health checks.
Checks every 10 min via cron, never restarts anything:
- LiteLLM key validation for all 4 Hermes agents
- GPU port conflict / ghost process detection
- Agent gateway liveness + Zulip streaming status
- Recent gateway error count

Port conflict detection added to all 3 GPU wrappers:
- .8 (qwen): wrapper detects ghost on port 8080 before starting
- .110 (gemma): same pattern
- .15 (ornith): port-cleanup.sh replaces blanket pkill -x llama-server
2026-07-06 02:23:54 +00:00
root 2ea9cac23a docs: hermes-agent-baseline — canonical good-state reference
Single source of truth for all 4 Hermes agents. Captures:
- Agent map (CT, IP, key, alias)
- Key architecture and config patterns
- Mandatory api_key workaround with explanation
- Known bug: auxiliary_client ignores api_key_env
- Audit procedure (full audit, leak check, key verification)
- Systemd patterns
- Attachment cache locations
2026-07-05 23:37:56 +00:00
root 35e864bdb7 audit: all 4 Hermes agents verified — no master key, api_key workaround applied
Agent audit (2026-07-05):
  Tanko    CT 112: sk-CggiHWlamQyShxWC3Hx6uw  — vision+compression patched
  Mumuni   CT 114: sk-VrqCNlwUgzoNGOpikJ7nwQ  — vision patched
  Tdunna   CT 111: sk-6sbCNjz2T6lTVDBdlNHXsA  — vision+compression patched
  Baggy    CT 113: sk-krnw_zGBwvvL5b7l2t-s-A  — vision+compression patched

All 4 agents: zero master key references, dedicated LiteLLM keys,
api_key workaround applied to bypass auxiliary_client bug.
2026-07-05 23:36:00 +00:00
root 53d4290886 docs: document auxiliary_client api_key_env bug + workaround
_resolve_task_provider_model() in agent/auxiliary_client.py ignores
api_key_env for auxiliary tasks (vision, compression). The custom
provider path resolves it, but aux tasks take a different code path.

Workaround: set api_key directly alongside api_key_env.
Applied to Tanko (vision + compression).
2026-07-05 23:17:37 +00:00
root c8dcd8b63d fix: Tanko key regenerated, Mumuni key verified from /etc/environment
Tanko: old key sk-620a05e95a... was from docker-compose router config,
not LiteLLM key DB. Deleted + regenerated via LiteLLM API:
  sk-CggiHWlamQyShxWC3Hx6uw  (alias: tanko, budget: $100)

Mumuni: verified via pct-run 114 — key is sk-VrqCNlwUgzoNGOpikJ7nwQ
(was previously listed as sk-XY2aUfvy2... in contracts)
2026-07-05 20:18:05 +00:00
root 5c90eb8f6c rename: Koonimo→Baggy, Koby→Tdunna — match hostnames
Standardize agent names across all contracts and pct-run.sh.
No functional changes — same CT IDs, same keys, same nodes.
2026-07-05 20:04:22 +00:00
root b65116b847 feat: pct-run.sh — run commands in CTs by ID, no IPs needed
Adds /scripts/pct-run.sh that maps CT IDs to PVE nodes and runs
pct exec directly. No hardcoded IPs — Proxmox is the source of truth.

Updates infrastructure-control with CT access table using pct-run.

Usage examples:
  pct-run 112 cat /etc/hostname       # Tanko
  pct-run 114 grep LITELLM /etc/environment  # Mumuni key check
  pct-run 116 docker ps               # LiteLLM containers
2026-07-05 20:02:20 +00:00
root 0db95a4d1c fix: correct IPs, CT IDs, and key status across all contracts
hermes-key-enforcement.prose.md:
  - Add IP column to verified agents table
  - Fix Tanko status: was using master key via systemd drop-in (not compliant)
  - Add detection query step 2: check systemd drop-ins for master key leaks
  - Add step 3: verify running process env against dedicated key
  - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)

gpu-fleet.prose.md:
  - Add CT ID column to agent table for cross-reference
  - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
  - Update Abiba LiteLLM key to match actual models.json key
  - Set Koonimo IP to unknown (was .114 — incorrect per user)
  - Fix pi-specific paths .24 → .65

hermes-config-template.prose.md:
  - Set Koonimo IP to unknown

infrastructure-control.prose.md:
  - Set Koonimo IP to unknown
  - Fix Koonimo CT reference
2026-07-05 19:58:14 +00:00
root 2d2259035a Merge branch 'fix/mumuni-review-suggestions': tested changes
- Recursive detection query catches sub-profile violations (verified)
- safe-mutate cross-reference added to Violation Response (verified)
2026-07-05 08:50:02 +00:00
root 7ce6cad01f fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference
Test results:
- Test #1: Recursive grep finds sub-profile violations old query missed (2/2 vs 1/2)
- Test #2: safe-mutate step correctly positioned in Violation Response, YAML valid
2026-07-05 08:49:58 +00:00
root 0147ac2abb Revert "fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference"
This reverts commit c4ab31e56b.
2026-07-05 08:49:23 +00:00
root c4ab31e56b fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference 2026-07-05 08:46:32 +00:00
root 9cf6304e23 fix: remove hardcoded LiteLLM defaults — rely on Gitea secrets 2026-07-04 23:33:34 +00:00
root 91ec7ea3d2 Merge PR #1: docs: update key enforcement — all agents named keys, systemd fix, CI pipeline 2026-07-04 23:32:19 +00:00
root e59f59d776 fix: ai-review gracefully skips when LiteLLM secrets not configured 2026-07-04 23:30:25 +00:00
root 889f4e4dd1 docs: update key enforcement — all agents named keys, systemd fix, CI pipeline
- All 4 Hermes agents now have dedicated named LiteLLM keys (tanko, mumuni, koby, koonimo)
- Documented systemd drop-in pattern fix (litellm-key.conf)
- Added one-step rotation procedure
- Added CI pipeline and branch protection documentation
- Removed old mumuni keys, generated fresh keys for koby/koonimo
2026-07-04 23:28:59 +00:00
root cd7d038f91 ci: verify branch protection — abiba-bot push should work 2026-07-04 23:15:56 +00:00
root b3ab3a6850 fix: remove all Gitea expression syntax from ai-review script 2026-07-04 23:09:29 +00:00
root 105844cf7b fix: ai-review script — handle Gitea env vars + push event diffs 2026-07-04 23:08:19 +00:00
root 52566ff36e fix: remove test file, add missing descriptions to zulip-health + memory-audit-maintenance 2026-07-04 23:06:40 +00:00
root 36da6cb1f7 fix: use http://git.sysloggh.net for checkout (no TLS inside network) 2026-07-04 23:00:55 +00:00
root 3dd2698ef8 fix: add native git checkout + fix validator increment/exit bugs
- Replaced actions/checkout@v4 with native git clone (no Node.js needed)
- Fixed FAILED counter: increment instead of resetting to 1
- Fixed exit condition: -gt 0 instead of -eq 1 (caught >1 violation)
- Test-violations.prose.md still present — pipeline should now catch it
2026-07-04 23:00:03 +00:00
root 89af42fa4c test: intentional violations — should fail pipeline 2026-07-04 22:58:15 +00:00
root b669d1becf fix: remove Node.js dependency from CI pipeline + add enforcement kind
- Removed actions/checkout@v4 (requires Node.js, unavailable on runner)
- Gitea act runner provides repo checkout automatically
- Added 'enforcement' to valid kind list for hermes-key-enforcement
- All steps are now pure shell — zero external dependencies
2026-07-04 22:54:42 +00:00
root ae5fb4d335 ci: verify pipeline with Node.js on runner CT 110 2026-07-04 22:53:25 +00:00
root d27892f7a5 ci: trigger pipeline — Node.js installed on runner CT 110 2026-07-04 22:52:37 +00:00
root 031a8253c7 fix: add push trigger for master branch to CI pipeline
Previously the pipeline only triggered on pull_request events.
Direct pushes to master bypassed all validation (auth, lint, ai-review).
Now push to master also triggers the pipeline.
2026-07-04 22:50:02 +00:00
root 895858c60a feat: LiteLLM API key management contract + longevity policy
- New contract: litellm-api-keys.prose.md — create/rotate/verify named API keys
- Renamed tanko key alias from tanko-jul2026 to tanko (bare name)
- Added Key Longevity Policy: permanent keys (duration:null), event-driven rotation
- Added default_key_generate_params to litellm config (permanent + 00 budget)
- Updated hermes-key-enforcement verified agents table for 2026-07-04
2026-07-04 22:48:48 +00:00
root 4eec96b851 docs: link authoring guide from AGENTS.md 2026-07-04 22:31:44 +00:00
root 436f583882 docs: add prose contract authoring guide (Anthropic-inspired)
NEW: docs/AUTHORING-GUIDE.md — canonical style guide for writing contracts

Patterns borrowed from Anthropic's frontend-design skill:
- Ground it in the subject: verify live state before writing
- Two-pass process: draft → critique → revise → ship
- Restraint: one contract, one concern; cut one thing before shipping
- Self-critique: would an agent trust this at 3am with 30s of context?

Core rules:
- Specificity over generality (lists rot, vagueness is invisible)
- Live-state fields in tables, not prose (grep-able)
- Descriptions answer: what, to what, why care?
- Templates for function and responsibility contracts
- CI enforces structure; author enforces quality

Linked from AGENTS.md for agent discovery.
2026-07-04 22:31:33 +00:00
root 57605c6f16 docs: add AGENTS.md + authorization enforcement gates
NEW: AGENTS.md — complete agent workflow documentation
- Step-by-step PR workflow for all agents
- Authorization matrix (who can change which contracts)
- Emergency bypass procedure
- Quick start commands

NEW: scripts/prose-auth-check.sh — authorization enforcement
- Blocks unauthorized agents from changing CRITICAL contracts
- infrastructure-control, proxmox-monitor: abiba only
- hermes-config-template: abiba, mumuni, tanko
- zulip-health: abiba, mumuni
- All scripts: abiba only
- Fails CI if unauthorized changes detected

UPDATED: .gitea/workflows/pr-pipeline.yaml
- Added Stage 0: auth check (runs first)
- Added gate job that confirms all 4 checks passed
- Expanded trigger paths to include scripts/*.sh

UPDATED: branch protection — now requires 4 contexts:
  pr-pipeline / auth, validate, lint, ai-review
2026-07-04 22:25:05 +00:00
root 263c4080f3 ci: add automated PR validation pipeline + fix regressions
NEW: Gitea Actions CI pipeline (.gitea/workflows/pr-pipeline.yaml)
- Stage 1 (validate): YAML frontmatter validation for all .prose.md files
- Stage 2 (lint): Structural checks + regression detection
  * /grafana/ nginx route (reverted 2026-07-02)
  * Stale CT IDs (122→112, no 123)
  * Verified: .19/.122/.123 = correct bridge IPs
- Stage 3 (ai-review): LiteLLM-powered diff review against ground truth
- Stage 4 (auto-merge): Gate — safe to merge when all green

NEW: scripts/prose-lint.sh — contract structure + regression checker
NEW: scripts/prose-ai-review.sh — AI review via syslog-auto model

FIXES FOUND BY LINT:
- gpu-fleet.prose.md: removed /grafana/ from nginx routing diagram
- gpu-fleet.prose.md: removed /grafana/ from config file description
- infrastructure-control.prose.md: CT 122→112 in topology diagram
- scripts/prose-lint.sh: updated to not flag .19 (verified correct IP)
2026-07-04 22:13:51 +00:00
root dd985f0e49 pm2-self-heal: migrate alerts from Zulip to Telegram
- abiba-zulip PM2 process decommissioned (2026-07-04)
- Replace Zulip stream alerts (agent-hub > alerts-pm2) with Telegram DM
- Remove stale abiba-zulip monitoring block (process no longer exists)
- Retain abiba-telegram auto-restart (critical — sole communication bridge)
- Token sourced from extension .env, not hardcoded
2026-07-04 22:07:35 +00:00
root 0d0215e9dd prose: annotate infrastructure-monitoring with current deployment status
- Core stack (Prometheus + Grafana + pve/node/docker exporters) already
  deployed by proxmox-monitor contract (Jul 2)
- GPU exporters (.8/.110/.15) NOT yet live — gpu-exporter crash-loops,
  sidecars never deployed
- Strike nginx /monitoring/ proxy suggestion (sub-path pattern already
  tried and reverted for /grafana/ per proxmox-monitor)
- Add cross-reference: this contract defines target state, proxmox-monitor
  is the as-built
2026-07-04 21:55:43 +00:00
root 253e19f680 prose: review, fix, and consolidate infrastructure contracts
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00