Commit Graph
134 Commits
Author SHA1 Message Date
Abiba cd52deda92 contracts: stable alias chain + no-master-key-for-inference rule
Agent configs aligned to stable aliases (gpu-light/gpu-dense/strix-moe/syslog-auto):
- Tanko fixed: base_url :4000->/v1 (nginx), max_context_window 131072->262144,
  aux.compression gemma->strix-moe, aux gemma->gpu-light
- Mumuni aligned: aux gemma->gpu-light, delegation/x_search qwen->gpu-dense
- Koby/Koonimo already used gpu-light (validated)
- Zero raw model names remain in any agent config

No-master-key rule: litellm_proxy_master_key (sk-litellm-...) is admin-only
(/key/list, /key/generate). All inference uses agent keys. Health-check model
tests use dedicated monitor key (/etc/litellm-monitor.env on CT 116).
Fixed: daily-infra-report.py, gpu-self-heal.py, litellm-health-check.sh.

Fixed wrong key-scope note: qwen3.6-35B-udq4 IS in scope for baggy/koby/mumuni/
abiba-pi keys (not 'NOT in allowlist' as previously documented).
2026-07-16 17:39:05 +00:00
Abiba 97f1cd77e4 litellm-self-heal: document 2026-07-16 ops sync (script fixes, systemd gpu-monitor, live-key agent check)
Health-check script: critical-only gpu-fleet alerting, strix-moe model test.
gpu-monitor: systemd unit (was bare &), Strix counted in gpu_count, VRAM
thresholds raised (93/97 — 256K ctx steady-state ~96% on RTX 3090).
agent-health-check.py: reads live LITELLM_API_KEY from gateway env (no hardcoded keys).
daily-infra-report.py: stale SYNTHETIC key → env-based master key.
2026-07-16 17:23:37 +00:00
Abiba dc572889f8 contracts: sync to ground truth — ornith-1.0-35b→strix-moe, 256K all GPUs, real LiteLLM timeouts
Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts:
- AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist
- All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1)
- LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s)
- Added LiteLLM model surface + key scoping to litellm-self-heal
- Patched health-check script path ref

Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal,
zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline,
delegation-prose-contract, mumuni-delegation-prose-contract
2026-07-16 17:02:44 +00:00
Abiba d0feb7881e Contracts: Rule 13 two key-injection patterns + koby/baggy rotation log (WAL #1300) 2026-07-16 15:10:50 +00:00
Abiba 9fd8c68bd2 Contracts: replace remaining ornith-1.0-35b refs with strix-moe 2026-07-16 13:57:57 +00:00
Abiba 39e2fa0cfa Contracts: Mumuni context-fix (WAL #1300) — strix-moe, 256K all GPUs, Rule 12/13, machine-identity vault procedure 2026-07-16 13:57:31 +00:00
root 622cf7b176 UPDATE 2026-07-15: Full fleet rebuild
- Stable role-based aliases: strix-moe, gpu-dense, gpu-light
- Strix Halo: ornith-1.0-35b -> unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 256K)
- RTX 5070: Q4_K_M -> IQ4_NL+MTP (256K, 125-213 tok/s, 2x faster)
- RTX 3090: bumped to 256K context
- Provider rename: harness -> litellm (all agents)
- syslog-auto: weighted pool restored with direct GPU routing
- Mumuni agent profile documented
- Self-heal thresholds updated for 256K context
- All model contexts corrected (RTX 3090 was 131K, not 256K)
2026-07-15 16:04:21 +00:00
root 30bf42b841 feat: Zulip v3 resilience contract + restore playbook
- zulip-resilience-v3.prose.md: Production resilience rewrite contract covering
  circuit breaker, retry with jitter, queue lifecycle management, supervisor
  watchdog, and PM2 hardening. Research-backed from Zulip event system docs.
- abiba-zulip-restore.prose.md: Quick-restore playbook for recovery scenarios.
2026-07-13 21:29:35 +00:00
abiba-bot 190ceb9be4 Merge pull request 'GPU workload redistribution + lessons learned July 2026' (#15) from feat/gpu-workload-compression-jul2026 into master 2026-07-12 23:40:12 +00:00
root 0c298eb9d9 feat: add routing configuration with RPM caps (ornith syslog-auto: 40→60)
- Strix Halo ornith RPM increased from 40 to 60 inside syslog-auto pool
- Prevented unbounded ornith traffic when multiple agents use syslog-auto
- Direct ornith remains at RPM=40
- All models verified healthy (ornith 80ms, gemma 110ms, qwen 840ms)
- Added full routing table to gpu-fleet contract
2026-07-12 23:38:21 +00:00
root af41f8f57a fix: remaining template YAML and topology table corrections
- hermes-config-template: Template YAML compression model gemma→ornith.
  Context guidance updated (qwen=256K, only gemma limited to 131K).
- litellm-self-heal: GPU topology table fixed (RTX 3090: 128K→256K, parallel 2→1)
2026-07-12 22:55:24 +00:00
root be02b0e843 fix: stale dates, context values, and architecture references
- gpu-fleet: Updated all context references (128K→256K for RTX 3090).
  Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram
  (router labeled deprecated/not in path). Corrected VRAM values.
  Updated dates: June→July 2026.
- hermes-config-template: Fixed description (removed incorrect '128K on
  NVIDIA' claim). Updated to reflect verified 256K on RTX 3090.
- litellm-self-heal: Updated last-verified date.
- litellm-api-keys: Updated last-verified date.
2026-07-12 22:54:30 +00:00
root 79d4a73895 docs: lessons learned from 2026-07-12 session
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
2026-07-12 22:49:39 +00:00
root 19b6db9891 feat: GPU workload redistribution — compression → Strix Halo
- Move compression model from gemma-4-12b (RTX 5070) to ornith-1.0-35b (Strix Halo)
- Add Rule 8: GPU Workload Distribution — per-GPU role assignment
- Add Rule 9: Compression Threshold for 256K models
- Update Rule 7: Auxiliary Model Consistency with new compression routing
- Add gpu-self-heal.prose.md contract with 10 remediation rules
- Strix Halo (64GB, 256K, 72.4 tok/s) → compression specialist
- RTX 5070 (12GB) → vision/web search specialist
- RTX 3090 (24GB, 256K) → heavy reasoning specialist
- All rules grilled and confirmed with Kwame 2026-07-12
2026-07-12 22:27:15 +00:00
jerome 22eaaf4254 Merge pull request 'fix: add data source integrity rule to delegation contract' (#14) from feat/data-source-integrity-rule into master
Reviewed-on: SyslogSolution/prose-contracts#14
2026-07-12 18:09:10 +00:00
jerome fdb22948d9 fix: add data source integrity rule to delegation contract
Workers MUST use provided input data, not fetch external sources.
Adds: Data Source Integrity section, dispatch guidance example,
anti-pattern entry. Root cause of e2e test discrepancies where
writer worker queried Proxmox API instead of using raw data.
2026-07-11 18:51:30 -04:00
jerome e9968e165d Merge pull request 'feat: add RA-H OS Custodianship Contract for shared memory protocol' (#13) from feat/ra-h-os-custodianship-contract into master
Reviewed-on: SyslogSolution/prose-contracts#13
2026-07-10 23:39:30 +00:00
mumuni-bot 44ba53cf71 fix: add YAML frontmatter to RA-H OS Custodianship Contract
- Added required frontmatter block with kind=pattern, name, and description
- Fixes CI validate step that was blocking the PR pipeline
- All other CI checks (lint, ai-review) were skipped due to validate failure
2026-07-10 19:16:30 -04:00
jerome b96b334283 feat: add RA-H OS Custodianship Contract for shared memory protocol 2026-07-10 09:43:49 -04:00
jerome dc829bb09d Merge pull request 'feat: mumuni-delegation-prose-contract — manager operating doctrine for worker delegation' (#12) from feat/delegation-prose-contract into master
Reviewed-on: SyslogSolution/prose-contracts#12
2026-07-09 07:43:26 +00:00
jerome a8354e88bc rename: delegation-prose-contract → mumuni-delegation-prose-contract
Contract is Mumuni-specific operating doctrine — different agents have
different architectures and don't need this. Renamed file and frontmatter
to reflect that.
2026-07-09 03:18:29 -04:00
jerome 98d8ce6772 fix: add topology section to satisfy lint checks
Added cluster/node topology section referencing all 5 Proxmox nodes and
clarifying that the contract is infrastructure-agnostic — defines the who
and when, not the where.
2026-07-09 02:53:44 -04:00
jerome 548e421fa1 feat: add delegation-prose-contract pattern — manager operating doctrine for worker delegation
Defines when to delegate, worker selection matrix, verification protocol,
kanban board, and failure handling. Addresses the context overflow and
iteration exhaustion issues where the manager was doing all work itself
instead of delegating to the 6 worker profiles.
2026-07-09 02:29:42 -04:00
abiba-bot 0134ad8be6 Merge pull request 'fix: cron alignment + VM→bare metal drift' (#11) from fix/cron-alignment-jul2026 into master 2026-07-09 06:28:04 +00:00
root 546ca79849 fix: cron alignment + VM→bare metal drift
- pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract)
- pm2-self-heal: added gpu-monitor to tracked PM2 processes
- infra-monitor+control: removed daytime-only restriction (now 24×7)
- infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110
- Added @reboot pm2 resurrect for persistence across reboots
- README: zulip-monitor.sh → agent-health-check.py
- Cleaned up commented-out zulip-monitor cron line

Live actions: gpu-monitor-server added to PM2 (persistent across reboots)
2026-07-09 06:27:48 +00:00
abiba-bot c31dbad95f Merge pull request 'fix: IP drift and container count from live testing' (#10) from fix/testing-drift-jul2026 into master 2026-07-09 06:24:07 +00:00
root d829f595b4 fix: IP drift and container count from live testing
Testing the consolidated litellm-self-heal contract against live infra found:
- IP drift: pi host is .24, not .65 (confused with ra-h-os bridge)
  - Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations)
  - Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables
- Container count: 10 running on CT 116, not 8 as contract stated
  - Added harness-docker-stats + harness-pve-exporter to litellm-self-heal
  - Updated container health check list

Verified: GPU monitor restarted on .24:9100, ornith model inference working,
all 10 containers healthy.
2026-07-09 06:23:44 +00:00
abiba-bot 4164ee5ed0 Merge pull request 'feat: consolidate litellm-health into litellm-self-heal' (#9) from feat/consolidate-litellm-contracts into master 2026-07-09 06:02:47 +00:00
root 053d30ce7c feat: consolidate litellm-health into litellm-self-heal
- litellm-self-heal: merged health check as §Health Check section,
  includes full architecture diagram (v4.0.0), GPU topology, timeout
  tables, container list, and execution flow. Eliminates duplication
  between the two contracts.
- litellm-health: marked DEPRECATED, retained for reference
- README.md: updated with consolidated contract, deprecation notice,
  disk-gc-threat-response, and infrastructure-monitoring status
2026-07-09 06:01:48 +00:00
abiba-bot e4ab43f1c6 Merge pull request 'fix: LiteLLM contracts — align with 2026-07-08 architecture changes' (#8) from fix/litellm-contracts-jul2026 into master 2026-07-09 05:37:48 +00:00
root a2c5a14c74 fix: LiteLLM contracts — align with 2026-07-08 architecture changes
- litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly),
  GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s),
  Strix Halo: CPU→Vulkan, context sizes + parallel slots added
- litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED
  (router not in path), GPU topology synced with gpu-fleet
- litellm-api-keys: reference gpu-fleet as source of truth for keys,
  removed qwen3.6-35B-A3B from model list (never deployed)
2026-07-09 05:37:16 +00:00
abiba-bot a7fedf71c6 Merge pull request 'fix: contract improvements from 2026-07-09 run log review' (#7) from fix/contract-improvements-jul2026 into master 2026-07-09 05:33:25 +00:00
root 97977f320c fix: contract improvements from 2026-07-09 run log review
- pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs
- disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope,
  remove jitsi (decommissioned), update access matrix with KVM VM section
- infrastructure-control: docker-vm container count 11→16, add Trove agents stack
- infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)
2026-07-09 05:32:00 +00:00
abiba-bot a3786ab289 Merge pull request 'disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB)' (#6) from feat/disk-gc-amdpve-2026-07-09 into master 2026-07-09 05:23:30 +00:00
root 15e8998289 disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs 2026-07-09 05:20:42 +00:00
root 25c6c76fb3 disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs 2026-07-09 05:17:05 +00:00
root 4819247e39 Add zulip-oidc-redirect-fix prose contract
Fixes Zulip OIDC authentication where redirect_uri used internal IP
(192.168.68.19) instead of public domain (chat.sysloggh.net) after
server restart. Monkey-patches social_core.strategy.BaseStrategy
to use ROOT_DOMAIN_URI for all redirect URI construction.
2026-07-09 00:02:18 +00:00
root 1dc4155d9a fix(hermes-zulip-plugin): default to master branch, add gateway restart
- Switched default branch from feat/zulip-streaming to master
- Added Step 5: gateway restart with smoke check after plugin install
- Plugin changes require restart to take effect for a valid test
- Outstanding feature branch merges should be addressed post-install
2026-07-08 17:51:19 +00:00
root 09efbdf4e5 feat: add hermes-zulip-plugin (install & repair) and relocate hermes-zulip-restore to repo
- hermes-zulip-plugin.prose.md: installs latest zulip-platform plugin from
  canonical repo, verifies _strip_html fix, sends relay success signal.
  Distinct from restore — plugin-layer only, no env/gateway/connection checks.
- hermes-zulip-restore.prose.md: relocated from /root/ into prose-contracts repo.
- Removed duplicate zulip-platform-verification from /root/vault/contracts/.
2026-07-08 17:47:23 +00:00
jerome 4058010e57 Merge pull request 'feat: hermes-zulip-restore contract' (#5) from feat/hermes-zulip-restore into master
Reviewed-on: SyslogSolution/prose-contracts#5
2026-07-08 17:38:16 +00:00
root 776df418ca gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3)
- .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache
- .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV
- .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm
- LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s
- Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma
- Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
2026-07-08 17:25:42 +00:00
jerome c0191c9edc feat: hermes-zulip-restore contract — restore Zulip on any Hermes agent
Deploys zulip-platform adapter to bundled plugin path, verifies
env credentials, restarts gateway, and validates Zulip connection.
Covers Mumuni (CT114), Tanko (CT112), Koby (CT111).

Includes _strip_html slash-command fix, known failure modes,
and owner-aware restart paths.
2026-07-08 03:19:01 -04:00
jerome f6e8791352 fix(ci): change kind from reference to template in hermes-agent-baseline.prose.md
The validate job was failing because 'reference' is not an allowed kind.
Changed to 'template' since this document serves as a canonical baseline
template for restoring agent configurations.
2026-07-07 20:57:48 -04:00
root b1ef4cbe9b docs: self-update contract — Gen 6 postconditions, status clarification, state persistence
- Added stream_subscription_check, dm_response_time_offline_verifiable, known_issues_tracked postconditions
- Added 'Status' section clarifying active for Hermes agents only (not retired with pi Zulip)
- Added 'State Persistence' section with KG node schema for build-zulip-plugin:state
- Execution step 4: always run offline tests (simulate_dm_latency, selftest)
- Execution step 5: update KG state node after each generation
2026-07-07 07:18:45 +00:00
kagentz-bot 41ef546d21 Merge pull request 'fix: use IP 192.168.68.17:3000 instead of git.sysloggh.net for CI checkout' (#4) from fix/ci-checkout-ip into master 2026-07-06 22:56:48 +00:00
Agent Zero 8f89603c31 fix: use IP 192.168.68.17:3000 instead of git.sysloggh.net hostname for CI checkout
The Gitea Actions runner cannot resolve the internal hostname
git.sysloggh.net during checkout. Using the direct IP address
resolves the DNS resolution issue in CI pipeline steps.
2026-07-06 18:54:12 -04:00
Agent Zero f1a031b682 fix: Firecrawl health endpoint is / not /health 2026-07-06 18:21:30 -04:00
Agent Zero f14b2fc04e infra-update contract: v1.1.0 - fix CT→VM refs, remove deprecated stacks 2026-07-06 18:05:16 -04:00
Agent Zero 09ffbf10aa infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths 2026-07-06 18:05:16 -04:00
root 5aa93117de docs: capture all session changes — port conflict, health check, streaming
Updated contracts to reflect all infrastructure changes made 2026-07-05/06:

gpu-fleet.prose.md:
  - Port conflict detection on all 3 GPU wrappers
  - Ghost process root cause documented (.8 stale pid 25836)

infrastructure-control.prose.md:
  - Section 7: Agent Health Check (consolidated, non-disruptive)
  - Disabled scripts documented (zulip-watchdog, zulip-monitor)

hermes-agent-baseline.prose.md:
  - Health verification section + GPU port conflict detection
  - Change log updated

zulip-health.prose.md:
  - Streaming support section (edit_message + PATCH API)
2026-07-06 02:46:41 +00:00