From b7e23e2592a1416bccbabb12f557a785b05a1795 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 16 Jul 2026 21:16:47 +0000 Subject: [PATCH] vault: Infisical cleanup + contract sync (WAL #1316) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - litellm-api-keys: vault audit, tanko/koby/koonimo migration, key rotation log - gpu-fleet: ornith-1.0-35b→strix-moe (remaining refs) - hermes-agent-baseline: 256K all GPUs, CT IPs updated, Shumba retired - delegation-prose-contract: removed (renamed to mumuni-delegation) - contract-registry.yaml + cron-prompts-review.md: from feat/contract-registry --- contract-registry.yaml | 1767 ++++++++++++++++++++++++++++ cron-prompts-review.md | 612 ++++++++++ delegation-prose-contract.prose.md | 256 ---- gpu-fleet.prose.md | 161 ++- hermes-agent-baseline.prose.md | 14 +- litellm-api-keys.prose.md | 156 ++- 6 files changed, 2654 insertions(+), 312 deletions(-) create mode 100644 contract-registry.yaml create mode 100644 cron-prompts-review.md delete mode 100644 delegation-prose-contract.prose.md diff --git a/contract-registry.yaml b/contract-registry.yaml new file mode 100644 index 0000000..9f7af92 --- /dev/null +++ b/contract-registry.yaml @@ -0,0 +1,1767 @@ +registry_version: 0.1.0 +last_updated: '2026-07-13T00:00:00Z' +updated_by: mumuni +categories: +- compliance +- monitoring +- remediation +- build_deploy +- maintenance +- reference +domains: +- hermes-agent +- zulip +- infrastructure +- gpu +- proxmox +- litellm +- memory +- relay +- shared-patterns +owners: +- abiba +- mumuni +- kwame +trigger_types: +- scheduled +- event_driven +- passive +- on_demand +- none +sensitivity_levels: +- critical +- high +- normal +- retired +index: + by_category: + compliance: + - hermes-key-enforcement + - hermes-config-template + - hermes-agent-baseline + monitoring: + - proxmox-monitor + - gpu-monitor + - infrastructure-monitoring + - zulip-health + - litellm-health + remediation: + - litellm-self-heal + - pm2-self-heal + - disk-gc-threat-response + - hermes-zulip-restore + build_deploy: + - hermes-zulip-plugin + - build-zulip-plugin + - stirling-pdf-agent-access + maintenance: + - memory-audit-maintenance + - gpu-fleet + - infrastructure-update + reference: + - infrastructure-control + - ra-h-os-custodianship-contract + - mumuni-delegation + - zulip-approval-fix + - zulip-oidc-redirect-fix + - hello-world + - zulip-adapter-lessons + - pi-approval-architecture + retired: + - zulip-mention-reliability + - zulip-self-heal + by_domain: + hermes-agent: + - hermes-key-enforcement + - hermes-config-template + - hermes-agent-baseline + - mumuni-delegation + zulip: + - zulip-health + - hermes-zulip-restore + - hermes-zulip-plugin + - build-zulip-plugin + - zulip-approval-fix + - zulip-oidc-redirect-fix + - zulip-adapter-lessons + - zulip-mention-reliability + - zulip-self-heal + infrastructure: + - infrastructure-control + - infrastructure-monitoring + - infrastructure-update + - pm2-self-heal + - disk-gc-threat-response + gpu: + - gpu-monitor + - gpu-fleet + proxmox: + - proxmox-monitor + litellm: + - litellm-health + - litellm-self-heal + memory: + - memory-audit-maintenance + shared-patterns: + - ra-h-os-custodianship-contract + - pi-approval-architecture + - stirling-pdf-agent-access + - hello-world + retired: + - zulip-mention-reliability + - zulip-self-heal + - zulip-adapter-lessons + - pi-approval-architecture + by_owner: + abiba: + - hermes-key-enforcement + - hermes-config-template + - hermes-agent-baseline + - proxmox-monitor + - gpu-monitor + - infrastructure-monitoring + - zulip-health + - litellm-health + - litellm-self-heal + - pm2-self-heal + - disk-gc-threat-response + - hermes-zulip-restore + - hermes-zulip-plugin + - build-zulip-plugin + - stirling-pdf-agent-access + - gpu-fleet + - infrastructure-update + - infrastructure-control + - zulip-adapter-lessons + - pi-approval-architecture + - zulip-approval-fix + - zulip-oidc-redirect-fix + - zulip-mention-reliability + - zulip-self-heal + mumuni: + - memory-audit-maintenance + - ra-h-os-custodianship-contract + - mumuni-delegation + kwame: + - hello-world + by_trigger: + scheduled: + - hermes-key-enforcement + - hermes-config-template + - hermes-agent-baseline + - proxmox-monitor + - gpu-monitor + - infrastructure-monitoring + - zulip-health + - litellm-health + - memory-audit-maintenance + - infrastructure-update + event_driven: + - litellm-self-heal + - pm2-self-heal + - disk-gc-threat-response + - hermes-zulip-restore + - gpu-fleet + - zulip-oidc-redirect-fix + passive: + - infrastructure-control + - ra-h-os-custodianship-contract + - mumuni-delegation + - zulip-adapter-lessons + - pi-approval-architecture + - zulip-approval-fix + on_demand: + - hermes-zulip-plugin + - build-zulip-plugin + - stirling-pdf-agent-access + - hello-world + none: + - zulip-mention-reliability + - zulip-self-heal + by_sensitivity: + critical: + - hermes-key-enforcement + - proxmox-monitor + - gpu-monitor + - litellm-health + - litellm-self-heal + - disk-gc-threat-response + - gpu-fleet + - infrastructure-control + high: + - hermes-config-template + - infrastructure-monitoring + - zulip-health + - pm2-self-heal + - hermes-zulip-restore + - hermes-zulip-plugin + - build-zulip-plugin + - infrastructure-update + - ra-h-os-custodianship-contract + - mumuni-delegation + normal: + - hermes-agent-baseline + - stirling-pdf-agent-access + - memory-audit-maintenance + - zulip-approval-fix + - zulip-oidc-redirect-fix + - hello-world + retired: + - zulip-adapter-lessons + - pi-approval-architecture + - zulip-mention-reliability + - zulip-self-heal +contracts: +- name: hermes-key-enforcement + file: hermes-key-enforcement.prose.md + kind: enforcement + category: compliance + sensitivity: critical + status: active + owner: abiba + version: 1.0.0 + trigger: + type: scheduled + cadence: 0 6 * * * + description: Daily compliance scan at 6am ET + cron_job_id: null + execution: + agent: abiba + timeout: 300 + requires: [] + protocol: + - Load contract from prose-contracts/main + - 'Verify prerequisites: config.yaml readable, harness/litellm sections present' + - 'Execute enforcement: grep config for plaintext keys' + - 'On failure: stop, don''t hallucinate forward' + - Log actions to ~/.hermes/runs/hermes-key-enforcement/ + verification: + postconditions: + - check: no plaintext API keys in config + verify: 'grep -rc ''api_key: sk-'' /root/.hermes/config.yaml' + expect: 0 matches + - check: api_key_env used for harness/litellm providers + verify: grep -c 'api_key_env.*LITELLM_API_KEY' /root/.hermes/config.yaml + expect: count > 0 + artifact: before/after verification report + verify_commands: + - 'grep -rc ''api_key: sk-'' /root/.hermes/config.yaml' + - grep -c 'api_key_env.*LITELLM_API_KEY' /root/.hermes/config.yaml + receipt: + format: json + storage: ~/.hermes/runs/hermes-key-enforcement/ + graph_node: true + schema: + contract: string + run_id: string + timestamp: ISO 8601 + agent: string + status: pass|fail|escalated + actions_taken: array + postconditions: array + drift_alerts: array + evidence_path: string + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + pause + notify: + - abiba + - mumuni + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: hermes-config-template + file: hermes-config-template.prose.md + kind: template + category: compliance + sensitivity: high + status: active + owner: abiba + version: 2.1.0 + trigger: + type: scheduled + cadence: 0 4 * * 1 + description: Weekly config drift check Monday at 4am ET + cron_job_id: null + execution: + agent: abiba + timeout: 300 + requires: [] + verification: + postconditions: + - check: agent config template_version matches template file + verify: 'grep -q ''template_version'' /root/.hermes/config.yaml && diff <(grep + ''template_version'' /root/.hermes/config.yaml | cut -d: -f2 | xargs) <(grep + ''template_version'' /root/prose-contracts/hermes-config-template.prose.md + | cut -d: -f2 | xargs) && echo match || echo mismatch' + expect: match + - check: config file is valid YAML + verify: python3 -c 'import yaml; yaml.safe_load(open("/root/.hermes/config.yaml"))' + && echo valid || echo invalid + expect: valid + artifact: config drift report + receipt: + format: json + storage: ~/.hermes/runs/hermes-config-template/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: hermes-agent-baseline + file: hermes-agent-baseline.prose.md + kind: template + category: compliance + sensitivity: normal + status: active + owner: abiba + version: 1.0.0 + trigger: + type: scheduled + cadence: 0 5 * * 1 + description: Weekly baseline verification Monday at 5am ET + cron_job_id: null + execution: + agent: abiba + timeout: 300 + requires: [] + verification: + postconditions: + - check: Hermes agent process running + verify: pgrep -f 'hermes' > /dev/null && echo running || echo stopped + expect: running + - check: agent config file exists and valid YAML + verify: test -f /root/.hermes/config.yaml && python3 -c 'import yaml; yaml.safe_load(open("/root/.hermes/config.yaml"))' + && echo valid || echo invalid + expect: valid + - check: no uncommitted changes in hermes directory + verify: cd /root/.hermes && git status --porcelain | wc -l + expect: '0' + artifact: baseline drift report + receipt: + format: json + storage: ~/.hermes/runs/hermes-agent-baseline/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: proxmox-monitor + file: proxmox-monitor.prose.md + kind: responsibility + category: monitoring + sensitivity: critical + status: active + owner: abiba + version: 1.0.0 + trigger: + type: scheduled + cadence: '*/15 * * * *' + description: Every 15 minutes + cron_job_id: null + execution: + agent: abiba + timeout: 120 + requires: [] + protocol: + - Load contract from prose-contracts/main + - Verify connectivity to all Proxmox nodes + - Execute monitoring checks per contract + - Log intermediate state + - 'On failure: stop, don''t hallucinate' + verification: + postconditions: + - check: all Proxmox nodes reachable + verify: curl -sf http://192.168.68.10:8006/api2/json/status | jq '.status' + expect: healthy + - check: no VMs in crashed state + verify: pvesh get /nodes -output-format=json | jq '.[] | select(.status=="Crashed")' + expect: empty + - check: backups running on schedule + verify: pbs-info --check + expect: last_backup < 24h ago + artifact: cluster health snapshot + verify_commands: + - curl -sf http:///status | jq '.status' + - pvesh get /nodes -output-format=json + receipt: + format: json + storage: ~/.hermes/runs/proxmox-monitor/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + trigger_remediation + notify: + - abiba + - mumuni + triggers: + - disk-gc-threat-response + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: gpu-monitor + file: gpu-monitor.prose.md + kind: responsibility + category: monitoring + sensitivity: critical + status: active + owner: abiba + version: 1.0.0 + trigger: + type: scheduled + cadence: '*/15 * * * *' + description: "Every 15 minutes \u2014 polls all GPU subsystems" + cron_job_id: null + execution: + agent: abiba + timeout: 120 + requires: [] + verification: + postconditions: + - check: GPU metrics accessible + verify: curl -sf http://localhost:9100/gpu-data + expect: 200 OK, populated data + - check: dashboard serving + verify: curl -sf http://localhost:9100/gpu-fleet.html + expect: 200 OK, HTML returned + - check: health endpoint responsive + verify: curl -sf http://localhost:9100/health + expect: 200 OK + artifact: GPU fleet snapshot + receipt: + format: json + storage: ~/.hermes/runs/gpu-monitor/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + trigger_remediation + notify: + - abiba + - mumuni + triggers: + - gpu-fleet + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: infrastructure-monitoring + file: infrastructure-monitoring.prose.md + kind: function + category: monitoring + sensitivity: high + status: active + owner: abiba + version: 1.0.0 + trigger: + type: scheduled + cadence: '*/30 * * * *' + description: Every 30 minutes + cron_job_id: null + execution: + agent: abiba + timeout: 180 + requires: [] + verification: + postconditions: + - check: Proxmox API reachable + verify: curl -sf http://192.168.68.10:8006/api2/json + expect: 200 OK + - check: Zulip API reachable + verify: curl -sf https://chat.sysloggh.net/api/v1/me + expect: 200 OK + - check: LiteLLM proxy reachable + verify: curl -sf http://192.168.68.116/litellm/v1/models + expect: 200 OK + - check: Gitea API reachable + verify: curl -sf https://git.sysloggh.net/api/v1/version + expect: 200 OK + - check: SearXNG reachable + verify: curl -sf http://192.168.68.17:8080 + expect: 200 OK + artifact: infrastructure health report + receipt: + format: json + storage: ~/.hermes/runs/infrastructure-monitoring/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + trigger_remediation + notify: + - abiba + - mumuni + triggers: + - litellm-health + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: zulip-health + file: zulip-health.prose.md + kind: responsibility + category: monitoring + sensitivity: high + status: active + owner: abiba + version: 3.0.0 + trigger: + type: scheduled + cadence: '*/15 * * * *' + description: "Every 15 minutes \u2014 monitors all Zulip-connected agents" + cron_job_id: null + execution: + agent: abiba + timeout: 120 + requires: + - Zulip API key for abiba-bot@chat.sysloggh.net + - SSH access to all Hermes agents + verification: + postconditions: + - check: bot registration active + verify: curl -sf https://chat.sysloggh.net/api/v1/me | jq '.user_id' + expect: bot_id present + - check: DM delivery working + verify: curl -sf https://chat.sysloggh.net/api/v1/users/me/is-online + expect: 'online: true' + artifact: Zulip mesh health report + receipt: + format: json + storage: ~/.hermes/runs/zulip-health/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + trigger_remediation + notify: + - abiba + - mumuni + triggers: + - hermes-zulip-restore + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: litellm-health + file: litellm-health.prose.md + kind: function + category: monitoring + sensitivity: critical + status: active + owner: abiba + version: 1.0.0 + trigger: + type: scheduled + cadence: '*/10 * * * *' + description: "Every 10 minutes \u2014 LiteLLM proxy health" + cron_job_id: null + execution: + agent: abiba + timeout: 60 + requires: [] + verification: + postconditions: + - check: LiteLLM proxy reachable + verify: curl -sf http://192.168.68.116/litellm/v1/models + expect: 200 OK, models returned + - check: router deprecated, nginx routes work + verify: curl -sf https://litellm.sysloggh.net/v1/models + expect: 200 OK (via nginx) + artifact: LiteLLM health snapshot + receipt: + format: json + storage: ~/.hermes/runs/litellm-health/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + trigger_remediation + notify: + - abiba + - mumuni + triggers: + - litellm-self-heal + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: litellm-self-heal + file: litellm-self-heal.prose.md + kind: responsibility + category: remediation + sensitivity: critical + status: active + owner: abiba + version: 1.0.0 + trigger: + type: event_driven + description: Triggered by relay message from litellm-health or infrastructure-monitoring + relay_message_type: remediation:litellm-self-heal + cron_job_id: null + execution: + agent: abiba + timeout: 600 + requires: + - Relay message with failure details from monitoring contract + verification: + postconditions: + - check: LiteLLM proxy responding + verify: curl -sf http://192.168.68.116/litellm/v1/models + expect: 200 OK + artifact: self-heal run report + receipt: + format: json + storage: ~/.hermes/runs/litellm-self-heal/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + pause + notify: + - abiba + - mumuni + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: pm2-self-heal + file: pm2-self-heal.prose.md + kind: responsibility + category: remediation + sensitivity: high + status: active + owner: abiba + version: 1.0.0 + trigger: + type: event_driven + description: Triggered by relay message when PM2 process crashes + relay_message_type: remediation:pm2-self-heal + cron_job_id: null + execution: + agent: abiba + timeout: 300 + requires: + - Relay message with process name and failure details + verification: + postconditions: + - check: PM2 process running + verify: pm2 list | grep + expect: online + artifact: self-heal run report + receipt: + format: json + storage: ~/.hermes/runs/pm2-self-heal/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: disk-gc-threat-response + file: disk-gc-threat-response.prose.md + kind: responsibility + category: remediation + sensitivity: critical + status: active + owner: abiba + version: 1.0.0 + trigger: + type: event_driven + description: Triggered by relay message from proxmox-monitor when disk >80% + relay_message_type: remediation:disk-gc-threat-response + cron_job_id: null + execution: + agent: abiba + timeout: 900 + requires: + - Relay message with target CT/host and current disk usage + protocol: + - Load contract from prose-contracts/main + - Verify target node reachable + - Run garbage collection per contract SOP + - Verify disk usage reduced + verification: + postconditions: + - check: disk usage below 80% + verify: df -h + expect: <80% + - check: no critical services stopped + verify: ps aux | grep + expect: all running + artifact: GC run report with reclaimed space + receipt: + format: json + storage: ~/.hermes/runs/disk-gc-threat-response/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + pause + notify: + - abiba + - mumuni + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: hermes-zulip-restore + file: hermes-zulip-restore.prose.md + kind: function + category: remediation + sensitivity: high + status: active + owner: abiba + version: 1.0.0 + trigger: + type: event_driven + description: Triggered by relay message from zulip-health when bot registration + fails + relay_message_type: remediation:hermes-zulip-restore + cron_job_id: null + execution: + agent: abiba + timeout: 600 + requires: + - Relay message with failure details + verification: + postconditions: + - check: Zulip bot registration active + verify: curl -sf https://chat.sysloggh.net/api/v1/me | jq '.user_id' + expect: bot_id present + - check: DM delivery working + verify: send test DM + expect: message delivered + artifact: restore run report + receipt: + format: json + storage: ~/.hermes/runs/hermes-zulip-restore/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + pause + notify: + - abiba + - mumuni + fatal: + action: relay_alert + pause + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: hermes-zulip-plugin + file: hermes-zulip-plugin.prose.md + kind: function + category: build_deploy + sensitivity: high + status: active + owner: abiba + version: 1.0.0 + trigger: + type: on_demand + description: Manual or change request to update Zulip plugin + cron_job_id: null + execution: + agent: abiba + timeout: 1800 + requires: [] + verification: + postconditions: + - check: 3 adapter files present + verify: ls -la /usr/local/lib/hermes-agent/plugins/platforms/zulip/ + expect: 3 files + - check: gateway connected + verify: ps aux | grep hermes + expect: zulip connected + artifact: deploy verification report + receipt: + format: json + storage: ~/.hermes/runs/hermes-zulip-plugin/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 1 + window: 7200 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: build-zulip-plugin + file: build-zulip-plugin.prose.md + kind: responsibility + category: build_deploy + sensitivity: high + status: active + owner: abiba + version: 1.0.0 + trigger: + type: on_demand + description: Generate or improve Zulip plugin via iterative refinement + cron_job_id: null + execution: + agent: abiba + timeout: 3600 + requires: [] + verification: + postconditions: + - check: plugin builds without errors + verify: npm install && npm run build + expect: 0 errors + - check: connectivity tests pass + verify: run plugin selftest + expect: all checks green + artifact: plugin version artifact + receipt: + format: json + storage: ~/.hermes/runs/build-zulip-plugin/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 1 + window: 7200 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: stirling-pdf-agent-access + file: stirling-pdf-agent-access.prose.md + kind: function + category: build_deploy + sensitivity: normal + status: active + owner: abiba + version: 1.0.0 + trigger: + type: on_demand + description: Grant agent access to Stirling PDF + cron_job_id: null + execution: + agent: abiba + timeout: 300 + requires: [] + verification: + postconditions: + - check: agent can access Stirling PDF + verify: curl -sf https://stirling.sysloggh.net + expect: 200 OK + artifact: access verification report + receipt: + format: json + storage: ~/.hermes/runs/stirling-pdf-agent-access/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 1 + window: 7200 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: memory-audit-maintenance + file: memory-audit-maintenance.prose.md + kind: responsibility + category: maintenance + sensitivity: normal + status: active + owner: mumuni + version: 1.0.0 + trigger: + type: scheduled + cadence: 0 3 * * * + description: Daily at 3am ET + cron_job_id: null + execution: + agent: mumuni + timeout: 600 + requires: [] + protocol: + - Load contract from prose-contracts/main + - 'Verify prerequisites: memory files accessible' + - Execute memory audit per contract SOP + - Log actions to ~/.hermes/runs/memory-audit-maintenance/ + verification: + postconditions: + - check: memory files below 80% capacity + verify: wc -l ~/.hermes/memories/*.md + expect: total lines < threshold + - check: no stale entries + verify: grep -r 'STALE' ~/.hermes/memories/ + expect: 0 matches + artifact: memory audit report + receipt: + format: json + storage: ~/.hermes/runs/memory-audit-maintenance/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - mumuni + critical: + action: relay_alert + notify: + - mumuni + - abiba + fatal: + action: relay_alert + human_required + notify: + - mumuni + - abiba + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: gpu-fleet + file: gpu-fleet.prose.md + kind: responsibility + category: maintenance + sensitivity: critical + status: active + owner: abiba + version: 1.0.0 + trigger: + type: event_driven + description: Triggered on model add/remove, GPU health degradation, agent key + rotation + cron_job_id: null + execution: + agent: abiba + timeout: 1800 + requires: [] + verification: + postconditions: + - check: all GPUs reported to LiteLLM + verify: curl -sf http://192.168.68.116/litellm/v1/models | jq '.data | length' + expect: count matches expected + - check: router deprecated + verify: check nginx routes + expect: no /v1/ prefix routes + artifact: GPU fleet status report + receipt: + format: json + storage: ~/.hermes/runs/gpu-fleet/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + - mumuni + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: infrastructure-update + file: infrastructure-update.prose.md + kind: responsibility + category: maintenance + sensitivity: high + status: active + owner: abiba + version: 1.0.0 + trigger: + type: scheduled + cadence: 0 2 * * 0 + description: Weekly system updates Sunday at 2am ET + cron_job_id: null + execution: + agent: abiba + timeout: 3600 + requires: [] + verification: + postconditions: + - check: all services running after update + verify: systemctl list-units --state=running + expect: all critical services + artifact: update report with changelog + receipt: + format: json + storage: ~/.hermes/runs/infrastructure-update/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 1 + window: 7200 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: infrastructure-control + file: infrastructure-control.prose.md + kind: pattern + category: reference + sensitivity: critical + status: active + owner: abiba + version: 1.0.0 + trigger: + type: passive + description: Loaded on-demand by executing agents as source of truth + cron_job_id: null + execution: + agent: none + timeout: 0 + requires: [] + verification: + postconditions: [] + artifact: none + receipt: + format: none + storage: none + graph_node: false + escalation: + info: + action: none + notify: [] + warning: + action: none + notify: [] + critical: + action: none + notify: [] + fatal: + action: none + notify: [] + circuit_breaker: + max_retries: 0 + window: 0 + trip_action: none + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: ra-h-os-custodianship-contract + file: ra-h-os-custodianship-contract.prose.md + kind: pattern + category: reference + sensitivity: high + status: active + owner: mumuni + version: 1.0.0 + trigger: + type: passive + description: Loaded on-demand by agents needing RA-H OS guidance + cron_job_id: null + execution: + agent: none + timeout: 0 + requires: [] + verification: + postconditions: [] + artifact: none + receipt: + format: none + storage: none + graph_node: false + escalation: + info: + action: none + notify: [] + warning: + action: none + notify: [] + critical: + action: none + notify: [] + fatal: + action: none + notify: [] + circuit_breaker: + max_retries: 0 + window: 0 + trip_action: none + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: mumuni-delegation + file: mumuni-delegation-prose-contract.prose.md + kind: pattern + category: reference + sensitivity: high + status: active + owner: mumuni + version: 1.0.0 + trigger: + type: passive + description: Loaded by Mumuni when delegating tasks + cron_job_id: null + execution: + agent: none + timeout: 0 + requires: [] + verification: + postconditions: [] + artifact: none + receipt: + format: none + storage: none + graph_node: false + escalation: + info: + action: none + notify: [] + warning: + action: none + notify: [] + critical: + action: none + notify: [] + fatal: + action: none + notify: [] + circuit_breaker: + max_retries: 0 + window: 0 + trip_action: none + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: zulip-adapter-lessons + file: zulip-adapter-lessons.prose.md + kind: pattern + category: reference + sensitivity: retired + status: historical + owner: abiba + version: 1.0.0 + trigger: + type: passive + description: "Historical reference \u2014 pi Zulip extension retired 2026-07-04" + cron_job_id: null + execution: + agent: none + timeout: 0 + requires: [] + verification: + postconditions: [] + artifact: none + receipt: + format: none + storage: none + graph_node: false + escalation: + info: + action: none + notify: [] + warning: + action: none + notify: [] + critical: + action: none + notify: [] + fatal: + action: none + notify: [] + circuit_breaker: + max_retries: 0 + window: 0 + trip_action: none + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: pi-approval-architecture + file: pi-approval-architecture.prose.md + kind: architecture + category: reference + sensitivity: retired + status: historical + owner: abiba + version: 1.0.0 + trigger: + type: passive + description: "Historical reference \u2014 pi Zulip extension retired" + cron_job_id: null + execution: + agent: none + timeout: 0 + requires: [] + verification: + postconditions: [] + artifact: none + receipt: + format: none + storage: none + graph_node: false + escalation: + info: + action: none + notify: [] + warning: + action: none + notify: [] + critical: + action: none + notify: [] + fatal: + action: none + notify: [] + circuit_breaker: + max_retries: 0 + window: 0 + trip_action: none + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: zulip-approval-fix + file: zulip-approval-fix.prose.md + kind: responsibility + category: reference + sensitivity: normal + status: active + owner: abiba + version: 2.0.0 + trigger: + type: passive + description: Loaded on-demand when /approve or /deny commands fail + cron_job_id: null + execution: + agent: none + timeout: 0 + requires: [] + verification: + postconditions: [] + artifact: none + receipt: + format: none + storage: none + graph_node: false + escalation: + info: + action: none + notify: [] + warning: + action: none + notify: [] + critical: + action: none + notify: [] + fatal: + action: none + notify: [] + circuit_breaker: + max_retries: 0 + window: 0 + trip_action: none + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: zulip-oidc-redirect-fix + file: zulip-oidc-redirect-fix.prose.md + kind: responsibility + category: reference + sensitivity: normal + status: active + owner: abiba + version: 1.0.0 + trigger: + type: event_driven + description: Triggered when Zulip OIDC login returns redirect URI error + cron_job_id: null + execution: + agent: abiba + timeout: 300 + requires: [] + verification: + postconditions: + - check: OIDC redirect uses public domain + verify: grep redirect_uri /opt/zulip/zulip_env.py + expect: chat.sysloggh.net + artifact: fix verification report + receipt: + format: json + storage: ~/.hermes/runs/zulip-oidc-redirect-fix/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: + - abiba + critical: + action: relay_alert + notify: + - abiba + - mumuni + fatal: + action: relay_alert + human_required + notify: + - abiba + - mumuni + - kwame + circuit_breaker: + max_retries: 1 + window: 7200 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: zulip-mention-reliability + file: zulip-mention-reliability.prose.md + kind: responsibility + category: retired + sensitivity: retired + status: retired + owner: abiba + version: 1.0.0 + trigger: + type: none + description: "Retired 2026-07-04 \u2014 pi Zulip extension decommissioned" + cron_job_id: null + execution: + agent: none + timeout: 0 + requires: [] + verification: + postconditions: [] + artifact: none + receipt: + format: none + storage: none + graph_node: false + escalation: + info: + action: none + notify: [] + warning: + action: none + notify: [] + critical: + action: none + notify: [] + fatal: + action: none + notify: [] + circuit_breaker: + max_retries: 0 + window: 0 + trip_action: none + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: zulip-self-heal + file: zulip-self-heal.prose.md + kind: responsibility + category: retired + sensitivity: retired + status: retired + owner: abiba + version: 1.0.0 + trigger: + type: none + description: "Retired 2026-07-04 \u2014 pi Zulip extension decommissioned" + cron_job_id: null + execution: + agent: none + timeout: 0 + requires: [] + verification: + postconditions: [] + artifact: none + receipt: + format: none + storage: none + graph_node: false + escalation: + info: + action: none + notify: [] + warning: + action: none + notify: [] + critical: + action: none + notify: [] + fatal: + action: none + notify: [] + circuit_breaker: + max_retries: 0 + window: 0 + trip_action: none + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] +- name: hello-world + file: hello-world.prose.md + kind: function + category: reference + sensitivity: normal + status: active + owner: kwame + version: 1.0.0 + trigger: + type: on_demand + description: Test contract for OpenProse on pi + cron_job_id: null + execution: + agent: kwame + timeout: 60 + requires: [] + verification: + postconditions: + - check: greeting returned + verify: contract output contains 'Hello' + expect: pass + artifact: greeting output + receipt: + format: json + storage: ~/.hermes/runs/hello-world/ + graph_node: true + escalation: + info: + action: log_to_receipt + notify: [] + warning: + action: relay_alert + notify: [] + critical: + action: relay_alert + notify: + - mumuni + fatal: + action: relay_alert + human_required + notify: + - mumuni + - kwame + circuit_breaker: + max_retries: 3 + window: 3600 + trip_action: escalate_to_fatal + depends_on: [] + last_run: null + last_status: null + drift_alerts: [] diff --git a/cron-prompts-review.md b/cron-prompts-review.md new file mode 100644 index 0000000..9275733 --- /dev/null +++ b/cron-prompts-review.md @@ -0,0 +1,612 @@ +# Cron Prompts Review — All 10 Scheduled Contracts + +Generated: 2026-07-13 20:59:18 ET + +--- + +## hermes-key-enforcement + +**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 6 * * * + +``` +Contract Enforcement: hermes-key-enforcement + +Category: compliance +Domain: hermes-agent +Owner: abiba +Schedule: Daily compliance scan at 6am ET + +This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found. + +Steps: +1. Load contract from prose-contracts/main (file: hermes-key-enforcement.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/hermes-key-enforcement/ + +Postconditions to verify: +[ + { + "check": "no plaintext API keys in config", + "verify": "grep -rc 'api_key: sk-' /root/.hermes/config.yaml", + "expect": "0 matches" + }, + { + "check": "api_key_env used for harness/litellm providers", + "verify": "grep -c 'api_key_env.*LITELLM_API_KEY' /root/.hermes/config.yaml", + "expect": "count > 0" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert + pause +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/hermes-key-enforcement/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-key-enforcement, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## hermes-config-template + +**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 4 * * 1 + +``` +Contract Enforcement: hermes-config-template + +Category: compliance +Domain: hermes-agent +Owner: abiba +Schedule: Weekly config drift check Monday at 4am ET + +This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found. + +Steps: +1. Load contract from prose-contracts/main (file: hermes-config-template.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/hermes-config-template/ + +Postconditions to verify: +[ + { + "check": "agent config template_version matches template file", + "verify": "grep -q 'template_version' /root/.hermes/config.yaml && diff <(grep 'template_version' /root/.hermes/config.yaml | cut -d: -f2 | xargs) <(grep 'template_version' /root/prose-contracts/hermes-config-template.prose.md | cut -d: -f2 | xargs) && echo match || echo mismatch", + "expect": "match" + }, + { + "check": "config file is valid YAML", + "verify": "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid", + "expect": "valid" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/hermes-config-template/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-config-template, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## hermes-agent-baseline + +**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 5 * * 1 + +``` +Contract Enforcement: hermes-agent-baseline + +Category: compliance +Domain: hermes-agent +Owner: abiba +Schedule: Weekly baseline verification Monday at 5am ET + +This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found. + +Steps: +1. Load contract from prose-contracts/main (file: hermes-agent-baseline.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/hermes-agent-baseline/ + +Postconditions to verify: +[ + { + "check": "Hermes agent process running", + "verify": "pgrep -f 'hermes' > /dev/null && echo running || echo stopped", + "expect": "running" + }, + { + "check": "agent config file exists and valid YAML", + "verify": "test -f /root/.hermes/config.yaml && python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid", + "expect": "valid" + }, + { + "check": "no uncommitted changes in hermes directory", + "verify": "cd /root/.hermes && git status --porcelain | wc -l", + "expect": "0" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/hermes-agent-baseline/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-agent-baseline, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## proxmox-monitor + +**Category:** monitoring | **Domain:** proxmox | **Owner:** abiba | **Schedule:** */15 * * * * + +``` +Contract Enforcement: proxmox-monitor + +Category: monitoring +Domain: proxmox +Owner: abiba +Schedule: Every 15 minutes + +This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state. + +Steps: +1. Load contract from prose-contracts/main (file: proxmox-monitor.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/proxmox-monitor/ + +Postconditions to verify: +[ + { + "check": "all Proxmox nodes reachable", + "verify": "curl -sf http://192.168.68.10:8006/api2/json/status | jq '.status'", + "expect": "healthy" + }, + { + "check": "no VMs in crashed state", + "verify": "pvesh get /nodes -output-format=json | jq '.[] | select(.status==\"Crashed\")'", + "expect": "empty" + }, + { + "check": "backups running on schedule", + "verify": "pbs-info --check", + "expect": "last_backup < 24h ago" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba, mumuni → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/proxmox-monitor/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=proxmox-monitor, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## gpu-monitor + +**Category:** monitoring | **Domain:** gpu | **Owner:** abiba | **Schedule:** */15 * * * * + +``` +Contract Enforcement: gpu-monitor + +Category: monitoring +Domain: gpu +Owner: abiba +Schedule: Every 15 minutes — polls all GPU subsystems + +This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state. + +Steps: +1. Load contract from prose-contracts/main (file: gpu-monitor.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/gpu-monitor/ + +Postconditions to verify: +[ + { + "check": "GPU metrics accessible", + "verify": "curl -sf http://localhost:9100/gpu-data", + "expect": "200 OK, populated data" + }, + { + "check": "dashboard serving", + "verify": "curl -sf http://localhost:9100/gpu-fleet.html", + "expect": "200 OK, HTML returned" + }, + { + "check": "health endpoint responsive", + "verify": "curl -sf http://localhost:9100/health", + "expect": "200 OK" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba, mumuni → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/gpu-monitor/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=gpu-monitor, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## infrastructure-monitoring + +**Category:** monitoring | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** */30 * * * * + +``` +Contract Enforcement: infrastructure-monitoring + +Category: monitoring +Domain: infrastructure +Owner: abiba +Schedule: Every 30 minutes + +This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state. + +Steps: +1. Load contract from prose-contracts/main (file: infrastructure-monitoring.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/infrastructure-monitoring/ + +Postconditions to verify: +[ + { + "check": "Proxmox API reachable", + "verify": "curl -sf http://192.168.68.10:8006/api2/json", + "expect": "200 OK" + }, + { + "check": "Zulip API reachable", + "verify": "curl -sf https://chat.sysloggh.net/api/v1/me", + "expect": "200 OK" + }, + { + "check": "LiteLLM proxy reachable", + "verify": "curl -sf http://192.168.68.116/litellm/v1/models", + "expect": "200 OK" + }, + { + "check": "Gitea API reachable", + "verify": "curl -sf https://git.sysloggh.net/api/v1/version", + "expect": "200 OK" + }, + { + "check": "SearXNG reachable", + "verify": "curl -sf http://192.168.68.17:8080", + "expect": "200 OK" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba, mumuni → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/infrastructure-monitoring/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-monitoring, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## zulip-health + +**Category:** monitoring | **Domain:** zulip | **Owner:** abiba | **Schedule:** */15 * * * * + +``` +Contract Enforcement: zulip-health + +Category: monitoring +Domain: zulip +Owner: abiba +Schedule: Every 15 minutes — monitors all Zulip-connected agents + +This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state. + +Steps: +1. Load contract from prose-contracts/main (file: zulip-health.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/zulip-health/ + +Postconditions to verify: +[ + { + "check": "bot registration active", + "verify": "curl -sf https://chat.sysloggh.net/api/v1/me | jq '.user_id'", + "expect": "bot_id present" + }, + { + "check": "DM delivery working", + "verify": "curl -sf https://chat.sysloggh.net/api/v1/users/me/is-online", + "expect": "online: true" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba, mumuni → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/zulip-health/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=zulip-health, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## litellm-health + +**Category:** monitoring | **Domain:** litellm | **Owner:** abiba | **Schedule:** */10 * * * * + +``` +Contract Enforcement: litellm-health + +Category: monitoring +Domain: litellm +Owner: abiba +Schedule: Every 10 minutes — LiteLLM proxy health + +This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state. + +Steps: +1. Load contract from prose-contracts/main (file: litellm-health.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/litellm-health/ + +Postconditions to verify: +[ + { + "check": "LiteLLM proxy reachable", + "verify": "curl -sf http://192.168.68.116/litellm/v1/models", + "expect": "200 OK, models returned" + }, + { + "check": "router deprecated, nginx routes work", + "verify": "curl -sf https://litellm.sysloggh.net/v1/models", + "expect": "200 OK (via nginx)" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba, mumuni → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/litellm-health/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=litellm-health, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## memory-audit-maintenance + +**Category:** maintenance | **Domain:** memory | **Owner:** mumuni | **Schedule:** 0 3 * * * + +``` +Contract Enforcement: memory-audit-maintenance + +Category: maintenance +Domain: memory +Owner: mumuni +Schedule: Daily at 3am ET + +This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found. + +Steps: +1. Load contract from prose-contracts/main (file: memory-audit-maintenance.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/memory-audit-maintenance/ + +Postconditions to verify: +[ + { + "check": "memory files below 80% capacity", + "verify": "wc -l ~/.hermes/memories/*.md", + "expect": "total lines < threshold" + }, + { + "check": "no stale entries", + "verify": "grep -r 'STALE' ~/.hermes/memories/", + "expect": "0 matches" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify mumuni → action: relay_alert +- CRITICAL: notify mumuni, abiba → action: relay_alert +- FATAL: notify mumuni, abiba, kwame → action: relay_alert + human_required + +Circuit Breaker: +- Max retries: 3 per 60.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/memory-audit-maintenance/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=memory-audit-maintenance, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + +--- + +## infrastructure-update + +**Category:** maintenance | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** 0 2 * * 0 + +``` +Contract Enforcement: infrastructure-update + +Category: maintenance +Domain: infrastructure +Owner: abiba +Schedule: Weekly system updates Sunday at 2am ET + +This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found. + +Steps: +1. Load contract from prose-contracts/main (file: infrastructure-update.prose.md) +2. Verify prerequisites (connectivity, tools, deps) +3. Execute contract per SOP +4. Run postconditions from contract registry +5. Generate receipt with status (pass/fail/escalated) +6. If any postcondition fails, escalate per contract escalation tiers +7. Log to ~/.hermes/runs/infrastructure-update/ + +Postconditions to verify: +[ + { + "check": "all services running after update", + "verify": "systemctl list-units --state=running", + "expect": "all critical services" + } +] + +Escalation Tiers: +- INFO: notify nobody → action: log_to_receipt +- WARNING: notify abiba → action: relay_alert +- CRITICAL: notify abiba, mumuni → action: relay_alert +- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required + +Circuit Breaker: +- Max retries: 1 per 120.0min window +- On trip: escalate_to_fatal + +Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path +Receipt storage: ~/.hermes/runs/infrastructure-update/receipt-{timestamp}.json +Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-update, status= + +If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution. + +IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward. +``` + + +--- +*End of review* diff --git a/delegation-prose-contract.prose.md b/delegation-prose-contract.prose.md deleted file mode 100644 index c42f9fc..0000000 --- a/delegation-prose-contract.prose.md +++ /dev/null @@ -1,256 +0,0 @@ ---- -kind: pattern -name: delegation-prose-contract -description: > - Manager (Mumuni) operating doctrine for task decomposition, worker - delegation, verification, and delivery. Defines when to delegate, which - worker to use for what, how to handle failures, and the kanban board - protocol. Enforces context-window discipline and separation of concerns. - Runs on Mumuni (CT 118, storepve, .6) via Hermes agent. -version: 1.0.0 ---- - -## Maintains - -- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`, - `syslog-research`, `syslog-review`, `syslog-writer`) -- Kanban board state at `~/.hermes/kanban/kanban.json` -- Context window budget: ~65K tokens per request (131K total, 60% threshold) - -## Topology - -**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve) -**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent -**Workers:** 6 profiles, all running on the same agent — no separate hosts needed - -This contract is infrastructure-agnostic in terms of which nodes are used. -Workers execute tasks on whatever infrastructure they're given — SSH to .6, -.pm, .9, .12, or .15 depending on the task. The contract defines the -**who** and **when** — not the **where**. - -## Why This Matters - -Without enforced delegation, the manager consumes the full iteration budget -(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs — -leaving no capacity for actual coordination. The result: context overflow -(59K tokens in system prompt), iteration exhaustion, and degraded response -quality. This contract exists because I blew through my budget checking -Proxmox node status instead of delegating to `syslog-devops`. - -## Context Window Discipline - -**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).** -**Volatile (MEMORY.md + USER.md): ~300 tokens.** -**Total base: ~6,800 tokens per request.** - -The remaining budget is the conversation. Every tool call result adds to it. -If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output -from multiple nodes), the context fills fast. That's why we delegate: workers -process in isolation and return compact results. - -## Trigger Conditions - -Delegation is **mandatory** when any of these apply: - -| Condition | Threshold | Example | -|-----------|-----------|---------| -| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report | -| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs | -| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter | -| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` | -| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis | -| Code builds or changes | Writing or modifying code | Scripts, configs, patches | -| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver | - -**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`, -`curl`, `hermes tools list` — these are decision-making tools. The manager -reads them directly. - -## Worker Selection Matrix - -| Worker | Model | Toolsets | Role | Use When | -|--------|-------|----------|------|----------| -| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files | -| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks | -| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations | -| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs | -| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings | -| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting | - -### Selection Rules - -1. **Match specialty first.** A code task → `syslog-code`. An infra task → - `syslog-devops`. Don't put a `syslog-email` worker on a code review. -2. **Research tasks with browser needs → `syslog-research`.** Other workers - don't have the browser toolset. -3. **Verification → `syslog-review`.** Never deliver raw worker output. -4. **Documentation/content → `syslog-writer`.** Let them own the prose. -5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset - (includes browser) and high reasoning effort. - -## Delegation Protocol - -### Step 1: Decompose - -Break the task into lanes. Each lane does ONE thing. Workers are independent — -no lane depends on another's output mid-flight. If lanes depend on each other, -dispatch sequentially. - -### Step 2: Dispatch - -Fire workers via `delegate_task`: - -**Parallel (independent lanes):** -``` -delegate_task( - tasks=[ - {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"}, - {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"}, - ] -) -``` - -**Sequential (dependent lanes):** -Dispatch lane 1 → wait for result → dispatch lane 2. - -### Step 3: Verify - -**MANDATORY for:** -- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`) -- Code builds and modifications -- Research findings (web data, external sources) -- Any output that will reach the user - -**Fire `syslog-review` to verify:** -``` -delegate_task( - goal="Review the output of the devops worker. Verify the node status - report is accurate, check for inconsistencies, confirm all nodes were - reachable.", - context="Worker was syslog-devops. Output is at /tmp/node-report.md. - Verify against live system." -) -``` - -**If verification fails:** -1. Send work back to original worker with review feedback -2. Re-verify -3. Max 2 re-verify cycles before escalating to Kwame - -### Step 4: Deliver - -Only verified results reach Kwame. Format per channel: -- Telegram: Use `telegram-formatting` skill -- Zulip: Use Zulip Markdown (CommonMark) -- Email: Use `syslog-email` skill - -## Kanban Board Protocol - -**File:** `~/.hermes/kanban/kanban.json` - -```json -{ - "task_id": "unique-id", - "title": "Task description", - "created": "2026-07-09T01:00:00", - "status": "backlog|in_progress|review|done", - "lanes": [ - { - "lane_id": "devops-check", - "worker": "syslog-devops", - "goal": "Check all 5 Proxmox nodes", - "status": "dispatched|completed|failed", - "output_file": "/tmp/node-report.md" - } - ] -} -``` - -**Update the board on every state change.** - -## Failure Handling - -### Worker Timeouts - -- Child timeout: **900 seconds** (15 minutes) -- Worker model `syslog-auto` is slow — it can hit the timeout limit with - 22+ API calls -- **If a worker times out:** Re-dispatch with a narrower scope. Break the - task into smaller pieces that fit in the timeout window. -- **Avoid delegating sequential SSH hops** — each SSH connection adds latency - that compounds quickly. Prefer API-based or local approaches when possible. - -### Worker Selection Failures - -- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker) -- `syslog-code` is best for code-level work (reading files, writing scripts) -- `syslog-research` has the browser toolset — use for web research -- `syslog-review` is the QA gate — always fire before delivery -- **Never fire more than 3 parallel workers** (max_concurrent_children: 3) -- **Never nest delegation** (max_spawn_depth: 1) - -### Context Overflow - -- If a task requires >10K tokens of output, delegate the processing -- Workers return compact summaries, not raw data dumps -- Pass file paths and concrete goals — never dump raw data into context - -## Anti-patterns - -- ❌ Reading large files into your own context before deciding → delegate the read -- ❌ Carrying SSH/grep/output results in your context → delegate the analysis -- ❌ Doing work yourself and then "pretending" to delegate → the user can tell -- ❌ Skipping verification → raw worker output never reaches the user -- ❌ Delegating single tool calls → keep quick reads/writes at manager level -- ❌ Firing more than 3 workers in parallel → hard limit - -## Emergency Exception - -**In an emergency (server down, service must be restored immediately):** -- Delegate the diagnosis (find the problem) -- Execute the fix yourself (minimize handoff latency) -- Verify the fix after delivery -- Log the exception in the kanban board - -The emergency exception exists because the user needs the service back NOW, -not after three worker round-trips. But it's an exception — not the rule. - -## What This Contract Doesn't Cover - -1. **Worker profile configuration** — covered by `hermes-config-template.prose.md` -2. **SSH key management** — covered by existing SSH/Proxmox contracts -3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo -4. **Cron job management** — covered by individual cron contracts -5. **Infra verification** — covered by `verify-before-mutate` protocol - -## Verification - -Run `scripts/worker-audit.py` to verify all 6 profiles are aligned: - -```bash -python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py -``` - -## References - -- `kanban-orchestrator` skill: The operational playbook (detailed execution steps) -- `worker-profile-audit.md` (skill reference): Worker configuration audit notes -- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns -- `verify-before-mutate` protocol: Infrastructure change verification -- `hermes-config-template.prose.md`: Worker profile configuration - -## Success Criteria - -This contract succeeds when: - -1. **No context overflow** — single-turn tasks don't exhaust the iteration budget -2. **Workers do the work** — manager coordinates, doesn't execute -3. **Verification before delivery** — all output passes through `syslog-review` -4. **Kanban board is current** — every task has a lane, every lane has a status -5. **User gets verified results** — raw worker output never reaches Kwame - ---- - -**Last updated:** 2026-07-09 -**Author:** Mumuni (with Kwame's input on triggers and exception criteria) -**Status:** Draft — awaiting PR review and merge to prose-contracts main diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 390c36d..8946c9f 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -5,13 +5,12 @@ description: > Manages the GPU inference fleet across all hosts. Handles model deployment, registration, health checks, LiteLLM sync, agent key management, GPU saturation watchdog, Prometheus/Grafana monitoring, and self-healing. - UPDATED 2026-07-12: Architecture is DIRECT GPU — LiteLLM routes directly - to llama-server on each GPU host (no router in inference path). Router - (port 9000) is running but NOT in request path. All GPUs standardized on - api-key 'not-needed'. RTX 5070 had api-key mismatch (sk-loc...5678) that - caused cascading 401→timeout→401 fallback loops — fixed. - Context: RTX 3090 verified at 256K (was documented as 128K — WRONG). - Workload: Compression moved to Strix Halo, RTX 5070 → vision/web only. + UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe, + gpu-dense, gpu-light. These never change — only the underlying model does. + Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB). + RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster). + RTX 5070 context: 131K → 256K. VRAM: 88% (10.8/12.2GB). + Compression timeout: 300s (was 120s). Mumuni context: 128K (was 256K). agent: abiba triggers: - on model add/remove @@ -68,48 +67,72 @@ triggers: │ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │ │ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │ │ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │ -│ 256K ctx │ │ 131K ctx │ │ 256K ctx │ │ Watchdog │ -│ qwen3.6 │ │ gemma-4-12b │ │ ornith35B │ │ Prometheus │ -│ 27B-code │ │ :8080 │ │ :8080 │ │ exporter │ +│ 256K ctx │ │ 256K ctx │ │ 256K ctx │ │ Watchdog │ +│ qwen3.6 │ │ gemma-4-12b │ │ qwen3.6 │ │ Prometheus │ +│ 27B-code │ │ :8080 │ │ -35B-udq4 │ │ exporter │ │ :8080 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │ │ :9400 │ └─────────────┘ └───────────┘ └──────────────┘ └──────────┘ ``` -## Current Model Assignments (2026-07-12) +## Stable Role-Based Aliases (Introduced 2026-07-15) + +Agent configs, cron jobs, and workflows MUST use these aliases, never model-specific names. +When a model is swapped on a GPU, ONLY the infrastructure layer changes — agent configs are untouched. + +| Alias | GPU | Current Model | Will Route To | +|-------|-----|---------------|---------------| +| `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo | +| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 | +| `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 | + +**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work +but are deprecated for agent configs. Only the stable aliases survive model swaps. + +## Current Model Assignments (2026-07-15) | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status | |-------|-----|------|------|-----|----------|----------|-------------|--------| -| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.7/24GB (84%) | **256K** | turbo4 | 1 | default | ✅ healthy | -| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy | -| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy | +| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | 22.2/24.6GB (90%) | **256K** 🚀 | turbo4 | 2 | default | ✅ 63 tok/s | +| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 256K | q4_0 | 2 | 2048/1024 | ✅ healthy | +| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy | ## Routing Configuration (LiteLLM — July 2026) -### syslog-auto Weighted Pool +### syslog-auto Weighted Pool (Direct GPU — bypasses router) -| Model | GPU | Weight | RPM Cap | Purpose | +| Model | GPU | Weight | RPM Cap | Timeout | |-------|-----|--------|---------|---------| -| qwen3.6-27B-code | RTX 3090 | 0.55 | 500 | Heavy reasoning, code, long context | -| ornith-1.0-35b | Strix Halo | 0.30 | **60** | Agentic workflows, tool calling | -| gemma-4-12b | RTX 5070 | 0.15 | 200 | Overflow + vision | +| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** | +| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** | +| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** | + +Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path. ### Direct Model Endpoints | Model | RPM Cap | Notes | |-------|---------|-------| -| ornith-1.0-35b | 40 | Tight cap — prevents Strix overload | +| qwen3.6-35B-udq4 | 40 | Tight cap — prevents Strix overload | | qwen3.6-27B-code | 500 | High cap — primary workhorse | -| gemma-4-12b | 500 | High cap — fast 12B | +| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s | + +### Stable Aliases (for agent configs — never change) + +| Alias | RPM Cap | Routes To | Purpose | +|-------|---------|-----------|---------| +| `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) | +| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning | +| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks | ### Fallback Chains - gemma → qwen - qwen → gemma -- ornith → qwen → gemma -- syslog-auto → qwen → gemma → ornith +- qwen3.6-35B-udq4 → qwen → gemma +- syslog-auto → qwen → gemma → qwen3.6-35B-udq4 -### Why ornith RPM Is Capped -- Direct: 40 RPM (tight) — Strix Halo is shared with compression tasks +### Why Strix Halo RPM Is Capped +- Direct (qwen3.6-35B-udq4): 40 RPM (tight) — Strix Halo is shared with compression tasks - Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously - Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C @@ -163,7 +186,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!") 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models` 5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml` - - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s + - gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (strix-moe does NOT exist — legacy name, do not use) - global request_timeout: 300s, nginx proxy_read_timeout: 600s 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power) 7. Check port conflicts: verify only one llama-server on :8080 per host @@ -203,7 +226,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. | `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server | | `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard | | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) | -| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. | +| `/etc/systemd/system/strix-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo) running unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. | ## Prometheus & Grafana @@ -220,14 +243,14 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first. Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster(). - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround. -- **VRAM (2026-07-12)**: RTX 3090 at 20.7/24GB (84%), RTX 5070 at 10.0/12.2GB (82%). RTX 3090 context increased to 256K (was incorrectly documented as 128K — verified via /proc/PID/cmdline). -- **RTX 3090 runs `--parallel 1`** (verified 2026-07-12). RTX 5070 and Strix at parallel 2. -- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching`. Service: `/home/llmuser/llama-wrapper.sh`. -- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 1024 --parallel 2`. Api-key standardized to `not-needed` (was `sk-loc...5678` causing 401 loops). Service: `/home/llmuser/llama-wrapper.sh`. -- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`. -- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV. +- **VRAM (2026-07-15)**: RTX 3090 at 22.2/24.6GB (90%) with **256K context** (corrected from 131K). RTX 5070 at 10.8/12.2GB (88%) with 256K context + MTP. Strix Halo at ~9GB/64GB. +- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2). +- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context corrected to 256K (2026-07-15). VRAM: 90%. Service: `/home/llmuser/llama-wrapper.sh`. +- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 256K context. Gen speed: 122 tok/s (was 70). VRAM: 10.8/12.2GB (88%). No draft model pre-upgrade due to VRAM constraints. Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 262144`. +- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. +- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080 (was `ornith-server.service`), model changed to `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (UD-Q4_K_M), alias `qwen3.6-35B-udq4`, 256K context, flash-attn + q8 KV. MTP support enabled for 1.4-2.2x faster inference. - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). -- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. +- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116. - **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback. - **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected. @@ -237,11 +260,14 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. ## GPU Inference Benchmarks (Current) -| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples | +| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context | |-----|-------|-----------|--------------|----------|---------| -| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 | -| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 | -| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 | +| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **256K** | +| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **256K** | +| Strix Halo (.15) | qwen3.6-35B-udq4 | **71** | — | — | **256K** | + +Benchmarks from 2026-07-15 verification run. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s. +All 3 GPUs now at 256K context (2026-07-15). Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes. Degradation alerts fire at 30% (warning) and 50% (critical) below baseline. @@ -249,12 +275,55 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window. **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration. -## Agent Config Implications (2026-07-12) +## Agent Config Implications (2026-07-15) -With RTX 3090 at 256K context (verified July 2026): -- Agents using `syslog-auto` (55/30/15 qwen+ornith+gemma): `context_length: 262144` — ornith and qwen both support it -- Agents using `qwen3.6-27B-code` directly: `context_length: 262144` (256K ctx verified) -- Agents using `gemma-4-12b` directly (auxiliary tasks): `context_length: 131072` -- Compression threshold at 0.65: fires at ~170K for 262K context window on Strix Halo -- All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap -- Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented) +### Stable Aliases — CRITICAL + +All agent configs MUST use stable role-based aliases, never model-specific names: +- `compression.model: strix-moe` (NOT `qwen3.6-35B-udq4`) +- `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`) +- `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`) +- `auxiliary.web_extract.model: gpu-light` + +When the underlying model is swapped, only the LiteLLM config changes — agent configs are untouched. + +### Context Windows +- RTX 3090: **256K** (was 131K, bumped 2026-07-15) | RTX 5070: **256K** (up from 131K) | Strix Halo: **256K** +- **Mumuni compression context**: 128K (down from 256K) — ensures compression model doesn't timeout +- Compression threshold 0.65: fires at ~85K for 128K context window +- Mumuni compression model alias: `strix-moe` with 300s timeout + +### Mumuni Agent Profile + +Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs: + +| Setting | Value | Notes | +|---------|-------|-------| +| `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) | +| `model.provider` | `custom:litellm` | LiteLLM on CT116 | +| `compression.model` | `strix-moe` | Stable alias — survives model swaps | +| `aux.compression.model` | `strix-moe` | Compression auxiliary model | +| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) | +| `aux.web_extract.model` | `gpu-light` | Web extraction | +| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) | +| `context.max_context_window` | 262144 (256K) | Fixed 2026-07-16 (was 131072 — caused premature compression, WAL #1300) | +| `compression.threshold` | 0.65 | Triggers at ~85K | +| `compression.target_ratio` | 0.3 | Compresses to ~38K | +| `compression.protect_last_n` | 40 | Preserves last 40 messages | +| `memory.memory_char_limit` | 800 | Brief memory entries | +| `personalities` | `creative` | Creative assistant personality | +| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms | +| Main model timeout | 300s | LiteLLM global timeout | +| Compression model timeout | 300s | strix-moe timeout increased from 120s | + +### Agent Update Status (2026-07-15) + +| Agent | Host | Status | +|-------|------|--------| +| **Mumuni** | CT114 (.123) | ✅ Updated to stable aliases | +| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases | +| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM | +| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM | +| **Kagenz0** | CT105 | ❌ SSH unreachable — needs Zulip DM | + +All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap. diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 34cc947..2e41223 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -5,7 +5,7 @@ version: 1.0.0 description: > Canonical known-good baseline for all Syslog Hermes agents. Captures the exact configuration state, keys, workarounds, and audit procedure. When an agent's - configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide. + configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo). author: Abiba (pi agent) --- @@ -26,11 +26,11 @@ done |-------|-----|------|-----|---------------|------------|----------| | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes | | Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes | -| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** | -| Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes | -| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes | +| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** | +| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes | +| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | -> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo). +> **Note**: CT hostnames (tdunna→CT129, baggy→CT114) differ from agent identities (koby, koonimo). Access: `pct-run ` — no IPs needed. GPU hosts (.8, .110, .15) use SSH. Keys are stored in Infisical vault (project=agents, env=production) and injected at @@ -169,9 +169,9 @@ pct-run grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY ``` -### For Koby (CT 111 / tdunna) +### For Koby (CT 129 / tdunna) -Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`. +Koby runs Hermes on CT 129 (tdunna). Config files at `/root/.hermes/config.yaml`. Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above. **LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper. diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index d3fc3e7..2e73178 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -14,9 +14,15 @@ description: > expire/404. The .env fallback prevents agents from running without keys. Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours. + UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service + token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the + infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed. + Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending. + Abiba's key is now a proper agent key (NOT the master key — stale note removed). + Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml - on CT 116. Last verified: 2026-07-12. + on CT 116. Last verified: 2026-07-16. --- ## Parameters @@ -51,8 +57,8 @@ description: > - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date) - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" } - Duration is null (permanent) — inherited from litellm default_key_generate_params - - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"] - - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU) + - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"] + - Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed). - Return the new key 5. **If action == "rotate"**: - Generate new key with same alias (LiteLLM replaces the old key) @@ -67,3 +73,147 @@ description: > - Test the key against LiteLLM /v1/models - Confirm key alias matches agent_name in LiteLLM key list - Verify agent gateway uses vault wrapper: `cat /proc//cmdline` shows `infisical run` + +## Production Vault Access Process (canonical, 2026-07-16) + +The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending — +runs as user `jerome`, not systemd root, needs user-scope adaptation). + +### The canonical pattern + +1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`). +2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`). + - Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project. + - Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities). +3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`): + ```bash + #!/bin/bash + export INFISICAL_API_URL="https://vault.sysloggh.net" + TOKEN=$(cat /root/.infisical-token) + LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs + while true; do + infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \ + --env=prod --domain=https://vault.sysloggh.net -- bash -c ' + . /root/.hermes/.env 2>/dev/null # [FALLBACK Rule 3] safety net only + export LITELLM_API_KEY="$_LITELLM_API_KEY" + exec /bin/python -m hermes_cli.main gateway run + ' >> $LOG 2>&1 + sleep 5 # restart on exit + done + ``` +4. **Agent key in vault** as `_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth. +5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation. +6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot). +7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault. + +### Why this is non-fail + +- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits. +- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable. +- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway. +- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source. + +### Migration status (2026-07-16) + +| Agent | Host | Pattern | Vault key | Status | +|-------|------|---------|-----------|--------| +| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed | +| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed | +| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram | +| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed | +> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent. +| tanko | .122 | infisical-gateway.sh + service token (migrated 2026-07-16) | TANKO_LITELLM_API_KEY | ✅ vault-backed | + +### Tanko migration (pending) + +Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in +`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced). +Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`) +with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering +enabled (`loginctl enable-linger jerome`) so the user service runs without a login session. + +### Koby migration lessons (2026-07-16) + +Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper. +**Two mistakes I made that broke the agent:** +1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed + in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were + inherited from the pre-migration gateway env, not stored in any file. Lost on restart. +2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials. + Agents need ALL their platform env vars. Missing vars cause silent adapter failures. + +**How Koby actually connects (2026-07-16):** +- Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`). + Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config). +- Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026). + Allowed users: 6679773481. Home channel: 6679773481. +- Both platforms now connect through the wrapper's env injection. + +**Golden rule for gateway restarts:** always `cat /proc//environ` before killing the old +process — captures the live env set. Especially important when migrating gateways between +injection mechanisms. + +### Key rotation procedure (one vault operation with this standard) + +1. Generate new key: `POST /key/generate` (master key, admin). +2. Update vault: `infisical secrets set _LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`. +3. Update `.env` fallback: `echo '_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`. +4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live. +5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200. + +## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300) + +**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com). +In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace +key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt +repo only ships 0.38.0, so `apt upgrade` does not help. + +**Proper fix — Machine Identity (Infisical automation best practice):** +Create a machine identity with READ+WRITE scope on the `agents` project (project_id= +`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely. +Then vault writes work from any host: +```bash +# Get a machine-identity access token +TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \ + -H 'Content-Type: application/json' \ + -d '{"clientId":"","clientSecret":""}' | jq -r .accessToken) +# Write a secret via REST API v3 + curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ + -d '{"environment":"prod","secretValue":"sk-","workspaceId":"","type":"shared"}' +# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ... +``` +**2026-07-16 UPDATE — vault SYNCED + CLEANED.** The abiba service token (`st.8e848433…`, READ+WRITE) +can write to the vault. All session-13 rotated keys in vault and validate 200 against LiteLLM. +4 stale secrets deprecated, 5 personal creds flagged for separate project. Tanko migrated from +hardcoded keys to `infisical-gateway.sh` wrapper. Koonimo Zulip key restored. + +**Machine Identity:** `8ddb9438-74fc-4ab6-bd74-929e7c47b53b` exists in Infisical UI. +Client secret needed to use universal auth for vault writes from automation. + +**Service Token Inventory (2026-07-16):** +| Token ID | Name | Permissions | Used By | +|----------|------|-------------|---------| +| `st.8e848433…` | tanko-gateway | READ+WRITE | Abiba, Koonimo | +| `st.353699cd…` | tanko-agent | READ-only | Tanko | + +## Key Rotation Log + +| Date | Agent | Action | Notes | +|------|-------|--------|-------| +| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault (Tt2bUL…). Updated wrapper to inject ZULIP_API_KEY + ZULIP_EMAIL. Restarted gateway → Zulip connected as koonimo-bot@ (bot_id=17). 3 platforms now. | +| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml key to infisical-gateway.sh wrapper + service token st.353699cd… (tanko-agent). Systemd user service updated, hardcoded key drop-ins removed. Verified LITELLM_API_KEY from vault, 3 platforms connected. | +| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated: KAGENZ0_LITELLM_API_KEY, HERMES_OPENROUTER_KEY, LITELLM_API_KEY (generic duplicate), ZULIP_API_KEY (generic duplicate). 5 personal creds flagged for separate project. | +| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. | +| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. | +| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. | + +## LiteLLM Master Key (use sparingly — agents should NOT use it directly) + +- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY) +- Used for /key/generate, /key/delete, /key/list (GET), DB queries +- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key. +It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`). +The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference — +see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference". +- LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"` -- 2.54.0