root
af41f8f57a
fix: remaining template YAML and topology table corrections
...
- hermes-config-template: Template YAML compression model gemma→ornith.
Context guidance updated (qwen=256K, only gemma limited to 131K).
- litellm-self-heal: GPU topology table fixed (RTX 3090: 128K→256K, parallel 2→1)
2026-07-12 22:55:24 +00:00
root
be02b0e843
fix: stale dates, context values, and architecture references
...
- gpu-fleet: Updated all context references (128K→256K for RTX 3090).
Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram
(router labeled deprecated/not in path). Corrected VRAM values.
Updated dates: June→July 2026.
- hermes-config-template: Fixed description (removed incorrect '128K on
NVIDIA' claim). Updated to reflect verified 256K on RTX 3090.
- litellm-self-heal: Updated last-verified date.
- litellm-api-keys: Updated last-verified date.
2026-07-12 22:54:30 +00:00
root
79d4a73895
docs: lessons learned from 2026-07-12 session
...
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
Updated context values (RTX 3090=256K, not 128K). Added api-key
standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
L2: Fallback chain cascading failure loop detection
L3: Verify running state, not documentation
L4: Infisical fallback requirement (.env must have uncommented key)
L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
2026-07-12 22:49:39 +00:00
root
19b6db9891
feat: GPU workload redistribution — compression → Strix Halo
...
- Move compression model from gemma-4-12b (RTX 5070) to ornith-1.0-35b (Strix Halo)
- Add Rule 8: GPU Workload Distribution — per-GPU role assignment
- Add Rule 9: Compression Threshold for 256K models
- Update Rule 7: Auxiliary Model Consistency with new compression routing
- Add gpu-self-heal.prose.md contract with 10 remediation rules
- Strix Halo (64GB, 256K, 72.4 tok/s) → compression specialist
- RTX 5070 (12GB) → vision/web search specialist
- RTX 3090 (24GB, 256K) → heavy reasoning specialist
- All rules grilled and confirmed with Kwame 2026-07-12
2026-07-12 22:27:15 +00:00
abiba-bot
0134ad8be6
Merge pull request 'fix: cron alignment + VM→bare metal drift' ( #11 ) from fix/cron-alignment-jul2026 into master
2026-07-09 06:28:04 +00:00
root
546ca79849
fix: cron alignment + VM→bare metal drift
...
- pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract)
- pm2-self-heal: added gpu-monitor to tracked PM2 processes
- infra-monitor+control: removed daytime-only restriction (now 24×7)
- infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110
- Added @reboot pm2 resurrect for persistence across reboots
- README: zulip-monitor.sh → agent-health-check.py
- Cleaned up commented-out zulip-monitor cron line
Live actions: gpu-monitor-server added to PM2 (persistent across reboots)
2026-07-09 06:27:48 +00:00
abiba-bot
c31dbad95f
Merge pull request 'fix: IP drift and container count from live testing' ( #10 ) from fix/testing-drift-jul2026 into master
2026-07-09 06:24:07 +00:00
root
d829f595b4
fix: IP drift and container count from live testing
...
Testing the consolidated litellm-self-heal contract against live infra found:
- IP drift: pi host is .24, not .65 (confused with ra-h-os bridge)
- Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations)
- Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables
- Container count: 10 running on CT 116, not 8 as contract stated
- Added harness-docker-stats + harness-pve-exporter to litellm-self-heal
- Updated container health check list
Verified: GPU monitor restarted on .24:9100, ornith model inference working,
all 10 containers healthy.
2026-07-09 06:23:44 +00:00
abiba-bot
4164ee5ed0
Merge pull request 'feat: consolidate litellm-health into litellm-self-heal' ( #9 ) from feat/consolidate-litellm-contracts into master
2026-07-09 06:02:47 +00:00
root
053d30ce7c
feat: consolidate litellm-health into litellm-self-heal
...
- litellm-self-heal: merged health check as §Health Check section,
includes full architecture diagram (v4.0.0), GPU topology, timeout
tables, container list, and execution flow. Eliminates duplication
between the two contracts.
- litellm-health: marked DEPRECATED, retained for reference
- README.md: updated with consolidated contract, deprecation notice,
disk-gc-threat-response, and infrastructure-monitoring status
2026-07-09 06:01:48 +00:00
abiba-bot
e4ab43f1c6
Merge pull request 'fix: LiteLLM contracts — align with 2026-07-08 architecture changes' ( #8 ) from fix/litellm-contracts-jul2026 into master
2026-07-09 05:37:48 +00:00
root
a2c5a14c74
fix: LiteLLM contracts — align with 2026-07-08 architecture changes
...
- litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly),
GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s),
Strix Halo: CPU→Vulkan, context sizes + parallel slots added
- litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED
(router not in path), GPU topology synced with gpu-fleet
- litellm-api-keys: reference gpu-fleet as source of truth for keys,
removed qwen3.6-35B-A3B from model list (never deployed)
2026-07-09 05:37:16 +00:00
abiba-bot
a7fedf71c6
Merge pull request 'fix: contract improvements from 2026-07-09 run log review' ( #7 ) from fix/contract-improvements-jul2026 into master
2026-07-09 05:33:25 +00:00
root
97977f320c
fix: contract improvements from 2026-07-09 run log review
...
- pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs
- disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope,
remove jitsi (decommissioned), update access matrix with KVM VM section
- infrastructure-control: docker-vm container count 11→16, add Trove agents stack
- infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)
2026-07-09 05:32:00 +00:00
abiba-bot
a3786ab289
Merge pull request 'disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB)' ( #6 ) from feat/disk-gc-amdpve-2026-07-09 into master
2026-07-09 05:23:30 +00:00
root
15e8998289
disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs
2026-07-09 05:20:42 +00:00
root
25c6c76fb3
disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs
2026-07-09 05:17:05 +00:00
root
4819247e39
Add zulip-oidc-redirect-fix prose contract
...
Fixes Zulip OIDC authentication where redirect_uri used internal IP
(192.168.68.19) instead of public domain (chat.sysloggh.net) after
server restart. Monkey-patches social_core.strategy.BaseStrategy
to use ROOT_DOMAIN_URI for all redirect URI construction.
2026-07-09 00:02:18 +00:00
root
1dc4155d9a
fix(hermes-zulip-plugin): default to master branch, add gateway restart
...
- Switched default branch from feat/zulip-streaming to master
- Added Step 5: gateway restart with smoke check after plugin install
- Plugin changes require restart to take effect for a valid test
- Outstanding feature branch merges should be addressed post-install
2026-07-08 17:51:19 +00:00
root
09efbdf4e5
feat: add hermes-zulip-plugin (install & repair) and relocate hermes-zulip-restore to repo
...
- hermes-zulip-plugin.prose.md: installs latest zulip-platform plugin from
canonical repo, verifies _strip_html fix, sends relay success signal.
Distinct from restore — plugin-layer only, no env/gateway/connection checks.
- hermes-zulip-restore.prose.md: relocated from /root/ into prose-contracts repo.
- Removed duplicate zulip-platform-verification from /root/vault/contracts/.
2026-07-08 17:47:23 +00:00
jerome
4058010e57
Merge pull request 'feat: hermes-zulip-restore contract' ( #5 ) from feat/hermes-zulip-restore into master
...
Reviewed-on: SyslogSolution/prose-contracts#5
2026-07-08 17:38:16 +00:00
root
776df418ca
gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes
...
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3)
- .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache
- .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV
- .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm
- LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s
- Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma
- Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
2026-07-08 17:25:42 +00:00
jerome
c0191c9edc
feat: hermes-zulip-restore contract — restore Zulip on any Hermes agent
...
Deploys zulip-platform adapter to bundled plugin path, verifies
env credentials, restarts gateway, and validates Zulip connection.
Covers Mumuni (CT114), Tanko (CT112), Koby (CT111).
Includes _strip_html slash-command fix, known failure modes,
and owner-aware restart paths.
2026-07-08 03:19:01 -04:00
jerome
f6e8791352
fix(ci): change kind from reference to template in hermes-agent-baseline.prose.md
...
The validate job was failing because 'reference' is not an allowed kind.
Changed to 'template' since this document serves as a canonical baseline
template for restoring agent configurations.
2026-07-07 20:57:48 -04:00
root
b1ef4cbe9b
docs: self-update contract — Gen 6 postconditions, status clarification, state persistence
...
- Added stream_subscription_check, dm_response_time_offline_verifiable, known_issues_tracked postconditions
- Added 'Status' section clarifying active for Hermes agents only (not retired with pi Zulip)
- Added 'State Persistence' section with KG node schema for build-zulip-plugin:state
- Execution step 4: always run offline tests (simulate_dm_latency, selftest)
- Execution step 5: update KG state node after each generation
2026-07-07 07:18:45 +00:00
kagentz-bot
41ef546d21
Merge pull request 'fix: use IP 192.168.68.17:3000 instead of git.sysloggh.net for CI checkout' ( #4 ) from fix/ci-checkout-ip into master
2026-07-06 22:56:48 +00:00
Agent Zero
8f89603c31
fix: use IP 192.168.68.17:3000 instead of git.sysloggh.net hostname for CI checkout
...
The Gitea Actions runner cannot resolve the internal hostname
git.sysloggh.net during checkout. Using the direct IP address
resolves the DNS resolution issue in CI pipeline steps.
2026-07-06 18:54:12 -04:00
Agent Zero
f1a031b682
fix: Firecrawl health endpoint is / not /health
2026-07-06 18:21:30 -04:00
Agent Zero
f14b2fc04e
infra-update contract: v1.1.0 - fix CT→VM refs, remove deprecated stacks
2026-07-06 18:05:16 -04:00
Agent Zero
09ffbf10aa
infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths
2026-07-06 18:05:16 -04:00
root
5aa93117de
docs: capture all session changes — port conflict, health check, streaming
...
Updated contracts to reflect all infrastructure changes made 2026-07-05/06:
gpu-fleet.prose.md:
- Port conflict detection on all 3 GPU wrappers
- Ghost process root cause documented (.8 stale pid 25836)
infrastructure-control.prose.md:
- Section 7: Agent Health Check (consolidated, non-disruptive)
- Disabled scripts documented (zulip-watchdog, zulip-monitor)
hermes-agent-baseline.prose.md:
- Health verification section + GPU port conflict detection
- Change log updated
zulip-health.prose.md:
- Streaming support section (edit_message + PATCH API)
2026-07-06 02:46:41 +00:00
root
79855ea9e1
feat: consolidated agent health check — key validation + GPU port conflict + streaming
...
Single non-disruptive script replacing 7 scattered Zulip health checks.
Checks every 10 min via cron, never restarts anything:
- LiteLLM key validation for all 4 Hermes agents
- GPU port conflict / ghost process detection
- Agent gateway liveness + Zulip streaming status
- Recent gateway error count
Port conflict detection added to all 3 GPU wrappers:
- .8 (qwen): wrapper detects ghost on port 8080 before starting
- .110 (gemma): same pattern
- .15 (ornith): port-cleanup.sh replaces blanket pkill -x llama-server
2026-07-06 02:23:54 +00:00
root
2ea9cac23a
docs: hermes-agent-baseline — canonical good-state reference
...
Single source of truth for all 4 Hermes agents. Captures:
- Agent map (CT, IP, key, alias)
- Key architecture and config patterns
- Mandatory api_key workaround with explanation
- Known bug: auxiliary_client ignores api_key_env
- Audit procedure (full audit, leak check, key verification)
- Systemd patterns
- Attachment cache locations
2026-07-05 23:37:56 +00:00
root
35e864bdb7
audit: all 4 Hermes agents verified — no master key, api_key workaround applied
...
Agent audit (2026-07-05):
Tanko CT 112: sk-CggiHWlamQyShxWC3Hx6uw — vision+compression patched
Mumuni CT 114: sk-VrqCNlwUgzoNGOpikJ7nwQ — vision patched
Tdunna CT 111: sk-6sbCNjz2T6lTVDBdlNHXsA — vision+compression patched
Baggy CT 113: sk-krnw_zGBwvvL5b7l2t-s-A — vision+compression patched
All 4 agents: zero master key references, dedicated LiteLLM keys,
api_key workaround applied to bypass auxiliary_client bug.
2026-07-05 23:36:00 +00:00
root
53d4290886
docs: document auxiliary_client api_key_env bug + workaround
...
_resolve_task_provider_model() in agent/auxiliary_client.py ignores
api_key_env for auxiliary tasks (vision, compression). The custom
provider path resolves it, but aux tasks take a different code path.
Workaround: set api_key directly alongside api_key_env.
Applied to Tanko (vision + compression).
2026-07-05 23:17:37 +00:00
root
c8dcd8b63d
fix: Tanko key regenerated, Mumuni key verified from /etc/environment
...
Tanko: old key sk-620a05e95a... was from docker-compose router config,
not LiteLLM key DB. Deleted + regenerated via LiteLLM API:
sk-CggiHWlamQyShxWC3Hx6uw (alias: tanko, budget: $100)
Mumuni: verified via pct-run 114 — key is sk-VrqCNlwUgzoNGOpikJ7nwQ
(was previously listed as sk-XY2aUfvy2... in contracts)
2026-07-05 20:18:05 +00:00
root
5c90eb8f6c
rename: Koonimo→Baggy, Koby→Tdunna — match hostnames
...
Standardize agent names across all contracts and pct-run.sh.
No functional changes — same CT IDs, same keys, same nodes.
2026-07-05 20:04:22 +00:00
root
b65116b847
feat: pct-run.sh — run commands in CTs by ID, no IPs needed
...
Adds /scripts/pct-run.sh that maps CT IDs to PVE nodes and runs
pct exec directly. No hardcoded IPs — Proxmox is the source of truth.
Updates infrastructure-control with CT access table using pct-run.
Usage examples:
pct-run 112 cat /etc/hostname # Tanko
pct-run 114 grep LITELLM /etc/environment # Mumuni key check
pct-run 116 docker ps # LiteLLM containers
2026-07-05 20:02:20 +00:00
root
0db95a4d1c
fix: correct IPs, CT IDs, and key status across all contracts
...
hermes-key-enforcement.prose.md:
- Add IP column to verified agents table
- Fix Tanko status: was using master key via systemd drop-in (not compliant)
- Add detection query step 2: check systemd drop-ins for master key leaks
- Add step 3: verify running process env against dedicated key
- Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)
gpu-fleet.prose.md:
- Add CT ID column to agent table for cross-reference
- Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
- Update Abiba LiteLLM key to match actual models.json key
- Set Koonimo IP to unknown (was .114 — incorrect per user)
- Fix pi-specific paths .24 → .65
hermes-config-template.prose.md:
- Set Koonimo IP to unknown
infrastructure-control.prose.md:
- Set Koonimo IP to unknown
- Fix Koonimo CT reference
2026-07-05 19:58:14 +00:00
root
2d2259035a
Merge branch 'fix/mumuni-review-suggestions': tested changes
...
- Recursive detection query catches sub-profile violations (verified)
- safe-mutate cross-reference added to Violation Response (verified)
2026-07-05 08:50:02 +00:00
root
7ce6cad01f
fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference
...
Test results:
- Test #1 : Recursive grep finds sub-profile violations old query missed (2/2 vs 1/2)
- Test #2 : safe-mutate step correctly positioned in Violation Response, YAML valid
2026-07-05 08:49:58 +00:00
root
0147ac2abb
Revert "fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference"
...
This reverts commit c4ab31e56b .
2026-07-05 08:49:23 +00:00
root
c4ab31e56b
fix: incorporate Mumuni review — recursive detection query + safe-mutate cross-reference
2026-07-05 08:46:32 +00:00
root
9cf6304e23
fix: remove hardcoded LiteLLM defaults — rely on Gitea secrets
2026-07-04 23:33:34 +00:00
root
91ec7ea3d2
Merge PR #1 : docs: update key enforcement — all agents named keys, systemd fix, CI pipeline
2026-07-04 23:32:19 +00:00
root
e59f59d776
fix: ai-review gracefully skips when LiteLLM secrets not configured
2026-07-04 23:30:25 +00:00
root
889f4e4dd1
docs: update key enforcement — all agents named keys, systemd fix, CI pipeline
...
- All 4 Hermes agents now have dedicated named LiteLLM keys (tanko, mumuni, koby, koonimo)
- Documented systemd drop-in pattern fix (litellm-key.conf)
- Added one-step rotation procedure
- Added CI pipeline and branch protection documentation
- Removed old mumuni keys, generated fresh keys for koby/koonimo
2026-07-04 23:28:59 +00:00
root
cd7d038f91
ci: verify branch protection — abiba-bot push should work
2026-07-04 23:15:56 +00:00
root
b3ab3a6850
fix: remove all Gitea expression syntax from ai-review script
2026-07-04 23:09:29 +00:00
root
105844cf7b
fix: ai-review script — handle Gitea env vars + push event diffs
2026-07-04 23:08:19 +00:00