root
776df418ca
gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3)
- .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache
- .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV
- .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm
- LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s
- Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma
- Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
2026-07-08 17:25:42 +00:00
root
5aa93117de
docs: capture all session changes — port conflict, health check, streaming
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped
Updated contracts to reflect all infrastructure changes made 2026-07-05/06:
gpu-fleet.prose.md:
- Port conflict detection on all 3 GPU wrappers
- Ghost process root cause documented (.8 stale pid 25836)
infrastructure-control.prose.md:
- Section 7: Agent Health Check (consolidated, non-disruptive)
- Disabled scripts documented (zulip-watchdog, zulip-monitor)
hermes-agent-baseline.prose.md:
- Health verification section + GPU port conflict detection
- Change log updated
zulip-health.prose.md:
- Streaming support section (edit_message + PATCH API)
2026-07-06 02:46:41 +00:00
root
35e864bdb7
audit: all 4 Hermes agents verified — no master key, api_key workaround applied
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
Agent audit (2026-07-05):
Tanko CT 112: sk-CggiHWlamQyShxWC3Hx6uw — vision+compression patched
Mumuni CT 114: sk-VrqCNlwUgzoNGOpikJ7nwQ — vision patched
Tdunna CT 111: sk-6sbCNjz2T6lTVDBdlNHXsA — vision+compression patched
Baggy CT 113: sk-krnw_zGBwvvL5b7l2t-s-A — vision+compression patched
All 4 agents: zero master key references, dedicated LiteLLM keys,
api_key workaround applied to bypass auxiliary_client bug.
2026-07-05 23:36:00 +00:00
root
c8dcd8b63d
fix: Tanko key regenerated, Mumuni key verified from /etc/environment
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
Tanko: old key sk-620a05e95a... was from docker-compose router config,
not LiteLLM key DB. Deleted + regenerated via LiteLLM API:
sk-CggiHWlamQyShxWC3Hx6uw (alias: tanko, budget: $100)
Mumuni: verified via pct-run 114 — key is sk-VrqCNlwUgzoNGOpikJ7nwQ
(was previously listed as sk-XY2aUfvy2... in contracts)
2026-07-05 20:18:05 +00:00
root
5c90eb8f6c
rename: Koonimo→Baggy, Koby→Tdunna — match hostnames
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
Standardize agent names across all contracts and pct-run.sh.
No functional changes — same CT IDs, same keys, same nodes.
2026-07-05 20:04:22 +00:00
root
0db95a4d1c
fix: correct IPs, CT IDs, and key status across all contracts
...
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped
hermes-key-enforcement.prose.md:
- Add IP column to verified agents table
- Fix Tanko status: was using master key via systemd drop-in (not compliant)
- Add detection query step 2: check systemd drop-ins for master key leaks
- Add step 3: verify running process env against dedicated key
- Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)
gpu-fleet.prose.md:
- Add CT ID column to agent table for cross-reference
- Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
- Update Abiba LiteLLM key to match actual models.json key
- Set Koonimo IP to unknown (was .114 — incorrect per user)
- Fix pi-specific paths .24 → .65
hermes-config-template.prose.md:
- Set Koonimo IP to unknown
infrastructure-control.prose.md:
- Set Koonimo IP to unknown
- Fix Koonimo CT reference
2026-07-05 19:58:14 +00:00
root
263c4080f3
ci: add automated PR validation pipeline + fix regressions
...
NEW: Gitea Actions CI pipeline (.gitea/workflows/pr-pipeline.yaml)
- Stage 1 (validate): YAML frontmatter validation for all .prose.md files
- Stage 2 (lint): Structural checks + regression detection
* /grafana/ nginx route (reverted 2026-07-02)
* Stale CT IDs (122→112, no 123)
* Verified: .19/.122/.123 = correct bridge IPs
- Stage 3 (ai-review): LiteLLM-powered diff review against ground truth
- Stage 4 (auto-merge): Gate — safe to merge when all green
NEW: scripts/prose-lint.sh — contract structure + regression checker
NEW: scripts/prose-ai-review.sh — AI review via syslog-auto model
FIXES FOUND BY LINT:
- gpu-fleet.prose.md: removed /grafana/ from nginx routing diagram
- gpu-fleet.prose.md: removed /grafana/ from config file description
- infrastructure-control.prose.md: CT 122→112 in topology diagram
- scripts/prose-lint.sh: updated to not flag .19 (verified correct IP)
2026-07-04 22:13:51 +00:00
root
b15771bfd9
prose: add Strix Halo thermal safeguard (-n 8192 cap) to gpu-fleet
...
Runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min,
pushing Tctl to 98°C (9°C past critical) and throttling 70->29 t/s.
Server-side -n 8192 hard cap now bounds generation. Documented connection
source (all from .116/LiteLLM) and the thermal constraint (fanless APU).
2026-07-02 03:51:36 +00:00
root
a07ff2d96a
prose: align all contracts with Vulkan rebuild + Koby key regen
...
- gpu-fleet: Koby key updated (sk-lDTsWy7H3T...), config table (ornith service
+ masked services), benchmarks (prompt 209→532 tok/s), ROCm→Vulkan status
- gpu-monitor: topology (sidecars :8090 removed, .15 via router), subsystem
polling (router /health/unified as source of truth), execution steps,
alert threshold (sidecar-unreachable downgraded from critical)
2026-07-02 00:29:10 +00:00
root
49e8d968f5
gpu-fleet: Strix Halo Vulkan rebuild + LiteLLM alignment (2026-07-01)
...
- ornith now on Vulkan build 4fc4ec5, 256K ctx, ~70 tok/s
- Fixed GPU_MOE_URL bug in docker-compose (.110→.15)
- Added router sidecar fallback for health-checks
- Updated roster context 4096→262144 + correct model_path
- Documented port 8080 firewall (restricted to .116)
2026-07-02 00:01:56 +00:00
root
38d2f7d956
feat: update contracts with GPU optimizations, Strix Halo/ROCm status, alert migration
...
- gpu-fleet: documented parallel=2/batch=4096 optimizations for RTX 3090/5070
- gpu-fleet: added Strix Halo ROCm status (installed, blocked by HSA runtime on Debian 13)
- gpu-fleet: documented alert migration from DM → #agent-hub stream topics
- gpu-monitor: added Alert Delivery section with stream topic conventions
- All GPU alerts now go to #agent-hub for cross-agent visibility
2026-07-01 22:25:26 +00:00
root
76fc2595ea
feat(gpu-fleet): GPU fleet management prose contract
...
Declarative GPU roster replaces hardcoded router.py model configs.
Operations: add-model, remove-model, heal, sync-keys, list.
Documents agent key architecture with LITELLM_API_KEY env var.
Covers all 3 GPU hosts (RTX 5070, RTX 3090, Strix Halo).
2026-06-28 16:24:40 +00:00