Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2ace79fcab | ||
|
|
87b4d67067 | ||
|
|
7d1db62a8e | ||
|
|
9943be5e68 | ||
|
|
fa26b7a579 |
@@ -35,8 +35,7 @@ Docker hosts get special attention:
|
||||
| All other CTs | — | LOW — no Docker | apt clean, log rotate |
|
||||
| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
|
||||
|
||||
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
|
||||
> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
|
||||
> **Note:** CT 118 is now jdownloader (active on storepve). CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
|
||||
|
||||
## Threat Levels
|
||||
|
||||
@@ -275,10 +274,10 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|
||||
### CT Access (via pct-run)
|
||||
| CT | Name | Node | Status |
|
||||
|----|------|------|--------|
|
||||
| 100 | abiba | amdpve | local |
|
||||
| 102 | adguard | acerpve | ✅ reachable |
|
||||
| 100 | abiba | hwepve | local |
|
||||
| 102 | adguard | minipve | ✅ reachable |
|
||||
| 104 | authentik | minipve | ✅ reachable |
|
||||
| 105 | kagentz | amdpve | ✅ reachable |
|
||||
| 105 | kagentz | hwepve | ✅ reachable |
|
||||
| 106 | ra-h-os | storepve | ✅ reachable |
|
||||
| 107 | pbs | storepve | ✅ reachable |
|
||||
| 108 | media | storepve | ✅ reachable |
|
||||
@@ -286,7 +285,7 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|
||||
| 111 | tdunna | amdpve | ✅ reachable |
|
||||
| 112 | tanko | amdpve | ✅ reachable |
|
||||
| 113 | baggy | amdpve | ✅ reachable |
|
||||
| 114 | mumuni | minipve | ✅ reachable |
|
||||
| 114 | mumuni | hwepve | ✅ reachable |
|
||||
| 115 | scottdenya | amdpve | ✅ reachable |
|
||||
| 116 | syslog-api | minipve | ✅ reachable |
|
||||
| 117 | zulip | storepve | ✅ reachable |
|
||||
@@ -303,4 +302,4 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|
||||
|------|-----|------|--------|
|
||||
| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
|
||||
|
||||
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
|
||||
> **Note:** CT 118 is now jdownloader (active on storepve). CT 119 (infisical-vault) added on minipve.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
|
||||
@@ -185,7 +185,7 @@ what, and why should I care?
|
||||
|
||||
```
|
||||
❌ "Monitors infrastructure health"
|
||||
✅ "Scans all 5 Proxmox nodes and 19 CTs for disk pressure, checks Docker
|
||||
✅ "Scans all 6 Proxmox nodes and 19 CTs for disk pressure, checks Docker
|
||||
container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL"
|
||||
```
|
||||
|
||||
|
||||
@@ -26,11 +26,11 @@ done
|
||||
|-------|-----|------|-----|---------------|------------|----------|
|
||||
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
||||
| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
|
||||
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
||||
| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||
| Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes |
|
||||
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
||||
|
||||
> **Note**: CT hostnames (tdunna→CT129, baggy→CT114) differ from agent identities (koby, koonimo).
|
||||
> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
|
||||
|
||||
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
|
||||
Keys are stored in Infisical vault (project=agents, env=production) and injected at
|
||||
@@ -169,9 +169,9 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
|
||||
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
|
||||
```
|
||||
|
||||
### For Koby (CT 129 / tdunna)
|
||||
### For Koby (CT 111 / tdunna)
|
||||
|
||||
Koby runs Hermes on CT 129 (tdunna). Config files at `/root/.hermes/config.yaml`.
|
||||
Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
|
||||
Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
|
||||
|
||||
**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
|
||||
|
||||
@@ -38,7 +38,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
|
||||
| Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
|
||||
| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
|
||||
| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
|
||||
| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
|
||||
| Koonimo | `koonimo` | CT 113 (baggy) | SSH root | — |
|
||||
| Kagenz0 | `kagenz0-*` | ? | Zulip | — |
|
||||
|
||||
> CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
|
||||
|
||||
@@ -16,7 +16,7 @@ description: >
|
||||
**Last verified:** 2026-07-24 — corrected Gitea IP (.17 not .110),
|
||||
AdGuard IP (.10 not .102), AdGuard placement (minipve not acerpve),
|
||||
Abiba placement (hwepve not amdpve), added hwepve as 6th node,
|
||||
added dns.sysloggh.net route. See data/learnings.md.
|
||||
added dns.sysloggh.net route.
|
||||
---
|
||||
|
||||
# Infrastructure Control Pattern
|
||||
@@ -109,7 +109,7 @@ description: >
|
||||
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat |
|
||||
| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
|
||||
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
|
||||
| hwepve | .4 | ? | ? | abiba, (mumuni CT 114 stopped) | Agents (new node) |
|
||||
| hwepve | .4 | 12C | 15GB | abiba, kagentz, (mumuni CT 114 stopped) | Agents (new node) |
|
||||
|
||||
> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
|
||||
> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve, not amdpve. Mumuni
|
||||
@@ -365,7 +365,7 @@ fine. Services that resolve directly to a LAN IP are NetBird-independent.
|
||||
|---------|--------|-------------|-------------|-------------|--------|
|
||||
| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
|
||||
| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
|
||||
| Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
|
||||
| Authentik | auth.sysloggh.net:443 | 192.168.68.11:9000 | CNAME → netbird | **Yes** | ⚠️ |
|
||||
| Gitea | git.sysloggh.net:443 | 192.168.68.17:3000 | CNAME → netbird | **Yes** | ⚠️ |
|
||||
| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
|
||||
| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
|
||||
@@ -586,8 +586,6 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
|
||||
|
||||
## Appendix B: CT Inventory
|
||||
|
||||
| CT | Name | Node | IP | Role | Agent |
|
||||
|----|------|------|----|------|-------|
|
||||
| CT | Name | Node | IP | Role | Agent |
|
||||
|----|------|------|----|------|-------|
|
||||
| 100 | abiba | **hwepve** | .24 | Pi agent | ✅ pi |
|
||||
@@ -630,14 +628,14 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
|
||||
| CT | Name | Node | pct-run |
|
||||
|-----|------|------|---------|
|
||||
| 100 | abiba | hwepve | `pct-run 100` |
|
||||
| 105 | kagentz | **hwepve** | `pct-run 105` |
|
||||
| 105 | kagentz | hwepve | `pct-run 105` |
|
||||
| 111 | tdunna | amdpve | `pct-run 111` |
|
||||
| 112 | tanko | amdpve | `pct-run 112` |
|
||||
| 113 | baggy | amdpve | `pct-run 113` |
|
||||
| 115 | scottdenya | amdpve | `pct-run 115` |
|
||||
| 104 | authentik | minipve | `pct-run 104` |
|
||||
| 110 | gitea | minipve | `pct-run 110` |
|
||||
| 114 | mumuni | **hwepve** | `pct-run 114` |
|
||||
| 114 | mumuni | hwepve | `pct-run 114` |
|
||||
| 116 | syslog-api | minipve | `pct-run 116` |
|
||||
| 106 | ra-h-os | storepve | `pct-run 106` |
|
||||
| 107 | proxmox-backup | storepve | `pct-run 107` |
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
kind: responsibility
|
||||
name: infrastructure-update
|
||||
description: >
|
||||
Autonomous system-wide update contract covering all 5 Proxmox nodes,
|
||||
Autonomous system-wide update contract covering all 6 Proxmox nodes,
|
||||
15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
|
||||
Docker images, and container stacks in safe waves with health checks
|
||||
and automatic rollback on failure.
|
||||
@@ -56,10 +56,11 @@ Before ANY update wave:
|
||||
| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| hwepve (.4) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
|
||||
| CT 114 (mumuni, hwepve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
|
||||
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
|
||||
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
|
||||
|
||||
@@ -117,7 +118,9 @@ After every node reboot, run these checks:
|
||||
container loses its Docker network assignment (SIGKILL during storage
|
||||
outage detaches it from `zulip_default` network). Run:
|
||||
```bash
|
||||
ssh root@192.168.68.19 'docker rm -f zulip-zulip-1 && cd /opt/zulip && docker compose up -d'
|
||||
ssh root@192.168.68.19
|
||||
docker rm -f zulip-zulip-1
|
||||
cd /opt/zulip && docker compose up -d
|
||||
```
|
||||
The compose restart recreates the container on the correct network.
|
||||
|
||||
@@ -218,7 +221,7 @@ When LiteLLM is upgraded to a version supporting per-key MCP grants:
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] All 5 PVE nodes updated, no reboot-loop
|
||||
- [ ] All 6 PVE nodes updated, no reboot-loop
|
||||
- [ ] All VMs/CTs running post-update
|
||||
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
|
||||
- [ ] LiteLLM inference passing (syslog-auto test)
|
||||
@@ -234,7 +237,7 @@ After completion, send Zulip DM:
|
||||
```
|
||||
📋 Infrastructure Update — YYYY-MM-DD
|
||||
|
||||
Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
|
||||
Updated: 6 PVE nodes, 12 CTs/VMs, 30+ containers
|
||||
Security fixes: N CVEs patched
|
||||
Downtime: <service> <duration>
|
||||
Failures: none / <details>
|
||||
|
||||
@@ -31,7 +31,7 @@ Workers execute tasks on whatever infrastructure they're given — SSH to .6,
|
||||
## Why This Matters
|
||||
|
||||
Without enforced delegation, the manager consumes the full iteration budget
|
||||
(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
|
||||
(60 calls) on single-turn tasks — SSH to 6 nodes, check each VM, read logs —
|
||||
leaving no capacity for actual coordination. The result: context overflow
|
||||
(59K tokens in system prompt), iteration exhaustion, and degraded response
|
||||
quality. This contract exists because I blew through my budget checking
|
||||
@@ -82,7 +82,7 @@ it asks the manager (via relay) — it doesn't go find it on its own.
|
||||
|
||||
**This is a hard rule, not a recommendation.** Violating it produces the exact
|
||||
type of discrepancy the kanban pipeline exists to prevent: a review worker finds
|
||||
"5 nodes present" in the raw data but "5/5 online" in the report — even though
|
||||
"6 nodes present" in the raw data but "6/6 online" in the report — even though
|
||||
one of those nodes was unreachable. The report lied because it used data the
|
||||
raw data never provided.
|
||||
|
||||
@@ -137,7 +137,7 @@ delegate_task(
|
||||
```
|
||||
delegate_task(
|
||||
tasks=[
|
||||
{"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
|
||||
{"goal": "Check all 6 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
|
||||
{"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
|
||||
]
|
||||
)
|
||||
@@ -191,7 +191,7 @@ Only verified results reach Kwame. Format per channel:
|
||||
{
|
||||
"lane_id": "devops-check",
|
||||
"worker": "syslog-devops",
|
||||
"goal": "Check all 5 Proxmox nodes",
|
||||
"goal": "Check all 6 Proxmox nodes",
|
||||
"status": "dispatched|completed|failed",
|
||||
"output_file": "/tmp/node-report.md"
|
||||
}
|
||||
|
||||
@@ -88,7 +88,7 @@ agent: abiba
|
||||
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
|
||||
| minipve | 192.168.68.12 | PVE |
|
||||
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) |
|
||||
| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 |
|
||||
| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts abiba (lxc/100), kagentz (lxc/105), mumuni (lxc/114). CTs 100/105 migrated from amdpve, CT 114 from minipve 2026-07-20 |
|
||||
|
||||
## Operations
|
||||
|
||||
|
||||
Regular → Executable
+9
-6
@@ -11,31 +11,33 @@ set -euo pipefail
|
||||
# ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ──
|
||||
declare -A CT_NODES=(
|
||||
# amdpve (192.168.68.15)
|
||||
[100]=amdpve # abiba
|
||||
[105]=amdpve # kagentz
|
||||
[111]=amdpve # tdunna
|
||||
[112]=amdpve # tanko
|
||||
[113]=amdpve # baggy
|
||||
[115]=amdpve # scottdenya
|
||||
# minipve (192.168.68.12)
|
||||
[102]=minipve # adguard (was acerpve)
|
||||
[104]=minipve # authentik
|
||||
[110]=minipve # gitea
|
||||
[114]=minipve # mumuni
|
||||
[116]=minipve # syslog-api
|
||||
[119]=minipve # infisical-vault
|
||||
# storepve (192.168.68.6)
|
||||
[106]=storepve # ra-h-os
|
||||
[107]=storepve # proxmox-backup
|
||||
[108]=storepve # media
|
||||
[117]=storepve # zulip
|
||||
# acerpve (192.168.68.9)
|
||||
[102]=acerpve # adguard
|
||||
[118]=storepve # jdownloader
|
||||
# acerpve (192.168.68.9) — no CTs (bare metal GPU .8)
|
||||
# hwepve (192.168.68.4)
|
||||
[100]=hwepve # abiba (was amdpve)
|
||||
[105]=hwepve # kagentz (was amdpve)
|
||||
[114]=hwepve # mumuni (was minipve)
|
||||
# ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
|
||||
#
|
||||
# REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
|
||||
# 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090)
|
||||
# 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070)
|
||||
# 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH)
|
||||
# 118 jitsi → stopped, not in service
|
||||
)
|
||||
|
||||
# Each node must be root-accessible via SSH hostname
|
||||
@@ -48,6 +50,7 @@ declare -A NODE_IPS=(
|
||||
[storepve]=192.168.68.6
|
||||
[acerpve]=192.168.68.9
|
||||
[ocupve]=192.168.68.5
|
||||
[hwepve]=192.168.68.4
|
||||
)
|
||||
|
||||
resolve_node() {
|
||||
|
||||
@@ -46,17 +46,19 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol
|
||||
|
||||
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
|
||||
|
||||
**Proxmox Cluster "Tabiri" (5 nodes):**
|
||||
- amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya
|
||||
- minipve (192.168.68.12): authentik, gitea, mumuni, syslog-api, jitsi
|
||||
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip
|
||||
- acerpve (192.168.68.9): llm-gpu, adguard
|
||||
**Proxmox Cluster "Tabiri" (6 nodes):**
|
||||
- amdpve (192.168.68.15): tanko, tdunna, baggy, scottdenya
|
||||
- minipve (192.168.68.12): adguard, authentik, gitea, syslog-api, infisical-vault
|
||||
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip
|
||||
- acerpve (192.168.68.9): llm-gpu
|
||||
- ocupve (192.168.68.5): ocu-llm
|
||||
- hwepve (192.168.68.4): abiba, kagentz, mumuni
|
||||
|
||||
**CT IDs (verified 2026-07-04 against PVE API):**
|
||||
**CT IDs (verified 2026-07-24 against PVE API):**
|
||||
100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os
|
||||
107:pbs 108:media 110:gitea 111:tdunna 112:tanko
|
||||
113:baggy 114:mumuni 115:scottdenya 116:syslog-api 117:zulip
|
||||
118:jdownloader 119:infisical-vault
|
||||
|
||||
**NO CT 122, CT 123, or .19 exist in the cluster.**
|
||||
|
||||
|
||||
Reference in New Issue
Block a user