|
|
@@ -3,7 +3,7 @@ kind: pattern
|
|
|
|
name: infrastructure-control
|
|
|
|
name: infrastructure-control
|
|
|
|
description: >
|
|
|
|
description: >
|
|
|
|
Full infrastructure monitoring and control pattern covering the
|
|
|
|
Full infrastructure monitoring and control pattern covering the
|
|
|
|
5-node Proxmox cluster, 3 Docker ecosystems (22 containers),
|
|
|
|
6-node Proxmox cluster, 3 Docker ecosystems (22 containers),
|
|
|
|
NFS storage, and network services. Defines monitors, remediations,
|
|
|
|
NFS storage, and network services. Defines monitors, remediations,
|
|
|
|
and the access matrix for all environments.
|
|
|
|
and the access matrix for all environments.
|
|
|
|
|
|
|
|
|
|
|
@@ -12,6 +12,11 @@ description: >
|
|
|
|
never mutate infrastructure based on them without first confirming
|
|
|
|
never mutate infrastructure based on them without first confirming
|
|
|
|
against the live system. Policy fields are authoritative. See the
|
|
|
|
against the live system. Policy fields are authoritative. See the
|
|
|
|
`verify-before-mutate` skill.
|
|
|
|
`verify-before-mutate` skill.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Last verified:** 2026-07-24 — corrected Gitea IP (.17 not .110),
|
|
|
|
|
|
|
|
AdGuard IP (.10 not .102), AdGuard placement (minipve not acerpve),
|
|
|
|
|
|
|
|
Abiba placement (hwepve not amdpve), added hwepve as 6th node,
|
|
|
|
|
|
|
|
added dns.sysloggh.net route. See data/learnings.md.
|
|
|
|
---
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
# Infrastructure Control Pattern
|
|
|
|
# Infrastructure Control Pattern
|
|
|
@@ -42,8 +47,8 @@ description: >
|
|
|
|
│ │ │ │ │
|
|
|
|
│ │ │ │ │
|
|
|
|
┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
|
|
|
|
┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
|
|
|
|
▼ ▼ ▼ ▼ ▼
|
|
|
|
▼ ▼ ▼ ▼ ▼
|
|
|
|
minipve amdpve storepve acerpve ocupve
|
|
|
|
minipve amdpve storepve acerpve ocupve hwepve
|
|
|
|
(.12) (.15) (.6) (.9) (.5)
|
|
|
|
(.12) (.15) (.6) (.9) (.5) (.4)
|
|
|
|
|
|
|
|
|
|
|
|
▼
|
|
|
|
▼
|
|
|
|
┌─────────────────────────────────────────────┐
|
|
|
|
┌─────────────────────────────────────────────┐
|
|
|
@@ -95,15 +100,21 @@ description: >
|
|
|
|
|
|
|
|
|
|
|
|
## Section 2: Proxmox Cluster — Monitoring
|
|
|
|
## Section 2: Proxmox Cluster — Monitoring
|
|
|
|
|
|
|
|
|
|
|
|
### Nodes (5)
|
|
|
|
### Nodes (6)
|
|
|
|
|
|
|
|
|
|
|
|
| Node | IP | CPU | RAM | VMs/CTs | Role |
|
|
|
|
| Node | IP | CPU | RAM | VMs/CTs | Role |
|
|
|
|
|------|----|-----|-----|---------|------|
|
|
|
|
|------|----|-----|-----|---------|------|
|
|
|
|
| minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging |
|
|
|
|
| minipve | .12 | 16C | 30GB | authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
|
|
|
|
| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute |
|
|
|
|
| amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute |
|
|
|
|
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat |
|
|
|
|
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat |
|
|
|
|
| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs |
|
|
|
|
| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
|
|
|
|
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
|
|
|
|
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
|
|
|
|
|
|
|
|
| hwepve | .4 | ? | ? | abiba, (mumuni CT 114 stopped) | Agents (new node) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
|
|
|
|
|
|
|
|
> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve, not amdpve. Mumuni
|
|
|
|
|
|
|
|
> (CT 114) is on hwepve (currently stopped), not minipve. Mumuni also has a
|
|
|
|
|
|
|
|
> second instance on minipve at .123 — distinguish by CT ID, not hostname.
|
|
|
|
|
|
|
|
|
|
|
|
### Checks (every 5 min)
|
|
|
|
### Checks (every 5 min)
|
|
|
|
|
|
|
|
|
|
|
@@ -355,15 +366,17 @@ fine. Services that resolve directly to a LAN IP are NetBird-independent.
|
|
|
|
| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
|
|
|
|
| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
|
|
|
|
| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
|
|
|
|
| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
|
|
|
|
| Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
|
|
|
|
| Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
|
|
|
|
| Gitea | git.sysloggh.net:443 | 192.168.68.110 | CNAME → netbird | **Yes** | ⚠️ |
|
|
|
|
| Gitea | git.sysloggh.net:443 | 192.168.68.17:3000 | CNAME → netbird | **Yes** | ⚠️ |
|
|
|
|
| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
|
|
|
|
| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
|
|
|
|
| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
|
|
|
|
| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
|
|
|
|
| DNS UI | dns.sysloggh.net:443 | 192.168.68.102 | CNAME → netbird | **Yes** | ⚠️ |
|
|
|
|
| DNS UI | dns.sysloggh.net:443 | 192.168.68.10:80 | CNAME → netbird | **Yes** | ⚠️ |
|
|
|
|
| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
|
|
|
|
| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
|
|
|
|
| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
|
|
|
|
| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
|
|
|
|
|
|
|
|
|
|
|
|
**Verified 2026-07-02:** NetBird VPS rebooted after a hang; all CNAME'd
|
|
|
|
**Verified 2026-07-24:** NetBird VPS rebooted after a hang; all CNAME'd
|
|
|
|
services recovered. LAN-IP-direct paths stayed up throughout the outage.
|
|
|
|
services recovered. LAN-IP-direct paths stayed up throughout the outage.
|
|
|
|
|
|
|
|
Also added `dns.sysloggh.net` route (was missing entirely).
|
|
|
|
|
|
|
|
See `scripts/netbird-add-domain.sh` for adding new proxy routes.
|
|
|
|
|
|
|
|
|
|
|
|
### 5.2 Checks (every 2 min)
|
|
|
|
### 5.2 Checks (every 2 min)
|
|
|
|
|
|
|
|
|
|
|
@@ -511,8 +524,8 @@ enforced by the `routing-regression.config_url_violations` check in Section
|
|
|
|
| LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
|
|
|
|
| LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
|
|
|
|
| LiteLLM (nginx) | `http://192.168.68.116` | — |
|
|
|
|
| LiteLLM (nginx) | `http://192.168.68.116` | — |
|
|
|
|
| Grafana | `http://192.168.68.116:3001` | — |
|
|
|
|
| Grafana | `http://192.168.68.116:3001` | — |
|
|
|
|
| Authentik | `https://192.168.68.11` | `https://auth.sysloggh.net` |
|
|
|
|
| Authentik | `https://192.168.68.11:9000` | `https://auth.sysloggh.net` |
|
|
|
|
| Gitea | `http://192.168.68.110:3000` | `https://git.sysloggh.net` |
|
|
|
|
| Gitea | `http://192.168.68.17:3000` | `https://git.sysloggh.net` |
|
|
|
|
| Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` |
|
|
|
|
| Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` |
|
|
|
|
| SearXNG | `http://192.168.68.7:8888` | — |
|
|
|
|
| SearXNG | `http://192.168.68.7:8888` | — |
|
|
|
|
| Firecrawl | `http://192.168.68.7:3002` | — |
|
|
|
|
| Firecrawl | `http://192.168.68.7:3002` | — |
|
|
|
@@ -575,25 +588,28 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
|
|
|
|
|
|
|
|
|
|
|
|
| CT | Name | Node | IP | Role | Agent |
|
|
|
|
| CT | Name | Node | IP | Role | Agent |
|
|
|
|
|----|------|------|----|------|-------|
|
|
|
|
|----|------|------|----|------|-------|
|
|
|
|
| 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi |
|
|
|
|
| CT | Name | Node | IP | Role | Agent |
|
|
|
|
|
|
|
|
|----|------|------|----|------|-------|
|
|
|
|
|
|
|
|
| 100 | abiba | **hwepve** | .24 | Pi agent | ✅ pi |
|
|
|
|
| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
|
|
|
|
| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
|
|
|
|
| 102 | adguard | acerpve | — | DNS | ❌ |
|
|
|
|
| 102 | adguard | **minipve** | **.10** | DNS | ❌ |
|
|
|
|
| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
|
|
|
|
| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
|
|
|
|
| 104 | authentik | minipve | .11 | OIDC | ❌ |
|
|
|
|
| 104 | authentik | minipve | .11 | OIDC | ❌ |
|
|
|
|
| 105 | kagentz | amdpve | — | Agent Zero | ✅ |
|
|
|
|
| 105 | kagentz | **hwepve** | — | Agent Zero | ✅ |
|
|
|
|
| 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
|
|
|
|
| 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
|
|
|
|
| 107 | pbs | storepve | — | Backups | ❌ |
|
|
|
|
| 107 | pbs | storepve | — | Backups | ❌ |
|
|
|
|
| 108 | media | storepve | — | Media | ❌ |
|
|
|
|
| 108 | media | storepve | — | Media | ❌ |
|
|
|
|
| 109 | docker-vm | storepve | .7 | Docker host | ❌ |
|
|
|
|
| 109 | docker-vm | storepve | .7 | Docker host | ❌ |
|
|
|
|
| 110 | gitea | minipve | — | Git | ❌ |
|
|
|
|
| 110 | gitea | minipve | **.17** | Git | ❌ |
|
|
|
|
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
|
|
|
|
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
|
|
|
|
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
|
|
|
|
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
|
|
|
|
| 113 | baggy | amdpve | ? | Hermes agent | ✅ |
|
|
|
|
| 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
|
|
|
|
| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
|
|
|
|
| 114 | mumuni | **hwepve** | .123 | Hermes agent (stopped) | ✅ |
|
|
|
|
| 115 | scottdenya | amdpve | — | ? | ❌ |
|
|
|
|
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
|
|
|
|
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
|
|
|
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
|
|
|
| 117 | zulip | storepve | — | Chat | ❌ |
|
|
|
|
| 117 | zulip | storepve | .19 | Chat | ❌ |
|
|
|
|
| 118 | jitsi | minipve | — | Video | ❌ |
|
|
|
|
| 118 | jdownloader | storepve | — | JDownloader container | ❌ |
|
|
|
|
|
|
|
|
| 119 | infisical-vault | minipve | — | Vault | ❌ |
|
|
|
|
|
|
|
|
|
|
|
|
## Appendix C: Docker Compose Files Location
|
|
|
|
## Appendix C: Docker Compose Files Location
|
|
|
|
|
|
|
|
|
|
|
@@ -613,27 +629,28 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
|
|
|
|
|
|
|
|
|
|
|
|
| CT | Name | Node | pct-run |
|
|
|
|
| CT | Name | Node | pct-run |
|
|
|
|
|-----|------|------|---------|
|
|
|
|
|-----|------|------|---------|
|
|
|
|
| 100 | abiba | amdpve | `pct-run 100` |
|
|
|
|
| 100 | abiba | hwepve | `pct-run 100` |
|
|
|
|
| 105 | kagentz | amdpve | `pct-run 105` |
|
|
|
|
| 105 | kagentz | **hwepve** | `pct-run 105` |
|
|
|
|
| 111 | tdunna | amdpve | `pct-run 111` |
|
|
|
|
| 111 | tdunna | amdpve | `pct-run 111` |
|
|
|
|
| 112 | tanko | amdpve | `pct-run 112` |
|
|
|
|
| 112 | tanko | amdpve | `pct-run 112` |
|
|
|
|
| 113 | baggy | amdpve | `pct-run 113` |
|
|
|
|
| 113 | baggy | amdpve | `pct-run 113` |
|
|
|
|
| 115 | scottdenya | amdpve | `pct-run 115` |
|
|
|
|
| 115 | scottdenya | amdpve | `pct-run 115` |
|
|
|
|
| 104 | authentik | minipve | `pct-run 104` |
|
|
|
|
| 104 | authentik | minipve | `pct-run 104` |
|
|
|
|
| 110 | gitea | minipve | `pct-run 110` |
|
|
|
|
| 110 | gitea | minipve | `pct-run 110` |
|
|
|
|
| 114 | mumuni | minipve | `pct-run 114` |
|
|
|
|
| 114 | mumuni | **hwepve** | `pct-run 114` |
|
|
|
|
| 116 | syslog-api | minipve | `pct-run 116` |
|
|
|
|
| 116 | syslog-api | minipve | `pct-run 116` |
|
|
|
|
| 106 | ra-h-os | storepve | `pct-run 106` |
|
|
|
|
| 106 | ra-h-os | storepve | `pct-run 106` |
|
|
|
|
| 107 | proxmox-backup | storepve | `pct-run 107` |
|
|
|
|
| 107 | proxmox-backup | storepve | `pct-run 107` |
|
|
|
|
| 108 | media | storepve | `pct-run 108` |
|
|
|
|
| 108 | media | storepve | `pct-run 108` |
|
|
|
|
| 117 | zulip | storepve | `pct-run 117` |
|
|
|
|
| 117 | zulip | storepve | `pct-run 117` |
|
|
|
|
| 102 | adguard | acerpve | `pct-run 102` |
|
|
|
|
| 102 | adguard | **minipve** | `pct-run 102` |
|
|
|
|
|
|
|
|
|
|
|
|
GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
|
|
|
|
GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
|
|
|
|
```bash
|
|
|
|
```bash
|
|
|
|
ssh root@192.168.68.8 # RTX 3090
|
|
|
|
ssh root@192.168.68.8 # RTX 3090
|
|
|
|
ssh root@192.168.68.110 # RTX 5070
|
|
|
|
ssh root@192.168.68.110 # RTX 5070
|
|
|
|
ssh root@192.168.68.15 # Strix Halo
|
|
|
|
ssh root@192.168.68.15 # Strix Halo
|
|
|
|
|
|
|
|
ssh root@192.168.68.4 # hwepve (abiba, kagentz, mumuni)
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## Section 7: Agent Health Check (consolidated — 2026-07-05)
|
|
|
|
## Section 7: Agent Health Check (consolidated — 2026-07-05)
|
|
|
|