prose-contracts/infrastructure-control.prose.md

---
kind: pattern
name: infrastructure-control
description: >
  Full infrastructure monitoring and control pattern covering the
  5-node Proxmox cluster, 3 Docker ecosystems (22 containers),
  NFS storage, and network services. Defines monitors, remediations,
  and the access matrix for all environments.

  FIELD TRUST: Live-state fields (IPs, ports, hostnames, credentials,
  container names, PIDs) are marked VERIFY-BEFORE-USE. They drift —
  never mutate infrastructure based on them without first confirming
  against the live system. Policy fields are authoritative. See the
  `verify-before-mutate` skill.

  **Last verified:** 2026-08-15 — hwepve removed from Tabiri cluster
  (now 5 nodes: minipve, amdpve, storepve, acerpve, ocupve). hwepve
  (192.168.68.4) is a standalone PVE node + NetBird routing peer;
  London relocation pending. CT 100 (abiba) is on minipve; CT 105
  (kagentz) is on amdpve.
---

# Infrastructure Control Pattern

## Topology

```
                    ┌─────────────────────────┐
                    │   OpenProse Contract    │
                    │  (declares what's true) │
                    └──────────┬──────────────┘
                               │
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
   ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
   │   Abiba     │    │   Tanko      │    │   Mumuni     │
   │   (pi)      │    │   (Hermes)   │    │   (Hermes)   │
   │   CT 100    │    │   CT 112     │    │   CT 100     │
   └──────┬──────┘    └──────┬───────┘    └──────┬───────┘
          │                  │                    │
          └──────────────────┼────────────────────┘
                             ▼
          ┌────────────────────────────────────┐
          │         Proxmox Cluster API        │
          │    minipve.sysloggh.net:443        │
          │   (monitoring@pve!mumuni token)    │
          └────┬──────┬──────┬──────┬──────────┘
               │      │      │      │
          ┌────┘ ┌────┘ ┌────┘ ┌────┘
          ▼      ▼      ▼      ▼
       minipve amdpve storepve acerpve ocupve
        (.12)   (.15)   (.6)    (.9)    (.5)

          ▼
   ┌─────────────────────────────────────────────┐
   │            Docker Ecosystems                │
   │  ┌────────────┬────────────┬──────────────┐ │
   │  │ docker-vm  │ CT 116     │ Netbird      │ │
   │  │ (.7)       │ (.116)     │ (72.61.0.17) │ │
   │  │ 11 ctrs    │ 6 ctrs     │ 5 ctrs       │ │
   │  └────────────┴────────────┴──────────────┘ │
   └─────────────────────────────────────────────┘
```

## Section 1: Access Matrix

### API Tokens & Credentials

| Resource | Auth Method | Credential Source | Status |
|----------|------------|-------------------|--------|
| Proxmox Cluster | PVE API Token | Infisical vault (`PROXMOX_API_TOKEN`) | ✅ |
| Proxmox Root | Password via API ticket | Infisical vault (`PROXMOX_ROOT_PASSWORD`) | ✅ |
| docker-vm (.7) | SSH root | SSH key | ✅ |
| CT 116 (syslog-api) | SSH root | SSH key | ✅ |
| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
| Baggy CT (113) | SSH jerome | ❌ no key access |
| Netbird (.17) | SSH root | SSH key | ✅ |
| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
| Zulip | Bot API key | Infisical vault (`ZULIP_BOT_KEY`) | ✅ |
| RA-H OS | MCP bridge | port 3100 | ✅ |
| LiteLLM Admin | master key | Infisical vault (`LITELLM_MASTER_KEY`) | ✅ VERIFY-BEFORE-USE |
| Grafana | admin password | Infisical vault (`GRAFANA_ADMIN_PASSWORD`) | ✅ VERIFY-BEFORE-USE |

> **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this
> contract are live-state fields. Test them against the live system before
> relying on them (e.g., `curl -H "Authorization: Bearer $KEY" .../v1/models`
> for a LiteLLM key; `pct config <ct>` for a CT IP). Drift is expected —
> see `verify-before-mutate` skill. Policy fields (rules, doctrines) are
> authoritative and do not need verification.

### Reachability Matrix

| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.24) | Baggy (.114) |
|-----------|---------|----------------|--------|-------------|---------------|----------------|
| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
| **Mumuni** (.123) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |

**Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.

## Section 2: Proxmox Cluster — Monitoring

### Nodes (5)

| Node | IP | CPU | RAM | VMs/CTs | Role |
|------|----|-----|-----|---------|------|
| minipve | .12 | 16C | 30GB | abiba, tanko, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | kagentz, baggy, scottdenya, adguard2 | Agents, compute |
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat |
| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |

> **Note:** CTs on storepve include jdownloader (CT 118) and tdunna (CT 111).
> AdGuard (CT 102) is on minipve at .10, not acerpve. Abiba (CT 100) is on
> minipve (moved from hwepve 2026-08-15); kagentz (CT 105) is on amdpve.
> Mumuni runs inside Abiba CT100 (.24); CT 114 (mumuni) no longer exists in the cluster.
>
> **hwepve (192.168.68.4) — STANDALONE (removed from Tabiri 2026-08-15):**
> Huawei MateBook 16 (KLVL-WXX9), pve-manager/9.2.10, kernel 7.0.14-8-pve.
> Zero VMs/CTs. Being relocated to London as a standalone PVE node + NetBird
> routing peer (relocation pending). Localizations applied: timezone
> Europe/London, lid-switch ignore, sleep/suspend/hibernate targets masked,
> cluster-shared storage removed (remaining: local, local-lvm, storage,
> mediastore). prometheus-node-exporter active on :9100; net.ipv4.ip_forward=1;
> NetBird client not yet installed (enrollment pending setup key).

### Checks (every 5 min)

```
## Maintains

- cluster-status: { online_nodes: int, offline_nodes: int, timestamp }
- node-cpu-usage: { node: pct, warnings: [] }
- node-memory-usage: { node: free_pct, warnings: [] }
- node-uptime: { node: seconds, just_rebooted: bool }

## Checks

- For each node in [minipve, amdpve, storepve, acerpve, ocupve]:
  - GET /api2/json/nodes/{node}/status → check status == "online"
  - GET /api2/json/nodes/{node}/status → cpu < 0.80
  - GET /api2/json/nodes/{node}/status → free_mem > 10%
  - GET /api2/json/nodes/{node}/status → uptime > 300 (warn if just rebooted)

- For each CT in inventory (19 total):
  - GET /api2/json/cluster/resources → filter by type=lxc
  - Warn if status != "running"

- Storage pools:
  - GET /api2/json/nodes/storepve/storage/mediastore → used < 80%
  - GET /api2/json/nodes/storepve/storage/pbs-backup → last backup < 48h

## Remediations

- Node offline → alert via Zulip DM, escalate after 3x
- CT stopped → `pct start <vmid>` via PVE API, verify after 30s
- VM stopped → `qm start <vmid>` via PVE API, verify after 60s
- Storage > 80% → warn via Zulip DM
- Storage > 95% → crit via Zulip + relay to maintainer
```

## Section 3: Docker Ecosystems — Monitoring

### Ecosystem A: docker-vm (192.168.68.7)

16 containers across 4 compose stacks + trove agents:

| Stack | Path | Containers |
|-------|------|-----------|
| **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis |
| **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
| **Home stack** | `/opt/home_stack/` | stirling-pdf, pulse (jdownloader decommissioned 2026-08-01 → dedicated CT 118 LXC) |
| **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |

**Last verified:** 2026-07-09 — 16/16 containers running healthy.

### Home Stack Services

**Stirling-PDF** (deployed 2026-07-03, Authentik SSO 2026-07-03):
- URL: `https://pdf.sysloggh.net` (public) / `http://192.168.68.7:8989` (direct)
- Swagger: `http://192.168.68.7:8989/swagger-ui.html`
- Admin credentials: `«vault: infrastructure/production STIRLING_ADMIN_USER»` / `«vault: infrastructure/production STIRLING_ADMIN_PASSWORD»`
- API key: `«vault: infrastructure/production STIRLING_API_KEY»`
- Authentik OAuth2: configured but disabled (requires paid Server license). Ready to enable: set `SECURITY_OAUTH2_ENABLED=true` + `SECURITY_LOGINMETHOD=all`
- Compose: `/opt/home_stack/docker-compose.yml`
- Control script: `/opt/home_stack/infra-control.sh`

**JDownloader** (decommissioned from docker-vm 2026-08-01 — moved to dedicated CT 118 LXC):
- LXC: CT 118 on storepve, `192.168.68.20` (JDownloader + VNC 5900 + web UI 6080)
- Web UI: `http://192.168.68.20:6080` (noVNC via websockify)
- Docker container `jdownloader-2` on .7 removed; compose entry stripped

**Pulse**:
- URL: `http://192.168.68.7:7655` (direct LAN)
- Public: `https://pulse.sysloggh.net` (NetBird CNAME proxy)
- Container: `rcourtman/pulse:5.1.35` in `/opt/home_stack` (port 7655, was 3001)

### Ecosystem B: CT 116 syslog-api (192.168.68.116)

12 containers on CT 116 — 11 in the inference-harness stack + trove-agent-docker (verified live 2026-09-11; LiteLLM upgraded 1.90.0-rc.1 -> 1.99.1; trove-agent-docker added 2026-09-11):

| Container | Image | Port | Role |
|-----------|-------|------|------|
| harness-litellm | ghcr.io/docker.litellm.ai/berriai/litellm:1.99.1 | :4000→:4000 | API proxy, key mgmt, fallbacks |
| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/ |
| harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) |
| harness-redis | redis:7-alpine | :6379 | LiteLLM cache + rate-limit state |
| harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI |
| harness-grafana | grafana/grafana | :3000→:3001 (direct LAN, not behind nginx) | GPU + Proxmox + Docker dashboards |
| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 6 jobs |
| trove-agent-docker | ghcr.io/techdox/trove-agent-docker:latest | outbound agent (no port) | Trove host agent — service inventory + metrics, added 2026-09-11 |

**Nginx routing**:
- `/v1/*` → harness-litellm:4000 (API)
- `/admin/*` → harness-litellm:4000
- `/dashboard/` → harness-dashboard:3000
- `/litellm/*` → harness-litellm:4000
- `/health/*` → harness-litellm:4000/health/liveliness
- `/gpu/` → 192.168.68.24:9100 (fleet monitor)

**Prometheus targets**:
- 192.168.68.8:9400 (RTX 3090 — qwen)
- 192.168.68.110:9400 (RTX 5070 — gpu-vision)
- 192.168.68.15:9400 (Strix Halo — strix-moe)
- harness-litellm:4000 (LiteLLM health)

### Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud)

5 containers, single compose stack at `/root/docker-compose.yml`:

| Container | Image | Role |
|-----------|-------|------|
| netbird-server | netbirdio/netbird-server:latest | VPN controller, management API |
| netbird-dashboard | netbirdio/dashboard:latest | Web management UI |
| netbird-proxy | netbirdio/reverse-proxy:latest | SNI router + TLS passthrough for *.sysloggh.net |
| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF / threat intelligence |
| netbird-traefik | traefik:v3.6 | Edge reverse proxy, Let's Encrypt TLS |

**Architecture:** Internet → Traefik (TLS + HTTP→HTTPS redirect) → Netbird proxy
(TLS passthrough via HostSNI, PROXY protocol) → Netbird mesh → LAN backends.
Traefik handles the Netbird dashboard directly; all other `*.sysloggh.net`
domains are TLS-passthrough to the proxy on port 8443.

**Proxy config:** `/root/proxy.env` — token, ACME certs in `/certs/`, geolocation
DB auto-downloaded. Certs are per-domain Let's Encrypt via TLS-ALPN-01 challenge.

> **Dependency note:** NetBird is an **access layer only** — it exists so a
> human on a PC or away from the LAN can reach services by URL. It is **not**
> a dependency of any service backend. Per Section 7, every config and agent
> must reach backends by LAN IP; the `*.sysloggh.net` URLs are reserved for
> human browsers.

#### Known Issues

**A. Access log bloat (CRITICAL)** — 2026-07-03 discovery. The management server
writes an `access_log_entries` row for every proxied request (~45K/day). SQLite
has no built-in TTL. By 2026-07-03 the table reached 10.7M rows / 5.4GB, causing:
- DB writes slow → gRPC deadlines exceeded → proxy loses management connection
- On proxy restart, management server too slow to push route mappings
- Proxy starts HTTPS listener before routes arrive → all domains "unknown"
- **Fix:** batch-delete old entries, VACUUM. Prevention: weekly systemd timer (`netbird-cleanup.timer`) deletes entries >7 days old when DB exceeds 500MB.
- **Recurrence signal:** proxy logs show `unknown domain "git.sysloggh.net"` or `match: false` for known domains → management route sync failed.

**B. Stale management state after proxy restart.** Management server sometimes
stops pushing route mappings to a reconnecting proxy. Symptom: proxy starts,
mapping stream established, but no routes arrive → all domains `unknown`.
**Fix:** restart management server FIRST, THEN proxy. Order matters.

**C. Peer `sha-OWLpE8nS4Rp4IWzep+y2ayFeZ6C3lJ4eNRB29L5yQTE=` flapping.**
One peer (connecting through Traefik relay at 172.30.0.10) disconnects every
~2-3 minutes. This may be a symptom of DB bloat (healthcheck timeout due to
slow DB writes) rather than a network issue. Monitor after bloat fix.

#### Recovery Procedures

1. **Proxy losing all routes ("unknown domain" for all services):**
   ```bash
   ssh root@72.61.0.17
   cd /root
   docker compose restart netbird-server
   sleep 12
   docker compose restart proxy
   ```
2. **Access log bloat detected (DB > 500MB):**
   ```bash
   ssh root@72.61.0.17
   /root/cleanup-access-logs.sh
   ```
3. **VPS hung (TCP accepts, SSH banner timeout):** hard reboot from Hostinger console.

### Checks (every 60s)

```
## Maintains

- docker-health: { ecosystem: string, healthy: int, unhealthy: int, total: int }
- container-status: { name: string, status: string, restarts: int, image: string }

## Checks

For each Docker host:
  - docker ps --format "{{.Names}}" → every required container is running
  - docker inspect <name> --format "{{.State.Health.Status}}" → "healthy"
  - docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
  - df -h / | awk '{print $5}' → usage < 80%

For Netbird VPS specifically:
  - netbird-store-db-size: du -m /var/lib/docker/volumes/root_netbird_data/_data/store.db → < 500MB
  - netbird-access-log-count: sqlite3 store.db 'SELECT COUNT(*) FROM access_log_entries' → < 500K
  - If DB > 500MB: alert + run /root/cleanup-access-logs.sh
  - proxy-route-check: curl https://git.sysloggh.net/ → 2xx (proves routes loaded)
  - proxy-sync-check: docker logs netbird-proxy --since 60s | grep 'Initial mapping sync complete' → present after restart

For docker-vm specifically:
  - mountpoint -q /media/storage → NFS mounted
  - mountpoint -q /media/mediastore → NFS mounted
  - docker compose ls → expected stacks present

## Remediations

- Container unhealthy → `docker compose up -d <name>`, wait 10s, re-check
- Container missing → `docker compose up -d` in its stack directory
- Docker daemon down → `systemctl restart docker`
- NFS mount lost → `mount -a`, if fails → alert (storepve issue)
- Disk > 80% → alert via Zulip DM
- Disk > 95% → crit + relay to maintainer
```

## Section 4: Storage — Monitoring

### NFS Mounts (docker-vm → storepve)

| Mount | Export | Capacity | Used | Alert |
|-------|--------|----------|------|-------|
| /media/storage | storepve:/media/storage | 3.6TB | 264GB (8%) | None |
| /media/mediastore | storepve:/media/mediastore | 7.3TB | 5.2TB (75%) | ⚠️ warn at 80% |

### Proxmox Storage Backends

| Storage | Type | Content | Active |
|---------|------|---------|--------|
| local | dir | ISO, vztmpl, backup | ✅ |
| local-lvm | lvmthin | images, rootdir | ✅ |
| zfs-vm | nfs | rootdir, images | ✅ |
| mediastore | dir | images, backup, rootdir | ✅ |
| zfs-iso | nfs | vztmpl, iso | ✅ |
| pbs-backup | pbs | backup | ✅ |

### Checks (every 10 min)

```
## Maintains

- storage-usage: { mount: string, used_pct: float, growth_rate: float }
- backup-status: { last_success: timestamp, age_hours: int }
- snapshot-status: { pool: string, newest_age_hours: int }

## Specific Alerts

- mediastore growth rate > 10GB/day → warn (check what's writing)
- mediastore > 90% → crit (only 730GB remaining)
- No PBS backup in 48h → fail
```

### Backup Safety Preconditions (2026-09-15)

#### Background & Rationale

Two incidents from 2026-09-13/14 demonstrate that backup operations can catastrophically fail when storage conditions are not verified first:

1. **acerpve thin-pool VM 101** (acerpve, 192.168.68.9, 2026-09-13): A snapshot-mode vzdump of VM 101 on acerpve filled the LVM thin pool. `dmsetup status pve-data-tpool` showed `thin-pool Error` (then `Fail`), the host root remounted `emergency_ro`, ordinary commands failed with I/O errors, LVM tools returned nothing and VM 101 (the RTX 3090 host) went unreachable while the host still answered ping and ssh. It happened TWICE in one day with different modes: snapshot at 13:39Z and a `--mode stop` cold run at 18:52Z. Both times a reboot rolled the failed transaction back and the pool returned rw (~30% data, ~1.2% metadata). Pool capacity was NOT the obvious explanation - ~816G with ~572G free - which is why the metadata/snapshot-pressure hypothesis stands unproven. A full or errored thin pool fails EVERY volume on the VG at once, including the host root.

2. **amdpve 0700 tmpdir** (amdpve, 192.168.68.15, 2026-09-14): A custom vzdump `tmpdir` created with mode 0700 broke a whole night of container backups: `fstat "<dir>/vzdumptmp<n>_<ct>//." failed - EACCES`, because the archive step runs through an unprivileged user namespace and could not traverse a root-owned 0700 directory. Fixed with `chmod 1777` (match /var/tmp) and proved with a real backup.

3. **acerpve GPU-host fact**: VM 101 (llm-gpu) and VM 103 (ocu-llm) are in NO scheduled job, so their only coverage is one-off runs - and for VM 101 that is deliberate until the thin-pool is understood.

> ⚠️ **Hostname Resolution Warning (2026-09-15)**: The PVE node hostnames (acerpve, amdpve, minipve, storepve, ocupve) all resolve to the VPS (72.61.0.17, the Netbird VPS at srv1079750.hstgr.cloud) via the wildcard `*.dns.sysloggh.net` record, NOT to the actual nodes. So `ssh acerpve` lands on the VPS. **Nodes must be addressed by IP**: acerpve 192.168.68.9, amdpve 192.168.68.15, storepve 192.168.68.6, minipve 192.168.68.12, ocupve 192.168.68.5. Guest CTs are reached through their node (`pct exec`). Guest hostnames that resolve on the LAN (e.g. kagentz = 192.168.68.14) are fine. (The DNS address records are a separate decision — row: dag-daemon-node-hostnames-resolve-to-the-vps-20260915.)

#### PREFLIGHT Preconditions (Before ANY snapshot-mode backup on thin-pool hosts)

Before starting ANY snapshot-mode vzdump on a host whose storage is an LVM thin pool, the following checks MUST pass:

```bash
# Check 1: Pool headroom (PRIMARY - yields percentages directly)
# Run on the NODE (e.g. ssh root@192.168.68.9 for acerpve) — NOT by bare hostname, see warning above
lvs -o lv_name,data_percent,metadata_percent,lv_size pve/data
# Example output (acerpve, 192.168.68.9):
#   LV   Data%  Meta%  LSize
#   data 29.95  1.22   <816.21g
# Required thresholds (documented minimum):
# data_percent < 90%  (80% recommended for safety margin)
# metadata_percent < 70%  (metadata fills faster than data)

# Check 2: Verify pool is not in error state (dmsetup shows the raw DM device)
# Run on the NODE (e.g. ssh root@192.168.68.9 for acerpve) — NOT by bare hostname
dmsetup status pve-data-tpool | grep -q "Error\|Fail" && exit 1
# dmsetup status pve-data-tpool field order (verified on 192.168.68.9):
#   $1=start  $2=length  $3="thin-pool"  $4=transaction-id
#   $5=metadata_used/metadata_total (blocks)  $6=data_used/data_total (sectors)
#   remaining fields are flags ("-", "rw", "discard_passdown", "queue_if_no_space", ...)
# This is only used for the ERROR-STATE check; use the lvs command above for percentages.
#   metadata_percent = $5 / ($5 split by /) [second number in pair]
```

**Minimum thresholds**: If either `data_percent >= 90%` or `metadata_percent >= 70%`, the backup MUST NOT start. State explicitly that these are hard stops, not warnings.

**Why this is a precondition**: A full or errored thin pool fails EVERY volume on the VG at once, including the host root. This is not a soft failure - it takes down the entire Proxmox host.

#### Staging Directory Requirement (2026-09-14 incident)

Any custom vzdump `tmpdir` MUST be world-traversable and writable exactly like `/var/tmp` (mode 1777). The archive step of vzdump runs in an unprivileged user namespace and cannot traverse a root-owned 0700 directory.

**Symptom to recognize**: `fstat "<dir>/vzdumptmp<n>_<ct>//." failed - EACCES` on every container in the backup run.

**Fix**: `chmod 1777 <custom-tmpdir>` before starting vzdump.

#### Task Start Rule for Truncating Shells

When starting a backup task from a shell that may truncate output (e.g., pipes, `head`), always use:

```bash
pvesh create /storage/backup --output-format json -- ... | head -2
# ❌ Can kill the backup task ("broken pipe" status)
```

Instead, capture JSON output without piping to truncating commands:

```bash
# Use --output-format json and capture to variable
result=$(pvesh create /storage/backup --output-format json -- ...)
# Then parse result if needed
```

#### GPU Host Backup Status (acerpve VM 101)

VM 101 (llm-gpu) and VM 103 (ocu-llm) have NO scheduled backup job. Coverage is manual one-off runs only. This is intentional for VM 101 until the thin-pool failure mechanism is understood and documented.

## Section 5: Network Services — Monitoring

### 5.1 Service Inventory

The `Resolves To` column is the load-bearing one. Services that CNAME to
`netbird.sysloggh.net` (72.61.0.17) are routed through the NetBird reverse
proxy and **fail whenever NetBird blips**, even though their LAN backend is
fine. Services that resolve directly to a LAN IP are NetBird-independent.

| Service | Domain | LAN Backend | Resolves To | NetBird-dep | Status |
|---------|--------|-------------|-------------|-------------|--------|
| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
| Authentik | auth.sysloggh.net:443 | 192.168.68.11:9000 | CNAME → netbird | **Yes** | ⚠️ |
| Gitea | git.sysloggh.net:443 | 192.168.68.17:3000 | CNAME → netbird | **Yes** | ⚠️ |
| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
| Pulse | pulse.sysloggh.net:443 | 192.168.68.7:7655 | CNAME → netbird | **Yes** | ✅ verified 2026-08-01 |
| DNS UI | dns.sysloggh.net:443 | 192.168.68.10:80 | CNAME → netbird | **Yes** | ⚠️ |
| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |

**Verified 2026-07-24:** NetBird VPS rebooted after a hang; all CNAME'd
services recovered. LAN-IP-direct paths stayed up throughout the outage.
Also added `dns.sysloggh.net` route (was missing entirely).
See `scripts/netbird-add-domain.sh` for adding new proxy routes.

### 5.2 Checks (every 2 min)

```
## Maintains

- dns-resolution: { services: [{name, resolves_to, is_lan_ip}] }
- ssl-expiry: { services: [{name, days_remaining}] }
- endpoint-reachability: { services: [{name, http_code}] }
- netbird-dependency: { services: [{name, depends_on_netbird: bool}] }
```

### 5.3 Network Verification & Routing (every 5 min)

This block is what catches a NetBird blip before it becomes an outage. It
proves two independent paths for every service: the **URL path** (what a
browser uses, may cross NetBird) and the **LAN-IP path** (what configs and
agents must use, never crosses NetBird).

```
## Maintains

- dual-path-reachability: {
    services: [{
      name: string,
      url_path: { http_code, tls_ok },          # the *.sysloggh.net path
      lan_path:  { ip, port, http_code },        # the IP:port path
      lan_independent: bool                       # lan_path works when url_path fails
    }]
  }
- netbird-ingress-health: {
    vps_up: bool,              # TCP 22/80/443 accept on 72.61.0.17
    ssh_banner_ok: bool,       # banner exchange completes (catches hung box)
    containers_up: int,        # all 5 netbird-* containers running
    proxy_routes_ok: int       # traefik routers resolving (chat/git/auth return 2xx/3xx)
  }
- routing-regression: {
    new_netbird_cnames: [domain],   # domains that newly CNAME to netbird.sysloggh.net
                                    # → alert: a service just became NetBird-dependent
    config_url_violations: [file]   # configs/agents found referencing a *.sysloggh.net
                                    # URL where a LAN IP is required (see Section 7)
  }

## Checks

- For each service in 5.1:
  - curl https://<domain>/ → record url_path (code, tls_ok)
  - curl http://<lan_ip>:<port>/ → record lan_path
  - If url_path fails AND lan_path succeeds → netbird-degraded, NOT service-down
  - If BOTH fail → service-down (real outage)

- NetBird ingress (72.61.0.17):
  - TCP probe 22, 80, 443 → all accept
  - SSH banner exchange completes within 10s (catches the hung-box symptom
    where TCP accepts but sshd never sends its banner)
  - ssh root@72.61.0.17 'docker ps' → 5/5 netbird-* containers Up
  - curl https://chat.sysloggh.net/ → 2xx/3xx (proxy route alive)

- DNS dependency drift:
  - For each *.sysloggh.net, dig +short → flag any new CNAME → netbird.sysloggh.net
  - A service moving FROM LAN-IP TO netbird CNAME is a regression (alert)
  - A service moving FROM netbird CNAME TO LAN-IP is an improvement (log)

## Remediations

- NetBird VPS hung (TCP up, SSH banner timeout, TLS hang):
  → This is a host-level hang, not a service fault. Cannot self-remediate via SSH.
  → Alert crit immediately: "NetBird VPS hung — needs hard reboot from provider console"
  → Do NOT declare backend services down; their LAN-IP paths are still up.
- Single proxy route missing (e.g. git returns 000 but chat ok):
  → ssh root@72.61.0.17 'docker restart netbird-traefik'
  → re-check after 30s
- New NetBird CNAME detected:
  → Alert: "<domain> became NetBird-dependent — violates IP-first doctrine (Section 7)"
- Config URL violation detected:
  → Alert with file + line; do not auto-edit configs
```

## Section 6: Alert Routing & Escalation

### Severity Levels

| Severity | Channel | Format | Rate Limit |
|----------|---------|--------|-----------|
| **warn** | Zulip DM to owner | "⚠️ {check}: {detail}" | 1x/check/hour |
| **crit** | Zulip DM + #agent-hub | "🚨 {check}: {detail}" | Immediate |
| **escalated** | Zulip DM + relay to maintainer | "🔥 {check}: {detail}" | Immediate |

### Escalation Chain

```
Container unhealthy (3 consecutive failures)
  → Alert owner
  → 15 min no response → Relay to maintainer

Node offline
  → Alert owner
  → 5 min → Attempt restart via PVE API
  → 2 failures → Escalate

Storage > 95%
  → Alert owner + maintainer relay
  → Immediate action required

SSL cert < 7 days
  → Alert owner daily
  → 3 days out → Escalate

ZFS pool degraded
  → Cannot auto-fix → Escalate immediately
```

## Section 7: Configuration Doctrine — IP-First, URLs for Browsers

This doctrine is the structural fix for the NetBird-dependency outage. It is
enforced by the `routing-regression.config_url_violations` check in Section
5.3.

### Rule

1. **Service configs use LAN IP addresses, always.** Any file that wires a
   service, agent, monitor, or integration to another internal service must
   reference the backend by its `192.168.68.x` LAN IP and port — never by a
   `*.sysloggh.net` URL. NetBird may blip at any time; LAN IPs do not.
2. **`*.sysloggh.net` URLs are reserved for human browser access.** They are
   a convenience layer (split-horizon DNS on the LAN, NetBird reverse proxy
   off the LAN) for a person at a keyboard. They must never be a load-bearing
   dependency in code or config.
3. **Agents and monitors reach backends by LAN IP.** Abiba, Tanko, and Mumuni
   all run on the LAN; there is no reason for them to hairpin through NetBird.
   The daily infra report and all `ssh`/`curl` checks already follow this.
4. **DNS records should prefer LAN IPs over NetBird CNAMEs.** A service that
   can resolve directly to its LAN IP (like `litellm` → `192.168.68.116` and
   `minipve` → `192.168.68.12`) is NetBird-independent. Moving a record from
   a NetBird CNAME to a LAN IP is an improvement; the reverse is a regression.
5. **NetBird is for remote/PC access only.** When you are out or on a PC,
   NetBird carries you in. When you are on the LAN, NetBird is not in the
   path. Configs must reflect this — they always assume LAN.

### Canonical IP map (configs MUST use these)

| Service | Use in configs | URL (browsers only) |
|---------|----------------|--------------------|
| Proxmox API | `https://192.168.68.12:8006` | `https://minipve.sysloggh.net` |
| LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
| LiteLLM (nginx) | `http://192.168.68.116` | — |
| Grafana | `http://192.168.68.116:3001` | — |
| Authentik | `https://192.168.68.11:9000` | `https://auth.sysloggh.net` |
| Gitea | `http://192.168.68.17:3000` | `https://git.sysloggh.net` |
| Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` |
| SearXNG | `http://192.168.68.7:8888` | — |
| Firecrawl | `http://192.168.68.7:3002` | — |
| RA-H OS bridge | `http://192.168.68.65:3100` | — |

### Violation examples (to be flagged by Section 5.3)

- ❌ An agent config setting `LITELLM_BASE_URL=https://litellm.sysloggh.net`
- ✅ Same config setting `LITELLM_BASE_URL=http://192.168.68.116:4000`
- ❌ A monitor curling `https://chat.sysloggh.net/api/v1/server_settings`
- ✅ A monitor curling `http://192.168.68.19/api/v1/server_settings` (and
      optionally the URL as a separate browser-path check)

### Goal state

Every `*.sysloggh.net` record resolves to a LAN IP (split-horizon DNS on LAN,
NetBird reverse proxy off LAN). NetBird then becomes purely the off-LAN
ingress — if it blips, only remote browser users notice, and no agent,
monitor, or integration breaks.

## Appendix A: Quick Health Commands

```bash
# Full cluster status
PVE="https://minipve.sysloggh.net"
AUTH="Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»"
curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"

# Docker health from Abiba
ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'"
ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'"

# GPU fleet quick check
curl -s http://192.168.68.116/health/unified | jq .status
curl -s http://192.168.68.24:9100/gpu-data | jq .summary

# Grafana status
curl -s http://admin:$(infisical secrets get GRAFANA_ADMIN_PASSWORD --project=infrastructure --env=production --plain)@192.168.68.116:3001/api/health

# Prometheus targets
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

# LiteLLM key check
curl -s -H "Authorization: Bearer $(infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production --plain)" \
  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'

# Storage check
ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"


# Restart stuck GPU (saturation watchdog alternative)
ssh root@192.168.68.8 "systemctl restart llama-server"
ssh root@192.168.68.110 "systemctl restart llama-server"
```

## Appendix B: CT Inventory

| CT | Name | Node | IP | Role | Agent |
|----|------|------|----|------|-------|
| 100 | abiba | minipve | .24 | Pi agent | ✅ pi |
| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
| 102 | adguard | **minipve** | **.10** | DNS | ❌ |
| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
| 104 | authentik | minipve | .11 | OIDC | ❌ |
| 105 | kagentz | amdpve | — | Agent Zero | ✅ |
| 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
| 107 | pbs | storepve | — | Backups | ❌ |
| 108 | media | storepve | — | Media | ❌ |
| 109 | docker-vm | storepve | .7 | Docker host | ❌ |
| 110 | gitea | minipve | **.17** | Git | ❌ |
| 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
| 112 | tanko | minipve | .122 | hybrid (DSH + Hermes) agent | ✅ |
| 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
| 117 | zulip | storepve | .19 | Chat | ❌ |
| 118 | jdownloader | storepve | .20 | JDownloader LXC (dedicated, migrated from docker-vm 2026-08-01) | ✅ |
| 119 | infisical-vault | minipve | — | Vault | ❌ |
| 120 | adguard2 | amdpve | — | DNS (secondary AdGuard) | ❌ |

## Appendix C: Docker Compose Files Location

| Host | Stack | Compose File |
|------|-------|-------------|
| docker-vm (.7) | Firecrawl | `/opt/search-stack/firecrawl-source/docker-compose.yaml` |
| docker-vm (.7) | SearXNG | `/opt/search-stack/searxng/docker-compose.yml` |
| docker-vm (.7) | Home stack | `/opt/home_stack/docker-compose.yml` |
| docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
| CT 116 (.116) | Inference Harness | `/opt/inference-harness/docker-compose.yml` |
| Netbird (.17) | Netbird | Docker run (not compose) |

## CT Access (pct-run — no IPs needed)

All CTs are accessible via `pct-run <CT_ID> <command>`. No IP addresses required.
Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.sh`.

| CT | Name | Node | pct-run |
|-----|------|------|---------|
| 100 | abiba | minipve | `pct-run 100` |
| 105 | kagentz | amdpve | `pct-run 105` |
| 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) |
| 112 | tanko | minipve | `pct-run 112` |
| 113 | baggy | amdpve | `pct-run 113` |
| 115 | scottdenya | amdpve | `pct-run 115` |
| 104 | authentik | minipve | `pct-run 104` |
| 110 | gitea | minipve | `pct-run 110` |
| 116 | syslog-api | minipve | `pct-run 116` |
| 106 | ra-h-os | storepve | `pct-run 106` |
| 107 | proxmox-backup | storepve | `pct-run 107` |
| 108 | media | storepve | `pct-run 108` |
| 117 | zulip | storepve | `pct-run 117` |
| 118 | jdownloader | storepve | `pct-run 118` |
| 119 | infisical-vault | minipve | `pct-run 119` |
| 120 | adguard2 | amdpve | `pct-run 120` |
| 102 | adguard | **minipve** | `pct-run 102` |

GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
```bash
ssh root@192.168.68.8   # RTX 3090
ssh root@192.168.68.110 # RTX 5070
ssh root@192.168.68.15  # Strix Halo
ssh root@192.168.68.4   # hwepve — standalone London node + NetBird routing peer (relocation pending)
```

## Section 7: Agent Health Check (consolidated — 2026-07-05)

Replaces 7 scattered Zulip health scripts with a single non-disruptive check
running every 10 minutes via cron (`/root/scripts/agent-health-check.py`).

The script is **read-only** — it never restarts, kills, or modifies anything.
Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh,
which was kill+nohup outside systemd) are banned by policy.

### Checks Performed

| Check | Frequency | What It Detects |
|-------|-----------|-----------------|
| LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models |
| GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
| Gateway liveness | 10 min | Gateway process running, state file readable |
| Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
| Recent errors | 10 min | Error count in journald for last 10 min |

### Disabled Scripts

| Script | Why Disabled |
|--------|-------------|
| `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch |
| `zulip-monitor.sh` (Abiba) | Standalone cron replaced by agent-health-check.py + PM2 auto-restart; the script itself remains active as the execution step of the `zulip-health` responsibility contract (see `zulip-health.prose.md`) |