Files
prose-contracts/infrastructure-control.prose.md
T

26 KiB

kind, name, description
kind name description
pattern infrastructure-control Full infrastructure monitoring and control pattern covering the 5-node Proxmox cluster, 3 Docker ecosystems (22 containers), NFS storage, and network services. Defines monitors, remediations, and the access matrix for all environments.

Infrastructure Control Pattern

Topology

                    ┌─────────────────────────┐
                    │   OpenProse Contract    │
                    │  (declares what's true) │
                    └──────────┬──────────────┘
                               │
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
   ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
   │   Abiba     │    │   Tanko      │    │   Mumuni     │
   │   (pi)      │    │   (Hermes)   │    │   (Hermes)   │
   │   CT 100    │    │   CT 122     │    │   CT 114     │
   └──────┬──────┘    └──────┬───────┘    └──────┬───────┘
          │                  │                    │
          └──────────────────┼────────────────────┘
                             ▼
          ┌──────────────────────────────────────┐
          │         Proxmox Cluster API          │
          │    minipve.sysloggh.net:443          │
          │   (monitoring@pve!mumuni token)      │
          └────┬──────┬──────┬──────┬──────┬─────┘
               │      │      │      │      │
          ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
          ▼      ▼      ▼      ▼      ▼
       minipve amdpve storepve acerpve ocupve
        (.12)   (.15)   (.6)    (.9)    (.5)

          ▼
   ┌─────────────────────────────────────────────┐
   │            Docker Ecosystems                │
   │  ┌────────────┬────────────┬──────────────┐ │
   │  │ docker-vm  │ CT 116     │ Netbird      │ │
   │  │ (.7)       │ (.116)     │ (72.61.0.17) │ │
   │  │ 11 ctrs    │ 6 ctrs     │ 5 ctrs       │ │
   │  └────────────┴────────────┴──────────────┘ │
   └─────────────────────────────────────────────┘

Section 1: Access Matrix

API Tokens & Credentials

Resource Auth Method Credential Source Status
Proxmox Cluster PVE API Token monitoring@pve!mumuni=...
Proxmox Root Password via API ticket root@pam:kakashi19
docker-vm (.7) SSH root SSH key
CT 116 (syslog-api) SSH root SSH key
Tanko CT (.122) SSH jerome id_ed25519
Mumuni CT (.123) SSH root id_ed25519
Koonimo CT (.114) SSH jerome no key access
Netbird (.17) SSH root SSH key
Gitea API token abiba-bot token
Zulip Bot API key abiba-bot@chat.sysloggh.net
RA-H OS MCP bridge port 3100
LiteLLM Admin master key sk-litellm-7f96...
Grafana admin password syslog-grafana-2026

Reachability Matrix

From / To PVE API docker-vm (.7) CT 116 Tanko (.122) Mumuni (.123) Koonimo (.114)
Abiba (.24) :443 SSH SSH SSH jerome SSH root SSH
Tanko (.122) via NetBird
Mumuni (.123)

Conclusion: Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.

Section 2: Proxmox Cluster — Monitoring

Nodes (5)

Node IP CPU RAM VMs/CTs Role
minipve .12 16C 30GB authentik, gitea, mumuni, syslog-api, jitsi Auth, git, messaging
amdpve .15 32C 62GB abiba, kagentz, tanko, tdunna, baggy, scottdenya Agents, compute
storepve .6 28C 31GB docker-vm, ra-h-os, PBS, media, zulip Docker, storage, chat
acerpve .9 28C 31GB llm-gpu, adguard GPU VMs
ocupve .5 12C 14GB ocu-llm GPU VMs

Checks (every 5 min)

## Maintains

- cluster-status: { online_nodes: int, offline_nodes: int, timestamp }
- node-cpu-usage: { node: pct, warnings: [] }
- node-memory-usage: { node: free_pct, warnings: [] }
- node-uptime: { node: seconds, just_rebooted: bool }

## Checks

- For each node in [minipve, amdpve, storepve, acerpve, ocupve]:
  - GET /api2/json/nodes/{node}/status → check status == "online"
  - GET /api2/json/nodes/{node}/status → cpu < 0.80
  - GET /api2/json/nodes/{node}/status → free_mem > 10%
  - GET /api2/json/nodes/{node}/status → uptime > 300 (warn if just rebooted)

- For each CT in inventory (19 total):
  - GET /api2/json/cluster/resources → filter by type=lxc
  - Warn if status != "running"

- Storage pools:
  - GET /api2/json/nodes/storepve/storage/mediastore → used < 80%
  - GET /api2/json/nodes/storepve/storage/pbs-backup → last backup < 48h

## Remediations

- Node offline → alert via Zulip DM, escalate after 3x
- CT stopped → `pct start <vmid>` via PVE API, verify after 30s
- VM stopped → `qm start <vmid>` via PVE API, verify after 60s
- Storage > 80% → warn via Zulip DM
- Storage > 95% → crit via Zulip + relay to maintainer

Section 3: Docker Ecosystems — Monitoring

Ecosystem A: docker-vm (192.168.68.7)

11 containers across 4 compose stacks:

Stack Path Containers
Firecrawl /opt/search-stack/firecrawl-source/ api, rabbitmq, postgres, playwright, redis
SearXNG /opt/search-stack/searxng/ searxng, valkey
Home stack /opt/home_stack/ jdownloader, stirling-pdf, pulse
Audiobookshelf /opt/audiobookshelf/ audiobookshelf

Home Stack Services

Stirling-PDF (deployed 2026-07-03):

  • URL: http://192.168.68.7:8989
  • Swagger: http://192.168.68.7:8989/swagger-ui.html
  • Admin credentials: admin / kakashi20stirling
  • API key: adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88
  • Compose: /opt/home_stack/docker-compose.yml
  • Control script: /opt/home_stack/infra-control.sh

JDownloader:

  • URL: http://192.168.68.7:5800 (web UI via VNC)

Pulse (Uptime Kuma):

  • URL: http://192.168.68.7:3001

Ecosystem B: CT 116 syslog-api (192.168.68.116)

8 containers in inference-harness stack:

Container Image Port Role
harness-litellm berriai/litellm:1.90.0-rc.1 :4000→:4001 API proxy, key mgmt, fallbacks
harness-router inference-harness-router :9000 (127.0.0.1) GPU routing, slot booking, CB
harness-nginx nginx:alpine :80 Entrypoint, /v1→LiteLLM, /dashboard/, /grafana/
harness-postgres postgres:16-alpine :5432 LiteLLM DB (keys, spend, config)
harness-redis redis:7-alpine :6379 Router slots, circuit breakers
harness-dashboard inference-harness-dashboard :3000 SyslogAI Harness UI
harness-grafana grafana/grafana :3000→:3001 GPU dashboards via Prometheus
harness-prometheus prom/prometheus :9090 Metrics scraper, 5 targets

Nginx routing:

  • /v1/* → harness-litellm:4000 (API)
  • /admin/* → harness-litellm:4000
  • /dashboard/ → harness-dashboard:3000
  • /litellm/* → harness-litellm:4000
  • /grafana/ → harness-grafana:3000
  • /health/* → harness-litellm:4000/health/liveliness
  • /gpu/ → 192.168.68.24:9100 (fleet monitor)

Prometheus targets:

  • 192.168.68.8:9400 (RTX 3090 — qwen)
  • 192.168.68.110:9400 (RTX 5070 — gemma)
  • 192.168.68.15:9400 (Strix Halo — ornith)
  • 192.168.68.24:9401 (Router metrics exporter)
  • harness-litellm:4000 (LiteLLM health)

Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud)

5 containers, single compose stack at /root/docker-compose.yml:

Container Image Role
netbird-server netbirdio/netbird-server:latest VPN controller, management API
netbird-dashboard netbirdio/dashboard:latest Web management UI
netbird-proxy netbirdio/reverse-proxy:latest SNI router + TLS passthrough for *.sysloggh.net
netbird-crowdsec crowdsecurity/crowdsec:v1.7.7 WAF / threat intelligence
netbird-traefik traefik:v3.6 Edge reverse proxy, Let's Encrypt TLS

Architecture: Internet → Traefik (TLS + HTTP→HTTPS redirect) → Netbird proxy (TLS passthrough via HostSNI, PROXY protocol) → Netbird mesh → LAN backends. Traefik handles the Netbird dashboard directly; all other *.sysloggh.net domains are TLS-passthrough to the proxy on port 8443.

Proxy config: /root/proxy.env — token, ACME certs in /certs/, geolocation DB auto-downloaded. Certs are per-domain Let's Encrypt via TLS-ALPN-01 challenge.

Dependency note: NetBird is an access layer only — it exists so a human on a PC or away from the LAN can reach services by URL. It is not a dependency of any service backend. Per Section 7, every config and agent must reach backends by LAN IP; the *.sysloggh.net URLs are reserved for human browsers.

Known Issues

A. Access log bloat (CRITICAL) — 2026-07-03 discovery. The management server writes an access_log_entries row for every proxied request (~45K/day). SQLite has no built-in TTL. By 2026-07-03 the table reached 10.7M rows / 5.4GB, causing:

  • DB writes slow → gRPC deadlines exceeded → proxy loses management connection
  • On proxy restart, management server too slow to push route mappings
  • Proxy starts HTTPS listener before routes arrive → all domains "unknown"
  • Fix: batch-delete old entries, VACUUM. Prevention: weekly systemd timer (netbird-cleanup.timer) deletes entries >7 days old when DB exceeds 500MB.
  • Recurrence signal: proxy logs show unknown domain "git.sysloggh.net" or match: false for known domains → management route sync failed.

B. Stale management state after proxy restart. Management server sometimes stops pushing route mappings to a reconnecting proxy. Symptom: proxy starts, mapping stream established, but no routes arrive → all domains unknown. Fix: restart management server FIRST, THEN proxy. Order matters.

C. Peer sha-OWLpE8nS4Rp4IWzep+y2ayFeZ6C3lJ4eNRB29L5yQTE= flapping. One peer (connecting through Traefik relay at 172.30.0.10) disconnects every ~2-3 minutes. This may be a symptom of DB bloat (healthcheck timeout due to slow DB writes) rather than a network issue. Monitor after bloat fix.

Recovery Procedures

  1. Proxy losing all routes ("unknown domain" for all services):
    ssh root@72.61.0.17
    cd /root
    docker compose restart netbird-server
    sleep 12
    docker compose restart proxy
    
  2. Access log bloat detected (DB > 500MB):
    ssh root@72.61.0.17
    /root/cleanup-access-logs.sh
    
  3. VPS hung (TCP accepts, SSH banner timeout): hard reboot from Hostinger console.

Checks (every 60s)

## Maintains

- docker-health: { ecosystem: string, healthy: int, unhealthy: int, total: int }
- container-status: { name: string, status: string, restarts: int, image: string }

## Checks

For each Docker host:
  - docker ps --format "{{.Names}}" → every required container is running
  - docker inspect <name> --format "{{.State.Health.Status}}" → "healthy"
  - docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
  - df -h / | awk '{print $5}' → usage < 80%

For Netbird VPS specifically:
  - netbird-store-db-size: du -m /var/lib/docker/volumes/root_netbird_data/_data/store.db → < 500MB
  - netbird-access-log-count: sqlite3 store.db 'SELECT COUNT(*) FROM access_log_entries' → < 500K
  - If DB > 500MB: alert + run /root/cleanup-access-logs.sh
  - proxy-route-check: curl https://git.sysloggh.net/ → 2xx (proves routes loaded)
  - proxy-sync-check: docker logs netbird-proxy --since 60s | grep 'Initial mapping sync complete' → present after restart

For docker-vm specifically:
  - mountpoint -q /media/storage → NFS mounted
  - mountpoint -q /media/mediastore → NFS mounted
  - docker compose ls → expected stacks present

## Remediations

- Container unhealthy → `docker compose up -d <name>`, wait 10s, re-check
- Container missing → `docker compose up -d` in its stack directory
- Docker daemon down → `systemctl restart docker`
- NFS mount lost → `mount -a`, if fails → alert (storepve issue)
- Disk > 80% → alert via Zulip DM
- Disk > 95% → crit + relay to maintainer

Section 4: Storage — Monitoring

NFS Mounts (docker-vm → storepve)

Mount Export Capacity Used Alert
/media/storage storepve:/media/storage 3.6TB 264GB (8%) None
/media/mediastore storepve:/media/mediastore 7.3TB 5.2TB (75%) ⚠️ warn at 80%

Proxmox Storage Backends

Storage Type Content Active
local dir ISO, vztmpl, backup
local-lvm lvmthin images, rootdir
zfs-vm nfs rootdir, images
mediastore dir images, backup, rootdir
zfs-iso nfs vztmpl, iso
pbs-backup pbs backup

Checks (every 10 min)

## Maintains

- storage-usage: { mount: string, used_pct: float, growth_rate: float }
- backup-status: { last_success: timestamp, age_hours: int }
- snapshot-status: { pool: string, newest_age_hours: int }

## Specific Alerts

- mediastore growth rate > 10GB/day → warn (check what's writing)
- mediastore > 90% → crit (only 730GB remaining)
- No PBS backup in 48h → fail

Section 5: Network Services — Monitoring

5.1 Service Inventory

The Resolves To column is the load-bearing one. Services that CNAME to netbird.sysloggh.net (72.61.0.17) are routed through the NetBird reverse proxy and fail whenever NetBird blips, even though their LAN backend is fine. Services that resolve directly to a LAN IP are NetBird-independent.

Service Domain LAN Backend Resolves To NetBird-dep Status
Proxmox API minipve.sysloggh.net:8006 192.168.68.12 LAN IP No
LiteLLM litellm.sysloggh.net 192.168.68.116 LAN IP No
Authentik auth.sysloggh.net:443 192.168.68.11 CNAME → netbird Yes ⚠️
Gitea git.sysloggh.net:443 192.168.68.110 CNAME → netbird Yes ⚠️
Zulip chat.sysloggh.net:443 192.168.68.117 CNAME → netbird Yes ⚠️
Pulse pulse.sysloggh.net:443 192.168.68.7 CNAME → netbird Yes ⚠️
DNS UI dns.sysloggh.net:443 192.168.68.102 CNAME → netbird Yes ⚠️
SearXNG searxng.sysloggh.net:8888 192.168.68.7:8888 LAN IP No
Firecrawl firecrawl.sysloggh.net:3002 192.168.68.7:3002 LAN IP No

Verified 2026-07-02: NetBird VPS rebooted after a hang; all CNAME'd services recovered. LAN-IP-direct paths stayed up throughout the outage.

5.2 Checks (every 2 min)

## Maintains

- dns-resolution: { services: [{name, resolves_to, is_lan_ip}] }
- ssl-expiry: { services: [{name, days_remaining}] }
- endpoint-reachability: { services: [{name, http_code}] }
- netbird-dependency: { services: [{name, depends_on_netbird: bool}] }

5.3 Network Verification & Routing (every 5 min)

This block is what catches a NetBird blip before it becomes an outage. It proves two independent paths for every service: the URL path (what a browser uses, may cross NetBird) and the LAN-IP path (what configs and agents must use, never crosses NetBird).

## Maintains

- dual-path-reachability: {
    services: [{
      name: string,
      url_path: { http_code, tls_ok },          # the *.sysloggh.net path
      lan_path:  { ip, port, http_code },        # the IP:port path
      lan_independent: bool                       # lan_path works when url_path fails
    }]
  }
- netbird-ingress-health: {
    vps_up: bool,              # TCP 22/80/443 accept on 72.61.0.17
    ssh_banner_ok: bool,       # banner exchange completes (catches hung box)
    containers_up: int,        # all 5 netbird-* containers running
    proxy_routes_ok: int       # traefik routers resolving (chat/git/auth return 2xx/3xx)
  }
- routing-regression: {
    new_netbird_cnames: [domain],   # domains that newly CNAME to netbird.sysloggh.net
                                    # → alert: a service just became NetBird-dependent
    config_url_violations: [file]   # configs/agents found referencing a *.sysloggh.net
                                    # URL where a LAN IP is required (see Section 7)
  }

## Checks

- For each service in 5.1:
  - curl https://<domain>/ → record url_path (code, tls_ok)
  - curl http://<lan_ip>:<port>/ → record lan_path
  - If url_path fails AND lan_path succeeds → netbird-degraded, NOT service-down
  - If BOTH fail → service-down (real outage)

- NetBird ingress (72.61.0.17):
  - TCP probe 22, 80, 443 → all accept
  - SSH banner exchange completes within 10s (catches the hung-box symptom
    where TCP accepts but sshd never sends its banner)
  - ssh root@72.61.0.17 'docker ps' → 5/5 netbird-* containers Up
  - curl https://chat.sysloggh.net/ → 2xx/3xx (proxy route alive)

- DNS dependency drift:
  - For each *.sysloggh.net, dig +short → flag any new CNAME → netbird.sysloggh.net
  - A service moving FROM LAN-IP TO netbird CNAME is a regression (alert)
  - A service moving FROM netbird CNAME TO LAN-IP is an improvement (log)

## Remediations

- NetBird VPS hung (TCP up, SSH banner timeout, TLS hang):
  → This is a host-level hang, not a service fault. Cannot self-remediate via SSH.
  → Alert crit immediately: "NetBird VPS hung — needs hard reboot from provider console"
  → Do NOT declare backend services down; their LAN-IP paths are still up.
- Single proxy route missing (e.g. git returns 000 but chat ok):
  → ssh root@72.61.0.17 'docker restart netbird-traefik'
  → re-check after 30s
- New NetBird CNAME detected:
  → Alert: "<domain> became NetBird-dependent — violates IP-first doctrine (Section 7)"
- Config URL violation detected:
  → Alert with file + line; do not auto-edit configs

Section 6: Alert Routing & Escalation

Severity Levels

Severity Channel Format Rate Limit
warn Zulip DM to owner "⚠️ {check}: {detail}" 1x/check/hour
crit Zulip DM + #agent-hub "🚨 {check}: {detail}" Immediate
escalated Zulip DM + relay to maintainer "🔥 {check}: {detail}" Immediate

Escalation Chain

Container unhealthy (3 consecutive failures)
  → Alert owner
  → 15 min no response → Relay to maintainer

Node offline
  → Alert owner
  → 5 min → Attempt restart via PVE API
  → 2 failures → Escalate

Storage > 95%
  → Alert owner + maintainer relay
  → Immediate action required

SSL cert < 7 days
  → Alert owner daily
  → 3 days out → Escalate

ZFS pool degraded
  → Cannot auto-fix → Escalate immediately

Section 7: Configuration Doctrine — IP-First, URLs for Browsers

This doctrine is the structural fix for the NetBird-dependency outage. It is enforced by the routing-regression.config_url_violations check in Section 5.3.

Rule

  1. Service configs use LAN IP addresses, always. Any file that wires a service, agent, monitor, or integration to another internal service must reference the backend by its 192.168.68.x LAN IP and port — never by a *.sysloggh.net URL. NetBird may blip at any time; LAN IPs do not.
  2. *.sysloggh.net URLs are reserved for human browser access. They are a convenience layer (split-horizon DNS on the LAN, NetBird reverse proxy off the LAN) for a person at a keyboard. They must never be a load-bearing dependency in code or config.
  3. Agents and monitors reach backends by LAN IP. Abiba, Tanko, and Mumuni all run on the LAN; there is no reason for them to hairpin through NetBird. The daily infra report and all ssh/curl checks already follow this.
  4. DNS records should prefer LAN IPs over NetBird CNAMEs. A service that can resolve directly to its LAN IP (like litellm192.168.68.116 and minipve192.168.68.12) is NetBird-independent. Moving a record from a NetBird CNAME to a LAN IP is an improvement; the reverse is a regression.
  5. NetBird is for remote/PC access only. When you are out or on a PC, NetBird carries you in. When you are on the LAN, NetBird is not in the path. Configs must reflect this — they always assume LAN.

Canonical IP map (configs MUST use these)

Service Use in configs URL (browsers only)
Proxmox API https://192.168.68.12:8006 https://minipve.sysloggh.net
LiteLLM API http://192.168.68.116:4000 https://litellm.sysloggh.net
LiteLLM (nginx) http://192.168.68.116
Grafana http://192.168.68.116:3001
Authentik https://192.168.68.11 https://auth.sysloggh.net
Gitea http://192.168.68.110:3000 https://git.sysloggh.net
Zulip API http://192.168.68.117 https://chat.sysloggh.net
SearXNG http://192.168.68.7:8888
Firecrawl http://192.168.68.7:3002
RA-H OS bridge http://192.168.68.65:3100

Violation examples (to be flagged by Section 5.3)

  • An agent config setting LITELLM_BASE_URL=https://litellm.sysloggh.net
  • Same config setting LITELLM_BASE_URL=http://192.168.68.116:4000
  • A monitor curling https://chat.sysloggh.net/api/v1/server_settings
  • A monitor curling http://192.168.68.117/api/v1/server_settings (and optionally the URL as a separate browser-path check)

Goal state

Every *.sysloggh.net record resolves to a LAN IP (split-horizon DNS on LAN, NetBird reverse proxy off LAN). NetBird then becomes purely the off-LAN ingress — if it blips, only remote browser users notice, and no agent, monitor, or integration breaks.

Appendix A: Quick Health Commands

# Full cluster status
PVE="https://minipve.sysloggh.net"
AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"

# Docker health from Abiba
ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'"
ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'"

# GPU fleet quick check
curl -s http://192.168.68.116/health/unified | jq .status
curl -s http://192.168.68.24:9100/gpu-data | jq .summary

# Grafana status
curl -s http://admin:syslog-grafana-2026@192.168.68.116:3001/api/health

# Prometheus targets
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

# LiteLLM key check
curl -s -H "Authorization: Bearer sk-litellm-7f96080dd99b15c36bd4b333b58a6796" \
  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'

# Storage check
ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"

# Router roster reload (if needed)
curl -s -X POST http://192.168.68.116:9000/admin/roster/reload \
  -H "Authorization: Bearer sk-admin-ee09fffd04978b61a1569ac670c68814"

# Restart stuck GPU (saturation watchdog alternative)
ssh root@192.168.68.8 "systemctl restart llama-server"
ssh root@192.168.68.110 "systemctl restart llama-server"

Appendix B: CT Inventory

CT Name Node IP Role Agent
100 abiba amdpve .24 Pi agent (this host) pi
101 llm-gpu acerpve .8 GPU RTX 3090
102 adguard acerpve DNS
103 ocu-llm ocupve .110 GPU RTX 5070
104 authentik minipve .11 OIDC
105 kagentz amdpve Agent Zero
106 ra-h-os storepve .65 KG bridge MCP
107 pbs storepve Backups
108 media storepve Media
109 docker-vm storepve .7 Docker host
110 gitea minipve Git
111 tdunna amdpve ?
112 tanko amdpve .122 Hermes agent
113 baggy/koonimo amdpve .114 Hermes agent
114 mumuni minipve .123 Hermes agent
115 scottdenya amdpve ?
116 syslog-api minipve .116 LiteLLM + Grafana
117 zulip storepve Chat
118 jitsi minipve Video

Appendix C: Docker Compose Files Location

Host Stack Compose File
docker-vm (.7) Firecrawl /opt/search-stack/firecrawl-source/docker-compose.yaml
docker-vm (.7) SearXNG /opt/search-stack/searxng/docker-compose.yml
docker-vm (.7) Home stack /opt/home_stack/docker-compose.yml
docker-vm (.7) Audiobookshelf /opt/audiobookshelf/docker-compose.yml
CT 116 (.116) Inference Harness /opt/inference-harness/docker-compose.yml
Netbird (.17) Netbird Docker run (not compose)