- infrastructure-control: add Section 5.3 (Network Verification & Routing) with
dual-path checks, NetBird ingress health, DNS drift detection, and routing
regression alerts. Add Section 7 (Configuration Doctrine — IP-First) mandating
LAN IPs for all configs/agents, URLs reserved for human browser access only.
Also enrich access matrix (Mumuni, Koonimo, LiteLLM Admin, Grafana entries),
reachability matrix, and CT 116 nginx routing + Prometheus targets.
- build-zulip-plugin: fold Success Criteria into Maintains postconditions.
- pi-approval-architecture: new reference doc — pi approval model vs Hermes,
architectural limits, /approve and /deny stub commands.
- zulip-approval-fix: new responsibility — documents the Zulip HTML/prefix bug
that broke /approve and /deny, and the one-line fix applied.
Full infrastructure monitoring and control pattern covering the 5-node Proxmox cluster, 3 Docker ecosystems (22 containers), NFS storage, and network services. Defines monitors, remediations, and the access matrix for all environments.
Dependency note: NetBird is an access layer only — it exists so a
human on a PC or away from the LAN can reach services by URL. It is not
a dependency of any service backend. Per Section 7, every config and agent
must reach backends by LAN IP; the *.sysloggh.net URLs are reserved for
human browsers. The 2026-07-02 outage (NetBird VPS hung) took out every
CNAME'd URL while every LAN-IP backend stayed up — proof the doctrine is
necessary.
Recovery: a hung NetBird VPS (TCP accepts, SSH banner exchange times
out, TLS hangs) cannot be self-remediated over the network. It requires a
hard reboot from the hosting provider's console. See Section 5.3.
Checks (every 60s)
## Maintains
- docker-health: { ecosystem: string, healthy: int, unhealthy: int, total: int }
- container-status: { name: string, status: string, restarts: int, image: string }
## Checks
For each Docker host:
- docker ps --format "{{.Names}}" → every required container is running
- docker inspect <name> --format "{{.State.Health.Status}}" → "healthy"
- docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
- df -h / | awk '{print $5}' → usage < 80%
For docker-vm specifically:
- mountpoint -q /media/storage → NFS mounted
- mountpoint -q /media/mediastore → NFS mounted
- docker compose ls → expected stacks present
## Remediations
- Container unhealthy → `docker compose up -d <name>`, wait 10s, re-check
- Container missing → `docker compose up -d` in its stack directory
- Docker daemon down → `systemctl restart docker`
- NFS mount lost → `mount -a`, if fails → alert (storepve issue)
- Disk > 80% → alert via Zulip DM
- Disk > 95% → crit + relay to maintainer
The Resolves To column is the load-bearing one. Services that CNAME to
netbird.sysloggh.net (72.61.0.17) are routed through the NetBird reverse
proxy and fail whenever NetBird blips, even though their LAN backend is
fine. Services that resolve directly to a LAN IP are NetBird-independent.
Service
Domain
LAN Backend
Resolves To
NetBird-dep
Status
Proxmox API
minipve.sysloggh.net:8006
192.168.68.12
LAN IP
No
✅
LiteLLM
litellm.sysloggh.net
192.168.68.116
LAN IP
No
✅
Authentik
auth.sysloggh.net:443
192.168.68.11
CNAME → netbird
Yes
⚠️
Gitea
git.sysloggh.net:443
192.168.68.110
CNAME → netbird
Yes
⚠️
Zulip
chat.sysloggh.net:443
192.168.68.117
CNAME → netbird
Yes
⚠️
Pulse
pulse.sysloggh.net:443
192.168.68.7
CNAME → netbird
Yes
⚠️
DNS UI
dns.sysloggh.net:443
192.168.68.102
CNAME → netbird
Yes
⚠️
SearXNG
searxng.sysloggh.net:8888
192.168.68.7:8888
LAN IP
No
✅
Firecrawl
firecrawl.sysloggh.net:3002
192.168.68.7:3002
LAN IP
No
✅
Verified 2026-07-02: NetBird VPS rebooted after a hang; all CNAME'd
services recovered. LAN-IP-direct paths stayed up throughout the outage.
This block is what catches a NetBird blip before it becomes an outage. It
proves two independent paths for every service: the URL path (what a
browser uses, may cross NetBird) and the LAN-IP path (what configs and
agents must use, never crosses NetBird).
## Maintains
- dual-path-reachability: {
services: [{
name: string,
url_path: { http_code, tls_ok }, # the *.sysloggh.net path
lan_path: { ip, port, http_code }, # the IP:port path
lan_independent: bool # lan_path works when url_path fails
}]
}
- netbird-ingress-health: {
vps_up: bool, # TCP 22/80/443 accept on 72.61.0.17
ssh_banner_ok: bool, # banner exchange completes (catches hung box)
containers_up: int, # all 5 netbird-* containers running
proxy_routes_ok: int # traefik routers resolving (chat/git/auth return 2xx/3xx)
}
- routing-regression: {
new_netbird_cnames: [domain], # domains that newly CNAME to netbird.sysloggh.net
# → alert: a service just became NetBird-dependent
config_url_violations: [file] # configs/agents found referencing a *.sysloggh.net
# URL where a LAN IP is required (see Section 7)
}
## Checks
- For each service in 5.1:
- curl https://<domain>/ → record url_path (code, tls_ok)
- curl http://<lan_ip>:<port>/ → record lan_path
- If url_path fails AND lan_path succeeds → netbird-degraded, NOT service-down
- If BOTH fail → service-down (real outage)
- NetBird ingress (72.61.0.17):
- TCP probe 22, 80, 443 → all accept
- SSH banner exchange completes within 10s (catches the hung-box symptom
where TCP accepts but sshd never sends its banner)
- ssh root@72.61.0.17 'docker ps' → 5/5 netbird-* containers Up
- curl https://chat.sysloggh.net/ → 2xx/3xx (proxy route alive)
- DNS dependency drift:
- For each *.sysloggh.net, dig +short → flag any new CNAME → netbird.sysloggh.net
- A service moving FROM LAN-IP TO netbird CNAME is a regression (alert)
- A service moving FROM netbird CNAME TO LAN-IP is an improvement (log)
## Remediations
- NetBird VPS hung (TCP up, SSH banner timeout, TLS hang):
→ This is a host-level hang, not a service fault. Cannot self-remediate via SSH.
→ Alert crit immediately: "NetBird VPS hung — needs hard reboot from provider console"
→ Do NOT declare backend services down; their LAN-IP paths are still up.
- Single proxy route missing (e.g. git returns 000 but chat ok):
→ ssh root@72.61.0.17 'docker restart netbird-traefik'
→ re-check after 30s
- New NetBird CNAME detected:
→ Alert: "<domain> became NetBird-dependent — violates IP-first doctrine (Section 7)"
- Config URL violation detected:
→ Alert with file + line; do not auto-edit configs
Section 6: Alert Routing & Escalation
Severity Levels
Severity
Channel
Format
Rate Limit
warn
Zulip DM to owner
"⚠️ {check}: {detail}"
1x/check/hour
crit
Zulip DM + #agent-hub
"🚨 {check}: {detail}"
Immediate
escalated
Zulip DM + relay to maintainer
"🔥 {check}: {detail}"
Immediate
Escalation Chain
Container unhealthy (3 consecutive failures)
→ Alert owner
→ 15 min no response → Relay to maintainer
Node offline
→ Alert owner
→ 5 min → Attempt restart via PVE API
→ 2 failures → Escalate
Storage > 95%
→ Alert owner + maintainer relay
→ Immediate action required
SSL cert < 7 days
→ Alert owner daily
→ 3 days out → Escalate
ZFS pool degraded
→ Cannot auto-fix → Escalate immediately
Section 7: Configuration Doctrine — IP-First, URLs for Browsers
This doctrine is the structural fix for the NetBird-dependency outage. It is
enforced by the routing-regression.config_url_violations check in Section
5.3.
Rule
Service configs use LAN IP addresses, always. Any file that wires a
service, agent, monitor, or integration to another internal service must
reference the backend by its 192.168.68.x LAN IP and port — never by a
*.sysloggh.net URL. NetBird may blip at any time; LAN IPs do not.
*.sysloggh.net URLs are reserved for human browser access. They are
a convenience layer (split-horizon DNS on the LAN, NetBird reverse proxy
off the LAN) for a person at a keyboard. They must never be a load-bearing
dependency in code or config.
Agents and monitors reach backends by LAN IP. Abiba, Tanko, and Mumuni
all run on the LAN; there is no reason for them to hairpin through NetBird.
The daily infra report and all ssh/curl checks already follow this.
DNS records should prefer LAN IPs over NetBird CNAMEs. A service that
can resolve directly to its LAN IP (like litellm → 192.168.68.116 and
minipve → 192.168.68.12) is NetBird-independent. Moving a record from
a NetBird CNAME to a LAN IP is an improvement; the reverse is a regression.
NetBird is for remote/PC access only. When you are out or on a PC,
NetBird carries you in. When you are on the LAN, NetBird is not in the
path. Configs must reflect this — they always assume LAN.
Canonical IP map (configs MUST use these)
Service
Use in configs
URL (browsers only)
Proxmox API
https://192.168.68.12:8006
https://minipve.sysloggh.net
LiteLLM API
http://192.168.68.116:4000
https://litellm.sysloggh.net
LiteLLM (nginx)
http://192.168.68.116
—
Grafana
http://192.168.68.116:3001
—
Authentik
https://192.168.68.11
https://auth.sysloggh.net
Gitea
http://192.168.68.110:3000
https://git.sysloggh.net
Zulip API
http://192.168.68.117
https://chat.sysloggh.net
SearXNG
http://192.168.68.7:8888
—
Firecrawl
http://192.168.68.7:3002
—
RA-H OS bridge
http://192.168.68.65:3100
—
Violation examples (to be flagged by Section 5.3)
❌ An agent config setting LITELLM_BASE_URL=https://litellm.sysloggh.net
✅ Same config setting LITELLM_BASE_URL=http://192.168.68.116:4000
❌ A monitor curling https://chat.sysloggh.net/api/v1/server_settings
✅ A monitor curling http://192.168.68.117/api/v1/server_settings (and
optionally the URL as a separate browser-path check)
Goal state
Every *.sysloggh.net record resolves to a LAN IP (split-horizon DNS on LAN,
NetBird reverse proxy off LAN). NetBird then becomes purely the off-LAN
ingress — if it blips, only remote browser users notice, and no agent,
monitor, or integration breaks.
Abiba
b0c0331727