feat: infrastructure-control network-verification + IP-first doctrine, plus 3 supporting contracts

- infrastructure-control: add Section 5.3 (Network Verification & Routing) with dual-path checks, NetBird ingress health, DNS drift detection, and routing regression alerts. Add Section 7 (Configuration Doctrine — IP-First) mandating LAN IPs for all configs/agents, URLs reserved for human browser access only. Also enrich access matrix (Mumuni, Koonimo, LiteLLM Admin, Grafana entries), reachability matrix, and CT 116 nginx routing + Prometheus targets. - build-zulip-plugin: fold Success Criteria into Maintains postconditions. - pi-approval-architecture: new reference doc — pi approval model vs Hermes, architectural limits, /approve and /deny stub commands. - zulip-approval-fix: new responsibility — documents the Zulip HTML/prefix bug that broke /approve and /deny, and the one-line fix applied.
2026-07-02 20:39:40 +00:00
parent c1d993fe01
commit b0c0331727
4 changed files with 433 additions and 61 deletions
@@ -60,19 +60,23 @@ description: >
 | Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ |
 | docker-vm (.7) | SSH root | SSH key | ✅ |
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
-| Tanko CT (.122) | SSH jerome | SSH key | ✅ |
+| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
+| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
+| Koonimo CT (.114) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
 | Gitea | API token | abiba-bot token | ✅ |
-| Zulip | Bot API key | per-bot tokens | ✅ |
+| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
 | RA-H OS | MCP bridge | port 3100 | ✅ |
+| LiteLLM Admin | master key | `sk-litellm-7f96...` | ✅ |
+| Grafana | admin password | `syslog-grafana-2026` | ✅ |

 ### Reachability Matrix

-| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Netbird (.17) |
-|-----------|---------|----------------|--------|-------------|---------------|
-| **Abiba** (CT 100) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH | ❌ no Netbird |
-| **Tanko** (CT 122) | ❌ | ❌ | ❌ | ✅ | ❌ |
-| **docker-vm** (.7) | ❌ | ✅ | ❌ | ❌ | ❌ |
+| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Koonimo (.114) |
+|-----------|---------|----------------|--------|-------------|---------------|----------------|
+| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
+| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
+| **Mumuni** (.123) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |

 **Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.

@@ -138,14 +142,34 @@ description: >

 ### Ecosystem B: CT 116 syslog-api (192.168.68.116)

-| Container | Image | Port |
-|-----------|-------|------|
-| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4001 |
-| harness-nginx | nginx:alpine | :80 |
-| harness-router | inference-harness-router | :9000 |
-| harness-postgres | postgres:16-alpine | :5432 |
-| harness-redis | redis:7-alpine | :6379 |
-| harness-dashboard | inference-harness-dashboard | :3000 |
+8 containers in inference-harness stack:
+
+| Container | Image | Port | Role |
+|-----------|-------|------|------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | API proxy, key mgmt, fallbacks |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | GPU routing, slot booking, CB |
+| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/, /grafana/ |
+| harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) |
+| harness-redis | redis:7-alpine | :6379 | Router slots, circuit breakers |
+| harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI |
+| harness-grafana | grafana/grafana | :3000→:3001 | GPU dashboards via Prometheus |
+| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 5 targets |
+
+**Nginx routing**:
+- `/v1/*` → harness-litellm:4000 (API)
+- `/admin/*` → harness-litellm:4000
+- `/dashboard/` → harness-dashboard:3000
+- `/litellm/*` → harness-litellm:4000
+- `/grafana/` → harness-grafana:3000
+- `/health/*` → harness-litellm:4000/health/liveliness
+- `/gpu/` → 192.168.68.24:9100 (fleet monitor)
+
+**Prometheus targets**:
+- 192.168.68.8:9400 (RTX 3090 — qwen)
+- 192.168.68.110:9400 (RTX 5070 — gemma)
+- 192.168.68.15:9400 (Strix Halo — ornith)
+- 192.168.68.24:9401 (Router metrics exporter)
+- harness-litellm:4000 (LiteLLM health)

 ### Ecosystem C: Netbird (72.61.0.17)

@@ -157,6 +181,18 @@ description: >
 | netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF |
 | netbird-traefik | traefik | Reverse proxy |

+> **Dependency note:** NetBird is an **access layer only** — it exists so a
+> human on a PC or away from the LAN can reach services by URL. It is **not**
+> a dependency of any service backend. Per Section 7, every config and agent
+> must reach backends by LAN IP; the `*.sysloggh.net` URLs are reserved for
+> human browsers. The 2026-07-02 outage (NetBird VPS hung) took out every
+> CNAME'd URL while every LAN-IP backend stayed up — proof the doctrine is
+> necessary.
+>
+> **Recovery:** a hung NetBird VPS (TCP accepts, SSH banner exchange times
+> out, TLS hangs) cannot be self-remediated over the network. It requires a
+> hard reboot from the hosting provider's console. See Section 5.3.
+
 ### Checks (every 60s)

 ```
@@ -226,25 +262,103 @@ For docker-vm specifically:

 ## Section 5: Network Services — Monitoring

-| Service | Domain | Status |
-|---------|--------|--------|
-| Proxmox API | minipve.sysloggh.net:443 | ✅ |
-| Authentik | auth.sysloggh.net:443 | ✅ |
-| Gitea | git.sysloggh.net:443 | ✅ |
-| Zulip | chat.sysloggh.net:443 | ✅ |
-| LiteLLM | litellm.sysloggh.net:443 | ✅ |
-| Pulse | pulse.sysloggh.net:443 | ✅ |
-| SearXNG | searxng.sysloggh.net:8888 | ✅ |
-| Firecrawl | firecrawl.sysloggh.net:3002 | ✅ |
+### 5.1 Service Inventory

-### Checks (every 2 min)
+The `Resolves To` column is the load-bearing one. Services that CNAME to
+`netbird.sysloggh.net` (72.61.0.17) are routed through the NetBird reverse
+proxy and **fail whenever NetBird blips**, even though their LAN backend is
+fine. Services that resolve directly to a LAN IP are NetBird-independent.
+
+| Service | Domain | LAN Backend | Resolves To | NetBird-dep | Status |
+|---------|--------|-------------|-------------|-------------|--------|
+| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
+| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
+| Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
+| Gitea | git.sysloggh.net:443 | 192.168.68.110 | CNAME → netbird | **Yes** | ⚠️ |
+| Zulip | chat.sysloggh.net:443 | 192.168.68.117 | CNAME → netbird | **Yes** | ⚠️ |
+| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
+| DNS UI | dns.sysloggh.net:443 | 192.168.68.102 | CNAME → netbird | **Yes** | ⚠️ |
+| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
+| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
+
+**Verified 2026-07-02:** NetBird VPS rebooted after a hang; all CNAME'd
+services recovered. LAN-IP-direct paths stayed up throughout the outage.
+
+### 5.2 Checks (every 2 min)

 ```
 ## Maintains

- dns-resolution: { services: [{name, resolves}] }
+- dns-resolution: { services: [{name, resolves_to, is_lan_ip}] }
 - ssl-expiry: { services: [{name, days_remaining}] }
 - endpoint-reachability: { services: [{name, http_code}] }
+- netbird-dependency: { services: [{name, depends_on_netbird: bool}] }
+```
+
+### 5.3 Network Verification & Routing (every 5 min)
+
+This block is what catches a NetBird blip before it becomes an outage. It
+proves two independent paths for every service: the **URL path** (what a
+browser uses, may cross NetBird) and the **LAN-IP path** (what configs and
+agents must use, never crosses NetBird).
+
+```
+## Maintains
+
+- dual-path-reachability: {
+    services: [{
+      name: string,
+      url_path: { http_code, tls_ok },          # the *.sysloggh.net path
+      lan_path:  { ip, port, http_code },        # the IP:port path
+      lan_independent: bool                       # lan_path works when url_path fails
+    }]
+  }
+- netbird-ingress-health: {
+    vps_up: bool,              # TCP 22/80/443 accept on 72.61.0.17
+    ssh_banner_ok: bool,       # banner exchange completes (catches hung box)
+    containers_up: int,        # all 5 netbird-* containers running
+    proxy_routes_ok: int       # traefik routers resolving (chat/git/auth return 2xx/3xx)
+  }
+- routing-regression: {
+    new_netbird_cnames: [domain],   # domains that newly CNAME to netbird.sysloggh.net
+                                    # → alert: a service just became NetBird-dependent
+    config_url_violations: [file]   # configs/agents found referencing a *.sysloggh.net
+                                    # URL where a LAN IP is required (see Section 7)
+  }
+
+## Checks
+
+- For each service in 5.1:
+  - curl https://<domain>/ → record url_path (code, tls_ok)
+  - curl http://<lan_ip>:<port>/ → record lan_path
+  - If url_path fails AND lan_path succeeds → netbird-degraded, NOT service-down
+  - If BOTH fail → service-down (real outage)
+
+- NetBird ingress (72.61.0.17):
+  - TCP probe 22, 80, 443 → all accept
+  - SSH banner exchange completes within 10s (catches the hung-box symptom
+    where TCP accepts but sshd never sends its banner)
+  - ssh root@72.61.0.17 'docker ps' → 5/5 netbird-* containers Up
+  - curl https://chat.sysloggh.net/ → 2xx/3xx (proxy route alive)
+
+- DNS dependency drift:
+  - For each *.sysloggh.net, dig +short → flag any new CNAME → netbird.sysloggh.net
+  - A service moving FROM LAN-IP TO netbird CNAME is a regression (alert)
+  - A service moving FROM netbird CNAME TO LAN-IP is an improvement (log)
+
+## Remediations
+
+- NetBird VPS hung (TCP up, SSH banner timeout, TLS hang):
+  → This is a host-level hang, not a service fault. Cannot self-remediate via SSH.
+  → Alert crit immediately: "NetBird VPS hung — needs hard reboot from provider console"
+  → Do NOT declare backend services down; their LAN-IP paths are still up.
+- Single proxy route missing (e.g. git returns 000 but chat ok):
+  → ssh root@72.61.0.17 'docker restart netbird-traefik'
+  → re-check after 30s
+- New NetBird CNAME detected:
+  → Alert: "<domain> became NetBird-dependent — violates IP-first doctrine (Section 7)"
+- Config URL violation detected:
+  → Alert with file + line; do not auto-edit configs
 ```

 ## Section 6: Alert Routing & Escalation
@@ -281,6 +395,63 @@ ZFS pool degraded
  → Cannot auto-fix → Escalate immediately
 ```

+## Section 7: Configuration Doctrine — IP-First, URLs for Browsers
+
+This doctrine is the structural fix for the NetBird-dependency outage. It is
+enforced by the `routing-regression.config_url_violations` check in Section
+5.3.
+
+### Rule
+
+1. **Service configs use LAN IP addresses, always.** Any file that wires a
+   service, agent, monitor, or integration to another internal service must
+   reference the backend by its `192.168.68.x` LAN IP and port — never by a
+   `*.sysloggh.net` URL. NetBird may blip at any time; LAN IPs do not.
+2. **`*.sysloggh.net` URLs are reserved for human browser access.** They are
+   a convenience layer (split-horizon DNS on the LAN, NetBird reverse proxy
+   off the LAN) for a person at a keyboard. They must never be a load-bearing
+   dependency in code or config.
+3. **Agents and monitors reach backends by LAN IP.** Abiba, Tanko, and Mumuni
+   all run on the LAN; there is no reason for them to hairpin through NetBird.
+   The daily infra report and all `ssh`/`curl` checks already follow this.
+4. **DNS records should prefer LAN IPs over NetBird CNAMEs.** A service that
+   can resolve directly to its LAN IP (like `litellm` → `192.168.68.116` and
+   `minipve` → `192.168.68.12`) is NetBird-independent. Moving a record from
+   a NetBird CNAME to a LAN IP is an improvement; the reverse is a regression.
+5. **NetBird is for remote/PC access only.** When you are out or on a PC,
+   NetBird carries you in. When you are on the LAN, NetBird is not in the
+   path. Configs must reflect this — they always assume LAN.
+
+### Canonical IP map (configs MUST use these)
+
+| Service | Use in configs | URL (browsers only) |
+|---------|----------------|--------------------|
+| Proxmox API | `https://192.168.68.12:8006` | `https://minipve.sysloggh.net` |
+| LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
+| LiteLLM (nginx) | `http://192.168.68.116` | — |
+| Grafana | `http://192.168.68.116:3001` | — |
+| Authentik | `https://192.168.68.11` | `https://auth.sysloggh.net` |
+| Gitea | `http://192.168.68.110:3000` | `https://git.sysloggh.net` |
+| Zulip API | `http://192.168.68.117` | `https://chat.sysloggh.net` |
+| SearXNG | `http://192.168.68.7:8888` | — |
+| Firecrawl | `http://192.168.68.7:3002` | — |
+| RA-H OS bridge | `http://192.168.68.65:3100` | — |
+
+### Violation examples (to be flagged by Section 5.3)
+
+- ❌ An agent config setting `LITELLM_BASE_URL=https://litellm.sysloggh.net`
+- ✅ Same config setting `LITELLM_BASE_URL=http://192.168.68.116:4000`
+- ❌ A monitor curling `https://chat.sysloggh.net/api/v1/server_settings`
+- ✅ A monitor curling `http://192.168.68.117/api/v1/server_settings` (and
+      optionally the URL as a separate browser-path check)
+
+### Goal state
+
+Every `*.sysloggh.net` record resolves to a LAN IP (split-horizon DNS on LAN,
+NetBird reverse proxy off LAN). NetBird then becomes purely the off-LAN
+ingress — if it blips, only remote browser users notice, and no agent,
+monitor, or integration breaks.
+
 ## Appendix A: Quick Health Commands

 ```bash
@@ -293,8 +464,30 @@ curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"
 ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'"
 ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'"

+# GPU fleet quick check
+curl -s http://192.168.68.116/health/unified | jq .status
+curl -s http://192.168.68.24:9100/gpu-data | jq .summary
+
+# Grafana status
+curl -s http://admin:syslog-grafana-2026@192.168.68.116:3001/api/health
+
+# Prometheus targets
+curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
+
+# LiteLLM key check
+curl -s -H "Authorization: Bearer sk-litellm-7f96080dd99b15c36bd4b333b58a6796" \
+  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'
+
 # Storage check
 ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
+
+# Router roster reload (if needed)
+curl -s -X POST http://192.168.68.116:9000/admin/roster/reload \
+  -H "Authorization: Bearer sk-admin-ee09fffd04978b61a1569ac670c68814"
+
+# Restart stuck GPU (saturation watchdog alternative)
+ssh root@192.168.68.8 "systemctl restart llama-server"
+ssh root@192.168.68.110 "systemctl restart llama-server"
 ```

 ## Appendix B: CT Inventory
@@ -302,9 +495,9 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | CT | Name | Node | IP | Role | Agent |
 |----|------|------|----|------|-------|
 | 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi |
-| 101 | llm-gpu | acerpve | — | GPU VM | ❌ |
+| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
 | 102 | adguard | acerpve | — | DNS | ❌ |
-| 103 | ocu-llm | ocupve | — | GPU VLM | ❌ |
+| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
 | 104 | authentik | minipve | .11 | OIDC | ❌ |
 | 105 | kagentz | amdpve | — | Agent Zero | ✅ |
 | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
@@ -314,10 +507,10 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | 110 | gitea | minipve | — | Git | ❌ |
 | 111 | tdunna | amdpve | — | ? | ❌ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
-| 113 | baggy | amdpve | — | ? | ❌ |
-| 114 | mumuni | minipve | — | Hermes agent | ✅ |
+| 113 | baggy/koonimo | amdpve | .114 | Hermes agent | ✅ |
+| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | — | ? | ❌ |
-| 116 | syslog-api | minipve | .116 | LiteLLM stack | ❌ |
+| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
 | 117 | zulip | storepve | — | Chat | ❌ |
 | 118 | jitsi | minipve | — | Video | ❌ |

@@ -329,5 +522,5 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | docker-vm (.7) | SearXNG | `/opt/search-stack/searxng/docker-compose.yml` |
 | docker-vm (.7) | Home stack | `/opt/home_stack/docker-compose.yml` |
 | docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
-| CT 116 (.116) | LiteLLM | `/root/docker-compose-litellm.yml` |
+| CT 116 (.116) | Inference Harness | `/opt/inference-harness/docker-compose.yml` |
 | Netbird (.17) | Netbird | Docker run (not compose) |