Files

T

root 6cc1b7d492 no-mistakes(document): Update 5→6 node references and Mumuni placement across all docs

2026-07-20 17:27:28 +00:00

30 KiB

Raw Blame History

kind, name, description

kind	name	description
pattern	infrastructure-control	Full infrastructure monitoring and control pattern covering the 6-node Proxmox cluster, 3 Docker ecosystems (22 containers), NFS storage, and network services. Defines monitors, remediations, and the access matrix for all environments. FIELD TRUST: Live-state fields (IPs, ports, hostnames, credentials, container names, PIDs) are marked VERIFY-BEFORE-USE. They drift — never mutate infrastructure based on them without first confirming against the live system. Policy fields are authoritative. See the `verify-before-mutate` skill.

Infrastructure Control Pattern

Topology

                    ┌─────────────────────────┐
                    │   OpenProse Contract    │
                    │  (declares what's true) │
                    └──────────┬──────────────┘
                               │
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
   ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
   │   Abiba     │    │   Tanko      │    │   Mumuni     │
   │   (pi)      │    │   (Hermes)   │    │   (Hermes)   │
   │   CT 100    │    │   CT 112     │    │   CT 114     │
   └──────┬──────┘    └──────┬───────┘    └──────┬───────┘
          │                  │                    │
          └──────────────────┼────────────────────┘
                             ▼
          ┌──────────────────────────────────────┐
          │         Proxmox Cluster API          │
          │    minipve.sysloggh.net:443          │
          │   (monitoring@pve!mumuni token)      │
          └────┬──────┬──────┬──────┬──────┬─────┘
               │      │      │      │      │
          ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
          ▼      ▼      ▼      ▼      ▼
       minipve amdpve storepve acerpve ocupve  hwepve
        (.12)   (.15)   (.6)    (.9)    (.5)    (.4)

          ▼
   ┌─────────────────────────────────────────────┐
   │            Docker Ecosystems                │
   │  ┌────────────┬────────────┬──────────────┐ │
   │  │ docker-vm  │ CT 116     │ Netbird      │ │
   │  │ (.7)       │ (.116)     │ (72.61.0.17) │ │
   │  │ 11 ctrs    │ 6 ctrs     │ 5 ctrs       │ │
   │  └────────────┴────────────┴──────────────┘ │
   └─────────────────────────────────────────────┘

Section 1: Access Matrix

API Tokens & Credentials

Resource	Auth Method	Credential Source	Status
Proxmox Cluster	PVE API Token	Infisical vault (`PROXMOX_API_TOKEN`)	✅
Proxmox Root	Password via API ticket	Infisical vault (`PROXMOX_ROOT_PASSWORD`)	✅
docker-vm (.7)	SSH root	SSH key	✅
CT 116 (syslog-api)	SSH root	SSH key	✅
Tanko CT (.122)	SSH jerome	id_ed25519	✅
Mumuni CT (.123)	SSH root	id_ed25519	✅
Baggy CT (113)	SSH jerome	❌ no key access
Netbird (.17)	SSH root	SSH key	✅
Gitea	API token	Infisical vault (`GITEA_BOT_TOKEN`)	✅
Zulip	Bot API key	Infisical vault (`ZULIP_BOT_KEY`)	✅
RA-H OS	MCP bridge	port 3100	✅
LiteLLM Admin	master key	Infisical vault (`LITELLM_MASTER_KEY`)	✅ VERIFY-BEFORE-USE
Grafana	admin password	Infisical vault (`GRAFANA_ADMIN_PASSWORD`)	✅ VERIFY-BEFORE-USE

VERIFY-BEFORE-USE: Credentials, IPs, ports, and hostnames in this contract are live-state fields. Test them against the live system before relying on them (e.g., curl -H "Authorization: Bearer $KEY" .../v1/models for a LiteLLM key; pct config <ct> for a CT IP). Drift is expected — see verify-before-mutate skill. Policy fields (rules, doctrines) are authoritative and do not need verification.

Reachability Matrix

From / To	PVE API	docker-vm (.7)	CT 116	Tanko (.122)	Mumuni (.123)	Baggy (.114)
Abiba (.24)	✅ :443	✅ SSH	✅ SSH	✅ SSH jerome	✅ SSH root	❌ SSH
Tanko (.122)	❌	❌	❌ via NetBird	✅	❌	❌
Mumuni (.123)	❌	❌	❌	❌	✅	❌

Conclusion: Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.

Section 2: Proxmox Cluster — Monitoring

Nodes (6)

Node	IP	CPU	RAM	VMs/CTs	Role
minipve	.12	16C	30GB	authentik, gitea, syslog-api, jitsi	Auth, git, messaging
amdpve	.15	32C	62GB	abiba, kagentz, tanko, tdunna, baggy, scottdenya	Agents, compute
storepve	.6	28C	31GB	docker-vm, ra-h-os, PBS, media, zulip	Docker, storage, chat
acerpve	.9	28C	31GB	llm-gpu, adguard	GPU VMs
ocupve	.5	12C	14GB	ocu-llm	GPU VMs
hwepve	.4	12C	15GB	mumuni	Huawei Matebook 16, agent host

Checks (every 5 min)

## Maintains

- cluster-status: { online_nodes: int, offline_nodes: int, timestamp }
- node-cpu-usage: { node: pct, warnings: [] }
- node-memory-usage: { node: free_pct, warnings: [] }
- node-uptime: { node: seconds, just_rebooted: bool }

## Checks

- For each node in [minipve, amdpve, storepve, acerpve, ocupve, hwepve]:
  - GET /api2/json/nodes/{node}/status → check status == "online"
  - GET /api2/json/nodes/{node}/status → cpu < 0.80
  - GET /api2/json/nodes/{node}/status → free_mem > 10%
  - GET /api2/json/nodes/{node}/status → uptime > 300 (warn if just rebooted)

- For each CT in inventory (19 total):
  - GET /api2/json/cluster/resources → filter by type=lxc
  - Warn if status != "running"

- Storage pools:
  - GET /api2/json/nodes/storepve/storage/mediastore → used < 80%
  - GET /api2/json/nodes/storepve/storage/pbs-backup → last backup < 48h

## Remediations

- Node offline → alert via Zulip DM, escalate after 3x
- CT stopped → `pct start <vmid>` via PVE API, verify after 30s
- VM stopped → `qm start <vmid>` via PVE API, verify after 60s
- Storage > 80% → warn via Zulip DM
- Storage > 95% → crit via Zulip + relay to maintainer

Section 3: Docker Ecosystems — Monitoring

Ecosystem A: docker-vm (192.168.68.7)

16 containers across 4 compose stacks + trove agents:

Stack	Path	Containers
Firecrawl	`/opt/search-stack/firecrawl-source/`	api, rabbitmq, postgres, playwright, redis
SearXNG	`/opt/search-stack/searxng/`	searxng, valkey
Home stack	`/opt/home_stack/`	jdownloader, stirling-pdf, pulse
Audiobookshelf	`/opt/audiobookshelf/`	audiobookshelf
Trove agents	docker run (standalone)	trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats

Last verified: 2026-07-09 — 16/16 containers running healthy.

Home Stack Services

Stirling-PDF (deployed 2026-07-03, Authentik SSO 2026-07-03):

URL: https://pdf.sysloggh.net (public) / http://192.168.68.7:8989 (direct)
Swagger: http://192.168.68.7:8989/swagger-ui.html
Admin credentials: admin / kakashi20stirling
API key: adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88
Authentik OAuth2: configured but disabled (requires paid Server license). Ready to enable: set SECURITY_OAUTH2_ENABLED=true + SECURITY_LOGINMETHOD=all
Compose: /opt/home_stack/docker-compose.yml
Control script: /opt/home_stack/infra-control.sh

JDownloader:

URL: http://192.168.68.7:5800 (web UI via VNC)

Pulse (Uptime Kuma):

URL: http://192.168.68.7:3001

Ecosystem B: CT 116 syslog-api (192.168.68.116)

8 containers in inference-harness stack:

Container	Image	Port	Role
harness-litellm	berriai/litellm:1.90.0-rc.1	:4000→:4001	API proxy, key mgmt, fallbacks
harness-router	inference-harness-router	:9000 (127.0.0.1)	GPU routing, slot booking, CB
harness-nginx	nginx:alpine	:80	Entrypoint, /v1→LiteLLM, /dashboard/
harness-postgres	postgres:16-alpine	:5432	LiteLLM DB (keys, spend, config)
harness-redis	redis:7-alpine	:6379	Router slots, circuit breakers
harness-dashboard	inference-harness-dashboard	:3000	SyslogAI Harness UI
harness-grafana	grafana/grafana	:3000→:3001 (direct LAN, not behind nginx)	GPU + Proxmox + Docker dashboards
harness-prometheus	prom/prometheus	:9090	Metrics scraper, 6 jobs

Nginx routing:

/v1/* → harness-litellm:4000 (API)
/admin/* → harness-litellm:4000
/dashboard/ → harness-dashboard:3000
/litellm/* → harness-litellm:4000
/health/* → harness-litellm:4000/health/liveliness
/gpu/ → 192.168.68.24:9100 (fleet monitor)

Prometheus targets:

192.168.68.8:9400 (RTX 3090 — qwen)
192.168.68.110:9400 (RTX 5070 — gemma)
192.168.68.15:9400 (Strix Halo — ornith)
192.168.68.24:9401 (Router metrics exporter)
harness-litellm:4000 (LiteLLM health)

Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud)

5 containers, single compose stack at /root/docker-compose.yml:

Container	Image	Role
netbird-server	netbirdio/netbird-server:latest	VPN controller, management API
netbird-dashboard	netbirdio/dashboard:latest	Web management UI
netbird-proxy	netbirdio/reverse-proxy:latest	SNI router + TLS passthrough for *.sysloggh.net
netbird-crowdsec	crowdsecurity/crowdsec:v1.7.7	WAF / threat intelligence
netbird-traefik	traefik:v3.6	Edge reverse proxy, Let's Encrypt TLS

Architecture: Internet → Traefik (TLS + HTTP→HTTPS redirect) → Netbird proxy (TLS passthrough via HostSNI, PROXY protocol) → Netbird mesh → LAN backends. Traefik handles the Netbird dashboard directly; all other *.sysloggh.net domains are TLS-passthrough to the proxy on port 8443.

Proxy config: /root/proxy.env — token, ACME certs in /certs/, geolocation DB auto-downloaded. Certs are per-domain Let's Encrypt via TLS-ALPN-01 challenge.

Dependency note: NetBird is an access layer only — it exists so a human on a PC or away from the LAN can reach services by URL. It is not a dependency of any service backend. Per Section 7, every config and agent must reach backends by LAN IP; the *.sysloggh.net URLs are reserved for human browsers.

Known Issues

A. Access log bloat (CRITICAL) — 2026-07-03 discovery. The management server writes an access_log_entries row for every proxied request (~45K/day). SQLite has no built-in TTL. By 2026-07-03 the table reached 10.7M rows / 5.4GB, causing:

DB writes slow → gRPC deadlines exceeded → proxy loses management connection
On proxy restart, management server too slow to push route mappings
Proxy starts HTTPS listener before routes arrive → all domains "unknown"
Fix: batch-delete old entries, VACUUM. Prevention: weekly systemd timer (netbird-cleanup.timer) deletes entries >7 days old when DB exceeds 500MB.
Recurrence signal: proxy logs show unknown domain "git.sysloggh.net" or match: false for known domains → management route sync failed.

B. Stale management state after proxy restart. Management server sometimes stops pushing route mappings to a reconnecting proxy. Symptom: proxy starts, mapping stream established, but no routes arrive → all domains unknown. Fix: restart management server FIRST, THEN proxy. Order matters.

C. Peer sha-OWLpE8nS4Rp4IWzep+y2ayFeZ6C3lJ4eNRB29L5yQTE= flapping. One peer (connecting through Traefik relay at 172.30.0.10) disconnects every ~2-3 minutes. This may be a symptom of DB bloat (healthcheck timeout due to slow DB writes) rather than a network issue. Monitor after bloat fix.

Recovery Procedures

Proxy losing all routes ("unknown domain" for all services):

ssh root@72.61.0.17
cd /root
docker compose restart netbird-server
sleep 12
docker compose restart proxy

Access log bloat detected (DB > 500MB):

ssh root@72.61.0.17
/root/cleanup-access-logs.sh

VPS hung (TCP accepts, SSH banner timeout): hard reboot from Hostinger console.

Checks (every 60s)

## Maintains

- docker-health: { ecosystem: string, healthy: int, unhealthy: int, total: int }
- container-status: { name: string, status: string, restarts: int, image: string }

## Checks

For each Docker host:
  - docker ps --format "{{.Names}}" → every required container is running
  - docker inspect <name> --format "{{.State.Health.Status}}" → "healthy"
  - docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
  - df -h / | awk '{print $5}' → usage < 80%

For Netbird VPS specifically:
  - netbird-store-db-size: du -m /var/lib/docker/volumes/root_netbird_data/_data/store.db → < 500MB
  - netbird-access-log-count: sqlite3 store.db 'SELECT COUNT(*) FROM access_log_entries' → < 500K
  - If DB > 500MB: alert + run /root/cleanup-access-logs.sh
  - proxy-route-check: curl https://git.sysloggh.net/ → 2xx (proves routes loaded)
  - proxy-sync-check: docker logs netbird-proxy --since 60s | grep 'Initial mapping sync complete' → present after restart

For docker-vm specifically:
  - mountpoint -q /media/storage → NFS mounted
  - mountpoint -q /media/mediastore → NFS mounted
  - docker compose ls → expected stacks present

## Remediations

- Container unhealthy → `docker compose up -d <name>`, wait 10s, re-check
- Container missing → `docker compose up -d` in its stack directory
- Docker daemon down → `systemctl restart docker`
- NFS mount lost → `mount -a`, if fails → alert (storepve issue)
- Disk > 80% → alert via Zulip DM
- Disk > 95% → crit + relay to maintainer

Section 4: Storage — Monitoring

NFS Mounts (docker-vm → storepve)

Mount	Export	Capacity	Used	Alert
/media/storage	storepve:/media/storage	3.6TB	264GB (8%)	None
/media/mediastore	storepve:/media/mediastore	7.3TB	5.2TB (75%)	⚠️ warn at 80%

Proxmox Storage Backends

Storage	Type	Content	Active
local	dir	ISO, vztmpl, backup	✅
local-lvm	lvmthin	images, rootdir	✅
zfs-vm	nfs	rootdir, images	✅
mediastore	dir	images, backup, rootdir	✅
zfs-iso	nfs	vztmpl, iso	✅
pbs-backup	pbs	backup	✅

Checks (every 10 min)

## Maintains

- storage-usage: { mount: string, used_pct: float, growth_rate: float }
- backup-status: { last_success: timestamp, age_hours: int }
- snapshot-status: { pool: string, newest_age_hours: int }

## Specific Alerts

- mediastore growth rate > 10GB/day → warn (check what's writing)
- mediastore > 90% → crit (only 730GB remaining)
- No PBS backup in 48h → fail

Section 5: Network Services — Monitoring

5.1 Service Inventory

The Resolves To column is the load-bearing one. Services that CNAME to netbird.sysloggh.net (72.61.0.17) are routed through the NetBird reverse proxy and fail whenever NetBird blips, even though their LAN backend is fine. Services that resolve directly to a LAN IP are NetBird-independent.

Service	Domain	LAN Backend	Resolves To	NetBird-dep	Status
Proxmox API	minipve.sysloggh.net:8006	192.168.68.12	LAN IP	No	✅
LiteLLM	litellm.sysloggh.net	192.168.68.116	LAN IP	No	✅
Authentik	auth.sysloggh.net:443	192.168.68.11	CNAME → netbird	Yes	⚠️
Gitea	git.sysloggh.net:443	192.168.68.110	CNAME → netbird	Yes	⚠️
Zulip	chat.sysloggh.net:443	192.168.68.19	CNAME → netbird	Yes	⚠️ VERIFY-BEFORE-USE
Pulse	pulse.sysloggh.net:443	192.168.68.7	CNAME → netbird	Yes	⚠️
DNS UI	dns.sysloggh.net:443	192.168.68.102	CNAME → netbird	Yes	⚠️
SearXNG	searxng.sysloggh.net:8888	192.168.68.7:8888	LAN IP	No	✅
Firecrawl	firecrawl.sysloggh.net:3002	192.168.68.7:3002	LAN IP	No	✅

Verified 2026-07-02: NetBird VPS rebooted after a hang; all CNAME'd services recovered. LAN-IP-direct paths stayed up throughout the outage.

5.2 Checks (every 2 min)

## Maintains

- dns-resolution: { services: [{name, resolves_to, is_lan_ip}] }
- ssl-expiry: { services: [{name, days_remaining}] }
- endpoint-reachability: { services: [{name, http_code}] }
- netbird-dependency: { services: [{name, depends_on_netbird: bool}] }

5.3 Network Verification & Routing (every 5 min)

This block is what catches a NetBird blip before it becomes an outage. It proves two independent paths for every service: the URL path (what a browser uses, may cross NetBird) and the LAN-IP path (what configs and agents must use, never crosses NetBird).

## Maintains

- dual-path-reachability: {
    services: [{
      name: string,
      url_path: { http_code, tls_ok },          # the *.sysloggh.net path
      lan_path:  { ip, port, http_code },        # the IP:port path
      lan_independent: bool                       # lan_path works when url_path fails
    }]
  }
- netbird-ingress-health: {
    vps_up: bool,              # TCP 22/80/443 accept on 72.61.0.17
    ssh_banner_ok: bool,       # banner exchange completes (catches hung box)
    containers_up: int,        # all 5 netbird-* containers running
    proxy_routes_ok: int       # traefik routers resolving (chat/git/auth return 2xx/3xx)
  }
- routing-regression: {
    new_netbird_cnames: [domain],   # domains that newly CNAME to netbird.sysloggh.net
                                    # → alert: a service just became NetBird-dependent
    config_url_violations: [file]   # configs/agents found referencing a *.sysloggh.net
                                    # URL where a LAN IP is required (see Section 7)
  }

## Checks

- For each service in 5.1:
  - curl https://<domain>/ → record url_path (code, tls_ok)
  - curl http://<lan_ip>:<port>/ → record lan_path
  - If url_path fails AND lan_path succeeds → netbird-degraded, NOT service-down
  - If BOTH fail → service-down (real outage)

- NetBird ingress (72.61.0.17):
  - TCP probe 22, 80, 443 → all accept
  - SSH banner exchange completes within 10s (catches the hung-box symptom
    where TCP accepts but sshd never sends its banner)
  - ssh root@72.61.0.17 'docker ps' → 5/5 netbird-* containers Up
  - curl https://chat.sysloggh.net/ → 2xx/3xx (proxy route alive)

- DNS dependency drift:
  - For each *.sysloggh.net, dig +short → flag any new CNAME → netbird.sysloggh.net
  - A service moving FROM LAN-IP TO netbird CNAME is a regression (alert)
  - A service moving FROM netbird CNAME TO LAN-IP is an improvement (log)

## Remediations

- NetBird VPS hung (TCP up, SSH banner timeout, TLS hang):
  → This is a host-level hang, not a service fault. Cannot self-remediate via SSH.
  → Alert crit immediately: "NetBird VPS hung — needs hard reboot from provider console"
  → Do NOT declare backend services down; their LAN-IP paths are still up.
- Single proxy route missing (e.g. git returns 000 but chat ok):
  → ssh root@72.61.0.17 'docker restart netbird-traefik'
  → re-check after 30s
- New NetBird CNAME detected:
  → Alert: "<domain> became NetBird-dependent — violates IP-first doctrine (Section 7)"
- Config URL violation detected:
  → Alert with file + line; do not auto-edit configs

Section 6: Alert Routing & Escalation

Severity Levels

Severity	Channel	Format	Rate Limit
warn	Zulip DM to owner	"⚠️ {check}: {detail}"	1x/check/hour
crit	Zulip DM + #agent-hub	"🚨 {check}: {detail}"	Immediate
escalated	Zulip DM + relay to maintainer	"🔥 {check}: {detail}"	Immediate

Escalation Chain

Container unhealthy (3 consecutive failures)
  → Alert owner
  → 15 min no response → Relay to maintainer

Node offline
  → Alert owner
  → 5 min → Attempt restart via PVE API
  → 2 failures → Escalate

Storage > 95%
  → Alert owner + maintainer relay
  → Immediate action required

SSL cert < 7 days
  → Alert owner daily
  → 3 days out → Escalate

ZFS pool degraded
  → Cannot auto-fix → Escalate immediately

Section 7: Configuration Doctrine — IP-First, URLs for Browsers

This doctrine is the structural fix for the NetBird-dependency outage. It is enforced by the routing-regression.config_url_violations check in Section 5.3.

Rule

Service configs use LAN IP addresses, always. Any file that wires a service, agent, monitor, or integration to another internal service must reference the backend by its 192.168.68.x LAN IP and port — never by a *.sysloggh.net URL. NetBird may blip at any time; LAN IPs do not.
*.sysloggh.net URLs are reserved for human browser access. They are a convenience layer (split-horizon DNS on the LAN, NetBird reverse proxy off the LAN) for a person at a keyboard. They must never be a load-bearing dependency in code or config.
Agents and monitors reach backends by LAN IP. Abiba, Tanko, and Mumuni all run on the LAN; there is no reason for them to hairpin through NetBird. The daily infra report and all ssh/curl checks already follow this.
DNS records should prefer LAN IPs over NetBird CNAMEs. A service that can resolve directly to its LAN IP (like litellm → 192.168.68.116 and minipve → 192.168.68.12) is NetBird-independent. Moving a record from a NetBird CNAME to a LAN IP is an improvement; the reverse is a regression.
NetBird is for remote/PC access only. When you are out or on a PC, NetBird carries you in. When you are on the LAN, NetBird is not in the path. Configs must reflect this — they always assume LAN.

Canonical IP map (configs MUST use these)

Service	Use in configs	URL (browsers only)
Proxmox API	`https://192.168.68.12:8006`	`https://minipve.sysloggh.net`
LiteLLM API	`http://192.168.68.116:4000`	`https://litellm.sysloggh.net`
LiteLLM (nginx)	`http://192.168.68.116`	—
Grafana	`http://192.168.68.116:3001`	—
Authentik	`https://192.168.68.11`	`https://auth.sysloggh.net`
Gitea	`http://192.168.68.110:3000`	`https://git.sysloggh.net`
Zulip API	`http://192.168.68.19`	`https://chat.sysloggh.net`
SearXNG	`http://192.168.68.7:8888`	—
Firecrawl	`http://192.168.68.7:3002`	—
RA-H OS bridge	`http://192.168.68.65:3100`	—

Violation examples (to be flagged by Section 5.3)

❌ An agent config setting LITELLM_BASE_URL=https://litellm.sysloggh.net
✅ Same config setting LITELLM_BASE_URL=http://192.168.68.116:4000
❌ A monitor curling https://chat.sysloggh.net/api/v1/server_settings
✅ A monitor curling http://192.168.68.19/api/v1/server_settings (and optionally the URL as a separate browser-path check)

Goal state

Every *.sysloggh.net record resolves to a LAN IP (split-horizon DNS on LAN, NetBird reverse proxy off LAN). NetBird then becomes purely the off-LAN ingress — if it blips, only remote browser users notice, and no agent, monitor, or integration breaks.

Appendix A: Quick Health Commands

# Full cluster status
PVE="https://minipve.sysloggh.net"
AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"

# Docker health from Abiba
ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'"
ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'"

# GPU fleet quick check
curl -s http://192.168.68.116/health/unified | jq .status
curl -s http://192.168.68.24:9100/gpu-data | jq .summary

# Grafana status
curl -s http://admin:$(infisical secrets get GRAFANA_ADMIN_PASSWORD --project=infrastructure --env=production --plain)@192.168.68.116:3001/api/health

# Prometheus targets
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

# LiteLLM key check
curl -s -H "Authorization: Bearer $(infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production --plain)" \
  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'

# Storage check
ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"

# Router roster reload (if needed)
curl -s -X POST http://192.168.68.116:9000/admin/roster/reload \
  -H "Authorization: Bearer sk-admin-ee09fffd04978b61a1569ac670c68814"

# Restart stuck GPU (saturation watchdog alternative)
ssh root@192.168.68.8 "systemctl restart llama-server"
ssh root@192.168.68.110 "systemctl restart llama-server"

Appendix B: CT Inventory

CT	Name	Node	IP	Role	Agent
100	abiba	amdpve	.24	Pi agent (this host)	✅ pi
101	llm-gpu	acerpve	.8	GPU RTX 3090	❌
102	adguard	acerpve	—	DNS	❌
103	ocu-llm	ocupve	.110	GPU RTX 5070	❌
104	authentik	minipve	.11	OIDC	❌
105	kagentz	amdpve	—	Agent Zero	✅
106	ra-h-os	storepve	.65	KG bridge	✅ MCP
107	pbs	storepve	—	Backups	❌
108	media	storepve	—	Media	❌
109	docker-vm	storepve	.7	Docker host	❌
110	gitea	minipve	—	Git	❌
111	tdunna	amdpve	.129	Hermes agent	✅
112	tanko	amdpve	.122	Hermes agent	✅
113	baggy	amdpve	?	Hermes agent	✅
114	mumuni	hwepve	.123	Hermes agent	✅
115	scottdenya	amdpve	—	?	❌
116	syslog-api	minipve	.116	LiteLLM + Grafana	❌
117	zulip	storepve	—	Chat	❌
118	jitsi	minipve	—	Video	❌

Appendix C: Docker Compose Files Location

Host	Stack	Compose File
docker-vm (.7)	Firecrawl	`/opt/search-stack/firecrawl-source/docker-compose.yaml`
docker-vm (.7)	SearXNG	`/opt/search-stack/searxng/docker-compose.yml`
docker-vm (.7)	Home stack	`/opt/home_stack/docker-compose.yml`
docker-vm (.7)	Audiobookshelf	`/opt/audiobookshelf/docker-compose.yml`
CT 116 (.116)	Inference Harness	`/opt/inference-harness/docker-compose.yml`
Netbird (.17)	Netbird	Docker run (not compose)

CT Access (pct-run — no IPs needed)

All CTs are accessible via pct-run <CT_ID> <command>. No IP addresses required. Source of truth: /root/scripts/pct-run.sh or prose-contracts/scripts/pct-run.sh.

CT	Name	Node	pct-run
100	abiba	amdpve	`pct-run 100`
105	kagentz	amdpve	`pct-run 105`
111	tdunna	amdpve	`pct-run 111`
112	tanko	amdpve	`pct-run 112`
113	baggy	amdpve	`pct-run 113`
115	scottdenya	amdpve	`pct-run 115`
104	authentik	minipve	`pct-run 104`
110	gitea	minipve	`pct-run 110`
114	mumuni	hwepve	`pct-run 114`
116	syslog-api	minipve	`pct-run 116`
106	ra-h-os	storepve	`pct-run 106`
107	proxmox-backup	storepve	`pct-run 107`
108	media	storepve	`pct-run 108`
117	zulip	storepve	`pct-run 117`
102	adguard	acerpve	`pct-run 102`

GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:

ssh root@192.168.68.8   # RTX 3090
ssh root@192.168.68.110 # RTX 5070
ssh root@192.168.68.15  # Strix Halo

Section 7: Agent Health Check (consolidated — 2026-07-05)

Replaces 7 scattered Zulip health scripts with a single non-disruptive check running every 10 minutes via cron (/root/scripts/agent-health-check.py).

The script is read-only — it never restarts, kills, or modifies anything. Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh, which was kill+nohup outside systemd) are banned by policy.

Checks Performed

Check	Frequency	What It Detects
LiteLLM key validation	10 min	All 4 agent keys authenticate and return models
GPU port conflict	10 min	Ghost processes squatting port 8080 (ss vs systemd MainPID)
Gateway liveness	10 min	Gateway process running, state file readable
Zulip streaming	10 min	`edit_message` present in adapter (streaming supported)
Recent errors	10 min	Error count in journald for last 10 min

Disabled Scripts

Script	Why Disabled
`zulip-watchdog.sh` (Mumuni)	kill+nohup bypassed systemd, 27 restarts, pattern mismatch
`zulip-monitor.sh` (Abiba)	Replaced by agent-health-check.py + PM2 auto-restart

30 KiB Raw Blame History