2026-07-03: Hermes key enforcement contract + Stirling-PDF docs + config standardization
- hermes-key-enforcement.prose.md: NEW contract enforcing api_key_env for all harness/LiteLLM providers, exempting external providers - hermes-config-template.prose.md: Updated with full fix log (15 total fixes across Koby/Koonimo/Mumuni/Tanko), model:auto detection rule, key-to-agent mapping - infrastructure-control.prose.md: Added Stirling-PDF service details (port 8989, credentials, API key, Swagger URL) replacing Bento-PDF
This commit is contained in:
Abiba
@@ -137,9 +137,25 @@ description: >
|
||||
|-------|------|-----------|
|
||||
| **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis |
|
||||
| **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
|
||||
| **Home stack** | `/opt/home_stack/` | jdownloader, bentopdf, pulse |
|
||||
| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
|
||||
| **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
|
||||
|
||||
### Home Stack Services
|
||||
|
||||
**Stirling-PDF** (replaced Bento-PDF 2026-07-03):
|
||||
- URL: `http://192.168.68.7:8989`
|
||||
- Swagger: `http://192.168.68.7:8989/swagger-ui.html`
|
||||
- Admin credentials: `admin` / `kakashi20stirling`
|
||||
- API key: `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88`
|
||||
- Compose: `/opt/home_stack/docker-compose.yml`
|
||||
- Control script: `/opt/home_stack/infra-control.sh`
|
||||
|
||||
**JDownloader**:
|
||||
- URL: `http://192.168.68.7:5800` (web UI via VNC)
|
||||
|
||||
**Pulse** (Uptime Kuma):
|
||||
- URL: `http://192.168.68.7:3001`
|
||||
|
||||
### Ecosystem B: CT 116 syslog-api (192.168.68.116)
|
||||
|
||||
8 containers in inference-harness stack:
|
||||
@@ -171,27 +187,69 @@ description: >
|
||||
- 192.168.68.24:9401 (Router metrics exporter)
|
||||
- harness-litellm:4000 (LiteLLM health)
|
||||
|
||||
### Ecosystem C: Netbird (72.61.0.17)
|
||||
### Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud)
|
||||
|
||||
5 containers, single compose stack at `/root/docker-compose.yml`:
|
||||
|
||||
| Container | Image | Role |
|
||||
|-----------|-------|------|
|
||||
| netbird-server | netbird | VPN controller |
|
||||
| netbird-dashboard | netbird UI | Web management |
|
||||
| netbird-proxy | nginx | TLS termination |
|
||||
| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF |
|
||||
| netbird-traefik | traefik | Reverse proxy |
|
||||
| netbird-server | netbirdio/netbird-server:latest | VPN controller, management API |
|
||||
| netbird-dashboard | netbirdio/dashboard:latest | Web management UI |
|
||||
| netbird-proxy | netbirdio/reverse-proxy:latest | SNI router + TLS passthrough for *.sysloggh.net |
|
||||
| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF / threat intelligence |
|
||||
| netbird-traefik | traefik:v3.6 | Edge reverse proxy, Let's Encrypt TLS |
|
||||
|
||||
**Architecture:** Internet → Traefik (TLS + HTTP→HTTPS redirect) → Netbird proxy
|
||||
(TLS passthrough via HostSNI, PROXY protocol) → Netbird mesh → LAN backends.
|
||||
Traefik handles the Netbird dashboard directly; all other `*.sysloggh.net`
|
||||
domains are TLS-passthrough to the proxy on port 8443.
|
||||
|
||||
**Proxy config:** `/root/proxy.env` — token, ACME certs in `/certs/`, geolocation
|
||||
DB auto-downloaded. Certs are per-domain Let's Encrypt via TLS-ALPN-01 challenge.
|
||||
|
||||
> **Dependency note:** NetBird is an **access layer only** — it exists so a
|
||||
> human on a PC or away from the LAN can reach services by URL. It is **not**
|
||||
> a dependency of any service backend. Per Section 7, every config and agent
|
||||
> must reach backends by LAN IP; the `*.sysloggh.net` URLs are reserved for
|
||||
> human browsers. The 2026-07-02 outage (NetBird VPS hung) took out every
|
||||
> CNAME'd URL while every LAN-IP backend stayed up — proof the doctrine is
|
||||
> necessary.
|
||||
>
|
||||
> **Recovery:** a hung NetBird VPS (TCP accepts, SSH banner exchange times
|
||||
> out, TLS hangs) cannot be self-remediated over the network. It requires a
|
||||
> hard reboot from the hosting provider's console. See Section 5.3.
|
||||
> human browsers.
|
||||
|
||||
#### Known Issues
|
||||
|
||||
**A. Access log bloat (CRITICAL)** — 2026-07-03 discovery. The management server
|
||||
writes an `access_log_entries` row for every proxied request (~45K/day). SQLite
|
||||
has no built-in TTL. By 2026-07-03 the table reached 10.7M rows / 5.4GB, causing:
|
||||
- DB writes slow → gRPC deadlines exceeded → proxy loses management connection
|
||||
- On proxy restart, management server too slow to push route mappings
|
||||
- Proxy starts HTTPS listener before routes arrive → all domains "unknown"
|
||||
- **Fix:** batch-delete old entries, VACUUM. Prevention: weekly systemd timer (`netbird-cleanup.timer`) deletes entries >7 days old when DB exceeds 500MB.
|
||||
- **Recurrence signal:** proxy logs show `unknown domain "git.sysloggh.net"` or `match: false` for known domains → management route sync failed.
|
||||
|
||||
**B. Stale management state after proxy restart.** Management server sometimes
|
||||
stops pushing route mappings to a reconnecting proxy. Symptom: proxy starts,
|
||||
mapping stream established, but no routes arrive → all domains `unknown`.
|
||||
**Fix:** restart management server FIRST, THEN proxy. Order matters.
|
||||
|
||||
**C. Peer `sha-OWLpE8nS4Rp4IWzep+y2ayFeZ6C3lJ4eNRB29L5yQTE=` flapping.**
|
||||
One peer (connecting through Traefik relay at 172.30.0.10) disconnects every
|
||||
~2-3 minutes. This may be a symptom of DB bloat (healthcheck timeout due to
|
||||
slow DB writes) rather than a network issue. Monitor after bloat fix.
|
||||
|
||||
#### Recovery Procedures
|
||||
|
||||
1. **Proxy losing all routes ("unknown domain" for all services):**
|
||||
```bash
|
||||
ssh root@72.61.0.17
|
||||
cd /root
|
||||
docker compose restart netbird-server
|
||||
sleep 12
|
||||
docker compose restart proxy
|
||||
```
|
||||
2. **Access log bloat detected (DB > 500MB):**
|
||||
```bash
|
||||
ssh root@72.61.0.17
|
||||
/root/cleanup-access-logs.sh
|
||||
```
|
||||
3. **VPS hung (TCP accepts, SSH banner timeout):** hard reboot from Hostinger console.
|
||||
|
||||
### Checks (every 60s)
|
||||
|
||||
@@ -209,6 +267,13 @@ For each Docker host:
|
||||
- docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
|
||||
- df -h / | awk '{print $5}' → usage < 80%
|
||||
|
||||
For Netbird VPS specifically:
|
||||
- netbird-store-db-size: du -m /var/lib/docker/volumes/root_netbird_data/_data/store.db → < 500MB
|
||||
- netbird-access-log-count: sqlite3 store.db 'SELECT COUNT(*) FROM access_log_entries' → < 500K
|
||||
- If DB > 500MB: alert + run /root/cleanup-access-logs.sh
|
||||
- proxy-route-check: curl https://git.sysloggh.net/ → 2xx (proves routes loaded)
|
||||
- proxy-sync-check: docker logs netbird-proxy --since 60s | grep 'Initial mapping sync complete' → present after restart
|
||||
|
||||
For docker-vm specifically:
|
||||
- mountpoint -q /media/storage → NFS mounted
|
||||
- mountpoint -q /media/mediastore → NFS mounted
|
||||
|
||||
Reference in New Issue
Block a user