fix: contract accuracy updates post fleet-wide reboot
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s

infrastructure-control.prose.md:
- Add hwepve as 6th Proxmox node
- Fix Gitea IP: .17 (was .110)
- Fix AdGuard IP: .10 on minipve (was .102 on acerpve)
- Fix Abiba placement: hwepve (was amdpve)
- Fix Mumuni placement: hwepve (was minipve)
- Fix Authentik port: add :9000
- Add CT 118 (jdownloader), CT 119 (infisical-vault)
- Add last-verified date (2026-07-24)

infrastructure-update.prose.md:
- Add post-reboot CT sweep procedure
- Add Zulip Docker network recovery steps
- Add WireGuard tunnel verification

scripts/netbird-add-domain.sh:
- New script to register domains in Netbird proxy store.db
This commit is contained in:
root
2026-07-24 20:01:43 +00:00
parent 2f0c3c1850
commit b17c60f997
3 changed files with 145 additions and 26 deletions
+39 -2
View File
@@ -91,17 +91,54 @@ Before ANY update wave:
- Firecrawl test: `curl :3002/`
- SearXNG test: `curl :8888`
## Wave 4: Proxmox Kernel Reboot (if needed)
## Wave 4: Proxmox Kernel Reboot
Only if `[ -f /var/run/reboot-required ]` on any node.
| Target | Action |
|--------|--------|
| Affected PVE node | Verify all CTs/VMs migrated or stopped |
| | `reboot` via PVE API |
| | `reboot` via PVE API (or `systemctl reboot -f` if dbus fails) |
| | Wait 120s for node to come back |
| | Start any stopped CTs |
### Post-reboot sweep (known gaps)
After every node reboot, run these checks:
1. **CT auto-start sweep** — LXC containers sometimes don't start despite
`onboot: 1`. Check every CT on the rebooted node and start any left stopped:
```bash
pct list | awk '/stopped/{print $1}' | xargs -I{} pct start {}
```
Known cases: scottdenya (CT 115 on amdpve), authentik (CT 104 on minipve).
2. **Zulip recovery** — When docker-vm or storepve reboots, the Zulip main
container loses its Docker network assignment (SIGKILL during storage
outage detaches it from `zulip_default` network). Run:
```bash
ssh root@192.168.68.19
docker rm -f zulip-zulip-1
cd /opt/zulip && docker compose up -d
```
The compose restart recreates the container on the correct network.
3. **docker-vm Docker daemon** — After reboot, Docker can take 3-4 minutes
to become `active`. The docker-proxy for Pulse (port 7655) starts early,
so Pulse is accessible before `docker ps` reports ready. Wait for Docker
before checking other stacks.
### VPS ↔ docker-vm tunnel
After any VPS or docker-vm reboot, verify the dedicated WireGuard tunnel:
```bash
ssh root@72.61.0.17 'wg show wg1' | grep "latest handshake"
# If no handshake in >60s:
ssh root@72.61.0.17 'wg-quick up wg1'
```
The tunnel uses PersistentKeepalive=25 and is systemd-enabled, but should
be verified after a reboot.
## Rollback Protocol
If ANY verification fails: