no-mistakes(review): sync tanko CT 112 mapping to minipve across consumers
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 12s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 17s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 12s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 17s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s
This commit is contained in:
@@ -643,7 +643,7 @@ contracts:
|
|||||||
timeout: 120
|
timeout: 120
|
||||||
requires:
|
requires:
|
||||||
- Zulip API key for abiba-bot@chat.sysloggh.net
|
- Zulip API key for abiba-bot@chat.sysloggh.net
|
||||||
- SSH access to amdpve (192.168.68.15) for Tanko (CT 112) and the Agent Zero Docker host (.14)
|
- SSH access to minipve (192.168.68.12) for Tanko (CT 112) and the Agent Zero Docker host (.14)
|
||||||
verification:
|
verification:
|
||||||
postconditions:
|
postconditions:
|
||||||
- check: bot registration active
|
- check: bot registration active
|
||||||
|
|||||||
@@ -414,7 +414,7 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|
|||||||
| 108 | media | storepve | lxc | ✅ reachable |
|
| 108 | media | storepve | lxc | ✅ reachable |
|
||||||
| 110 | gitea | minipve | lxc | ✅ reachable |
|
| 110 | gitea | minipve | lxc | ✅ reachable |
|
||||||
| 111 | tdunna | **storepve** | lxc | ⛔ **REPORT-ONLY** (192.168.68.129, Theo's box — no GC at any level) |
|
| 111 | tdunna | **storepve** | lxc | ⛔ **REPORT-ONLY** (192.168.68.129, Theo's box — no GC at any level) |
|
||||||
| 112 | tanko | amdpve | lxc | ✅ reachable |
|
| 112 | tanko | minipve | lxc | ✅ reachable |
|
||||||
| 113 | baggy | amdpve | lxc | ✅ reachable |
|
| 113 | baggy | amdpve | lxc | ✅ reachable |
|
||||||
| 115 | scottdenya | amdpve | lxc | ✅ reachable |
|
| 115 | scottdenya | amdpve | lxc | ✅ reachable |
|
||||||
| 116 | syslog-api | minipve | lxc | ✅ reachable |
|
| 116 | syslog-api | minipve | lxc | ✅ reachable |
|
||||||
|
|||||||
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.
|
|||||||
|
|
||||||
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
||||||
|------|-----|---------|-------------|-------------|------|
|
|------|-----|---------|-------------|-------------|------|
|
||||||
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
|
| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
|
||||||
| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
|
| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
|
||||||
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
|
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
|
||||||
|
|
||||||
@@ -72,7 +72,7 @@ connectivity recovery including end-to-end DM validation.
|
|||||||
### Step 1: Resolve Target
|
### Step 1: Resolve Target
|
||||||
|
|
||||||
Map `target` to host, CT ID, hermes_home, and user from the live-state table.
|
Map `target` to host, CT ID, hermes_home, and user from the live-state table.
|
||||||
For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
|
For CT112 route through `ssh root@minipve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
|
||||||
|
|
||||||
### Step 2: Pull Latest Plugin Source
|
### Step 2: Pull Latest Plugin Source
|
||||||
|
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ gateway restart, and connection validation.
|
|||||||
### Step 1: Locate Target
|
### Step 1: Locate Target
|
||||||
|
|
||||||
Map `target` to connectivity parameters from the live-state table above.
|
Map `target` to connectivity parameters from the live-state table above.
|
||||||
For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
|
For CT112 route through `ssh root@minipve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
|
||||||
|
|
||||||
### Step 2: Deploy Zulip Adapter
|
### Step 2: Deploy Zulip Adapter
|
||||||
|
|
||||||
|
|||||||
@@ -105,8 +105,8 @@ description: >
|
|||||||
|
|
||||||
| Node | IP | CPU | RAM | VMs/CTs | Role |
|
| Node | IP | CPU | RAM | VMs/CTs | Role |
|
||||||
|------|----|-----|-----|---------|------|
|
|------|----|-----|-----|---------|------|
|
||||||
| minipve | .12 | 16C | 30GB | abiba, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
|
| minipve | .12 | 16C | 30GB | abiba, tanko, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
|
||||||
| amdpve | .15 | 32C | 62GB | kagentz, tanko, baggy, scottdenya, adguard2 | Agents, compute |
|
| amdpve | .15 | 32C | 62GB | kagentz, baggy, scottdenya, adguard2 | Agents, compute |
|
||||||
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat |
|
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat |
|
||||||
| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
|
| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
|
||||||
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
|
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
|
||||||
@@ -682,7 +682,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
|
|||||||
| 109 | docker-vm | storepve | .7 | Docker host | ❌ |
|
| 109 | docker-vm | storepve | .7 | Docker host | ❌ |
|
||||||
| 110 | gitea | minipve | **.17** | Git | ❌ |
|
| 110 | gitea | minipve | **.17** | Git | ❌ |
|
||||||
| 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
|
| 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
|
||||||
| 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ |
|
| 112 | tanko | minipve | .122 | DSH (DeepSeek Harness) agent | ✅ |
|
||||||
| 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
|
| 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
|
||||||
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
|
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
|
||||||
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
||||||
@@ -712,7 +712,7 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
|
|||||||
| 100 | abiba | minipve | `pct-run 100` |
|
| 100 | abiba | minipve | `pct-run 100` |
|
||||||
| 105 | kagentz | amdpve | `pct-run 105` |
|
| 105 | kagentz | amdpve | `pct-run 105` |
|
||||||
| 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) |
|
| 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) |
|
||||||
| 112 | tanko | amdpve | `pct-run 112` |
|
| 112 | tanko | minipve | `pct-run 112` |
|
||||||
| 113 | baggy | amdpve | `pct-run 113` |
|
| 113 | baggy | amdpve | `pct-run 113` |
|
||||||
| 115 | scottdenya | amdpve | `pct-run 115` |
|
| 115 | scottdenya | amdpve | `pct-run 115` |
|
||||||
| 104 | authentik | minipve | `pct-run 104` |
|
| 104 | authentik | minipve | `pct-run 104` |
|
||||||
|
|||||||
@@ -59,7 +59,7 @@ Before ANY update wave:
|
|||||||
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||||
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
|
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
|
||||||
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
|
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
|
||||||
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
|
| CT 112 (tanko, minipve) | Tanko | `apt update && apt upgrade -y` | 3 min |
|
||||||
| CT 105 (kagentz, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
|
| CT 105 (kagentz, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
|
||||||
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
|
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
|
||||||
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
|
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
|
||||||
|
|||||||
@@ -101,8 +101,6 @@ GUESTS: list[Guest] = [
|
|||||||
# amdpve (192.168.68.15)
|
# amdpve (192.168.68.15)
|
||||||
Guest(ct_id="105", hostname="kagentz", ip="192.168.68.105", node="amdpve",
|
Guest(ct_id="105", hostname="kagentz", ip="192.168.68.105", node="amdpve",
|
||||||
access_method="ssh-host", probe_target="kagentz (CT 105, amdpve)"),
|
access_method="ssh-host", probe_target="kagentz (CT 105, amdpve)"),
|
||||||
Guest(ct_id="112", hostname="tanko", ip="192.168.68.112", node="amdpve",
|
|
||||||
access_method="pct-run", probe_target="tanko (CT 112, amdpve)"),
|
|
||||||
Guest(ct_id="113", hostname="baggy", ip="192.168.68.113", node="amdpve",
|
Guest(ct_id="113", hostname="baggy", ip="192.168.68.113", node="amdpve",
|
||||||
access_method="pct-run", probe_target="baggy (CT 113, amdpve)"),
|
access_method="pct-run", probe_target="baggy (CT 113, amdpve)"),
|
||||||
Guest(ct_id="115", hostname="scottdenya", ip="192.168.68.115", node="amdpve",
|
Guest(ct_id="115", hostname="scottdenya", ip="192.168.68.115", node="amdpve",
|
||||||
@@ -110,6 +108,8 @@ GUESTS: list[Guest] = [
|
|||||||
Guest(ct_id="120", hostname="adguard2", ip="192.168.68.120", node="amdpve",
|
Guest(ct_id="120", hostname="adguard2", ip="192.168.68.120", node="amdpve",
|
||||||
access_method="pct-run", probe_target="adguard2 (CT 120, amdpve)"),
|
access_method="pct-run", probe_target="adguard2 (CT 120, amdpve)"),
|
||||||
# minipve (192.168.68.12)
|
# minipve (192.168.68.12)
|
||||||
|
Guest(ct_id="112", hostname="tanko", ip="192.168.68.112", node="minipve",
|
||||||
|
access_method="pct-run", probe_target="tanko (CT 112, minipve)"),
|
||||||
Guest(ct_id="100", hostname="abiba", ip="192.168.68.100", node="minipve",
|
Guest(ct_id="100", hostname="abiba", ip="192.168.68.100", node="minipve",
|
||||||
access_method="pct-run", probe_target="abiba (CT 100, minipve)"),
|
access_method="pct-run", probe_target="abiba (CT 100, minipve)"),
|
||||||
Guest(ct_id="102", hostname="adguard", ip="192.168.68.102", node="minipve",
|
Guest(ct_id="102", hostname="adguard", ip="192.168.68.102", node="minipve",
|
||||||
|
|||||||
+1
-1
@@ -12,7 +12,6 @@ set -euo pipefail
|
|||||||
declare -A CT_NODES=(
|
declare -A CT_NODES=(
|
||||||
# amdpve (192.168.68.15)
|
# amdpve (192.168.68.15)
|
||||||
[105]=amdpve # kagentz (was hwepve — corrected 2026-09-12; live per pvesh)
|
[105]=amdpve # kagentz (was hwepve — corrected 2026-09-12; live per pvesh)
|
||||||
[112]=amdpve # tanko
|
|
||||||
[113]=amdpve # baggy
|
[113]=amdpve # baggy
|
||||||
[115]=amdpve # scottdenya
|
[115]=amdpve # scottdenya
|
||||||
[120]=amdpve # adguard2 (added 2026-09-12)
|
[120]=amdpve # adguard2 (added 2026-09-12)
|
||||||
@@ -21,6 +20,7 @@ declare -A CT_NODES=(
|
|||||||
[102]=minipve # adguard (was acerpve)
|
[102]=minipve # adguard (was acerpve)
|
||||||
[104]=minipve # authentik
|
[104]=minipve # authentik
|
||||||
[110]=minipve # gitea
|
[110]=minipve # gitea
|
||||||
|
[112]=minipve # tanko (was amdpve — migrated 2026-09-27; live per pvesh)
|
||||||
[116]=minipve # syslog-api
|
[116]=minipve # syslog-api
|
||||||
[119]=minipve # infisical-vault
|
[119]=minipve # infisical-vault
|
||||||
# storepve (192.168.68.6)
|
# storepve (192.168.68.6)
|
||||||
|
|||||||
@@ -47,8 +47,8 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol
|
|||||||
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
|
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
|
||||||
|
|
||||||
**Proxmox Cluster "Tabiri" (5 nodes):**
|
**Proxmox Cluster "Tabiri" (5 nodes):**
|
||||||
- amdpve (192.168.68.15): kagentz, tanko, baggy, scottdenya, adguard2
|
- amdpve (192.168.68.15): kagentz, baggy, scottdenya, adguard2
|
||||||
- minipve (192.168.68.12): abiba, adguard, authentik, gitea, syslog-api, infisical-vault
|
- minipve (192.168.68.12): abiba, tanko, adguard, authentik, gitea, syslog-api, infisical-vault
|
||||||
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna
|
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna
|
||||||
- acerpve (192.168.68.9): llm-gpu
|
- acerpve (192.168.68.9): llm-gpu
|
||||||
- ocupve (192.168.68.5): ocu-llm
|
- ocupve (192.168.68.5): ocu-llm
|
||||||
|
|||||||
@@ -135,16 +135,16 @@ case "$PI_VERDICT" in
|
|||||||
esac
|
esac
|
||||||
# -- abiba-leg-end
|
# -- abiba-leg-end
|
||||||
|
|
||||||
# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ──
|
# ── Platform B: Tanko (DSH dsh-web on minipve CT 112) ──
|
||||||
# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
|
# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
|
||||||
# key availability varies — so probes run from the amdpve vantage via `pct exec`.
|
# key availability varies — so probes run from the minipve vantage via `pct exec`.
|
||||||
# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on amdpve
|
# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on minipve
|
||||||
# (192.168.68.15). The gateway binds 127.0.0.1:3080 loopback-only by design — a
|
# (192.168.68.12). The gateway binds 127.0.0.1:3080 loopback-only by design — a
|
||||||
# remote :3080 probe is refused and is NOT a fault.
|
# remote :3080 probe is refused and is NOT a fault.
|
||||||
TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
|
TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.12 \
|
||||||
"pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true)
|
"pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true)
|
||||||
[ -n "$TANKO_SVC" ] || TANKO_SVC="unknown"
|
[ -n "$TANKO_SVC" ] || TANKO_SVC="unknown"
|
||||||
TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
|
TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.12 \
|
||||||
"pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true)
|
"pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true)
|
||||||
[ -n "$TANKO_HTTP" ] || TANKO_HTTP="000"
|
[ -n "$TANKO_HTTP" ] || TANKO_HTTP="000"
|
||||||
|
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ HEALTH_CONTRACT = ROOT / "zulip-health.prose.md"
|
|||||||
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
|
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
|
||||||
|
|
||||||
MUMUNI_IP = "192.168.68.24" # Mumuni's old (decommissioned) deployment
|
MUMUNI_IP = "192.168.68.24" # Mumuni's old (decommissioned) deployment
|
||||||
TANKO_VANTAGE = "192.168.68.15" # amdpve — Tanko CT 112 via pct exec
|
TANKO_VANTAGE = "192.168.68.12" # minipve — Tanko CT 112 via pct exec
|
||||||
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
|
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
|
||||||
|
|
||||||
|
|
||||||
@@ -82,7 +82,7 @@ done
|
|||||||
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
|
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
|
||||||
cmd="${*: -1}"
|
cmd="${*: -1}"
|
||||||
case "$host" in
|
case "$host" in
|
||||||
192.168.68.15)
|
192.168.68.12)
|
||||||
case "$cmd" in
|
case "$cmd" in
|
||||||
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
|
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
|
||||||
*curl*) printf '%s' "$TANKO_HTTP" ;;
|
*curl*) printf '%s' "$TANKO_HTTP" ;;
|
||||||
|
|||||||
@@ -104,6 +104,26 @@ def test_koby_ct111_is_on_storepve(ahc):
|
|||||||
assert ahc.AGENTS["koby"]["pve"] == "storepve"
|
assert ahc.AGENTS["koby"]["pve"] == "storepve"
|
||||||
|
|
||||||
|
|
||||||
|
def test_tanko_ct112_is_probed_on_minipve(ahc, monkeypatch, capsys):
|
||||||
|
# CT 112 (tanko) was live-migrated to minipve (.12) on 2026-09-27; the
|
||||||
|
# amdpve mapping made `pct status 112` fail and read as ct-unreachable.
|
||||||
|
# Execute the probe and assert the host the script actually contacts.
|
||||||
|
probes = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
ahc, "ssh",
|
||||||
|
lambda host, cmd, user="root": probes.append((host, cmd)) or "status: running",
|
||||||
|
)
|
||||||
|
ahc.FAIL.clear()
|
||||||
|
ahc.REPORT_ONLY.clear()
|
||||||
|
try:
|
||||||
|
ahc.check_ct_liveness()
|
||||||
|
tanko_hosts = [h for h, cmd in probes if cmd == "pct status 112 2>/dev/null"]
|
||||||
|
assert tanko_hosts == ["192.168.68.12"]
|
||||||
|
finally:
|
||||||
|
ahc.FAIL.clear()
|
||||||
|
ahc.REPORT_ONLY.clear()
|
||||||
|
|
||||||
|
|
||||||
def test_report_only_legs_never_count_as_failures(ahc):
|
def test_report_only_legs_never_count_as_failures(ahc):
|
||||||
for agent, report_only in (("koby", True), ("koonimo", False), ("tanko", False)):
|
for agent, report_only in (("koby", True), ("koonimo", False), ("tanko", False)):
|
||||||
ahc.FAIL.clear()
|
ahc.FAIL.clear()
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ ROOT = pathlib.Path(__file__).resolve().parents[1]
|
|||||||
ZULIP_MONITOR = ROOT / "scripts" / "zulip-monitor.sh"
|
ZULIP_MONITOR = ROOT / "scripts" / "zulip-monitor.sh"
|
||||||
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
|
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
|
||||||
|
|
||||||
TANKO_VANTAGE = "192.168.68.15" # amdpve — Tanko CT 112 via pct exec
|
TANKO_VANTAGE = "192.168.68.12" # minipve — Tanko CT 112 via pct exec
|
||||||
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
|
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
|
||||||
|
|
||||||
|
|
||||||
@@ -52,7 +52,7 @@ done
|
|||||||
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
|
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
|
||||||
cmd="${*: -1}"
|
cmd="${*: -1}"
|
||||||
case "$host" in
|
case "$host" in
|
||||||
192.168.68.15)
|
192.168.68.12)
|
||||||
case "$cmd" in
|
case "$cmd" in
|
||||||
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
|
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
|
||||||
*curl*) printf '%s' "$TANKO_HTTP" ;;
|
*curl*) printf '%s' "$TANKO_HTTP" ;;
|
||||||
|
|||||||
+24
-24
@@ -28,7 +28,7 @@ session start.
|
|||||||
## Requires
|
## Requires
|
||||||
|
|
||||||
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
||||||
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
|
- **SSH access** to minipve (192.168.68.12) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
|
||||||
- **PM2** on localhost for pi process management
|
- **PM2** on localhost for pi process management
|
||||||
- **Network access** to `chat.sysloggh.net`, `kagentz.sysloggh.net` (C3 public path), `localhost:9200`
|
- **Network access** to `chat.sysloggh.net`, `kagentz.sysloggh.net` (C3 public path), `localhost:9200`
|
||||||
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
||||||
@@ -228,20 +228,20 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
|
|||||||
| Crash loop >10/h | Alert user |
|
| Crash loop >10/h | Alert user |
|
||||||
|
|
||||||
|
|
||||||
### Step 3: Platform B — Tanko (DSH on amdpve CT 112)
|
### Step 3: Platform B — Tanko (DSH on minipve CT 112)
|
||||||
|
|
||||||
Mumuni is out of scope for this host (see the note above): she runs on her own
|
Mumuni is out of scope for this host (see the note above): she runs on her own
|
||||||
container and is monitored on her side.
|
container and is monitored on her side.
|
||||||
|
|
||||||
Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
|
Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
|
||||||
there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
|
there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
|
||||||
as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve**
|
as the `dsh-web` systemd unit inside **CT 112**, which resides on the **minipve**
|
||||||
PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is not a dependency
|
PVE host (**192.168.68.12**). Direct SSH to 192.168.68.122 is not a dependency
|
||||||
of this contract — per-worker key availability varies — so CT 112 probes run
|
of this contract — per-worker key availability varies — so CT 112 probes run
|
||||||
from the amdpve vantage via `pct exec`:
|
from the minipve vantage via `pct exec`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- <command>"
|
ssh root@192.168.68.12 "pct exec 112 -- <command>"
|
||||||
```
|
```
|
||||||
|
|
||||||
> **By design (verified 2026-09-08):** the `dsh-web` gateway binds
|
> **By design (verified 2026-09-08):** the `dsh-web` gateway binds
|
||||||
@@ -253,7 +253,7 @@ ssh root@192.168.68.15 "pct exec 112 -- <command>"
|
|||||||
**B1: Gateway Service State (Tanko)**
|
**B1: Gateway Service State (Tanko)**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web"
|
ssh root@192.168.68.12 "pct exec 112 -- systemctl is-active dsh-web"
|
||||||
```
|
```
|
||||||
|
|
||||||
Expected: `active`. Anything else → gateway service down → apply the Tanko heal
|
Expected: `active`. Anything else → gateway service down → apply the Tanko heal
|
||||||
@@ -262,7 +262,7 @@ Expected: `active`. Anything else → gateway service down → apply the Tanko h
|
|||||||
**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**
|
**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
|
ssh root@192.168.68.12 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
|
||||||
```
|
```
|
||||||
|
|
||||||
Alive = **ANY** HTTP status response from the endpoint — the expected set is
|
Alive = **ANY** HTTP status response from the endpoint — the expected set is
|
||||||
@@ -273,13 +273,13 @@ process answering `503` is running and self-heal must NOT restart-loop it.
|
|||||||
Down = connection refused (`000`) or timeout only. Statuses outside the
|
Down = connection refused (`000`) or timeout only. Statuses outside the
|
||||||
expected set are logged/reported as a warning — reported, never healed on.
|
expected set are logged/reported as a warning — reported, never healed on.
|
||||||
|
|
||||||
**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)**
|
**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to minipve)**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
|
curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
|
||||||
```
|
```
|
||||||
|
|
||||||
Fallback only — used when the monitoring node has no pct/SSH path to amdpve.
|
Fallback only — used when the monitoring node has no pct/SSH path to minipve.
|
||||||
Alive = **ANY** HTTP status response from the endpoint — healthy signals are
|
Alive = **ANY** HTTP status response from the endpoint — healthy signals are
|
||||||
`302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other
|
`302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other
|
||||||
status, including `404`/`5xx`, also counts alive: the endpoint is up and
|
status, including `404`/`5xx`, also counts alive: the endpoint is up and
|
||||||
@@ -404,29 +404,29 @@ ExecStartPost=/bin/systemctl --no-block start dsh-web-token.service
|
|||||||
4. Every later request through `/` presents that cookie; the token is not needed
|
4. Every later request through `/` presents that cookie; the token is not needed
|
||||||
again until the cookie expires or a new browser is used.
|
again until the cookie expires or a new browser is used.
|
||||||
|
|
||||||
**Verification** (amdpve vantage):
|
**Verification** (minipve vantage):
|
||||||
```bash
|
```bash
|
||||||
# 1. Login endpoint is Authentik-gated: unauthenticated -> 302 (not 200/303).
|
# 1. Login endpoint is Authentik-gated: unauthenticated -> 302 (not 200/303).
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}\n' \
|
ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}\n' \
|
||||||
-H 'Host: tankodhs.sysloggh.net' http://127.0.0.1/dsh-web-login"
|
-H 'Host: tankodhs.sysloggh.net' http://127.0.0.1/dsh-web-login"
|
||||||
# Expected: 302
|
# Expected: 302
|
||||||
|
|
||||||
# 2. Legacy :8081 endpoint is gone (connection refused -> 000).
|
# 2. Legacy :8081 endpoint is gone (connection refused -> 000).
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \
|
ssh root@192.168.68.12 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \
|
||||||
-w '%{http_code}\n' http://192.168.68.122:8081/"
|
-w '%{http_code}\n' http://192.168.68.122:8081/"
|
||||||
# Expected: 000
|
# Expected: 000
|
||||||
|
|
||||||
# 3. Backend cookie mint + reuse (exactly what /dsh-web-login proxies to).
|
# 3. Backend cookie mint + reuse (exactly what /dsh-web-login proxies to).
|
||||||
TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token")
|
TOKEN=$(ssh root@192.168.68.12 "pct exec 112 -- cat /etc/dsh-web/launch-token")
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \
|
ssh root@192.168.68.12 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \
|
||||||
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
|
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
|
ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
|
||||||
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
|
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
|
||||||
# Expected: 200 — the minted dsh-auth-... cookie (authority
|
# Expected: 200 — the minted dsh-auth-... cookie (authority
|
||||||
# tankodhs.sysloggh.net) is replayed on the next request and accepted.
|
# tankodhs.sysloggh.net) is replayed on the next request and accepted.
|
||||||
|
|
||||||
# 4. Token refresh is non-disruptive and idempotent.
|
# 4. Token refresh is non-disruptive and idempotent.
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh"
|
ssh root@192.168.68.12 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh"
|
||||||
# Expected: "token unchanged; nginx not reloaded" when nothing changed
|
# Expected: "token unchanged; nginx not reloaded" when nothing changed
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -437,32 +437,32 @@ fresh cookie. Both verified live 2026-09-11.
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 5. Cookie survives a dsh-web restart, and the new token mints a new cookie.
|
# 5. Cookie survives a dsh-web restart, and the new token mints a new cookie.
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- systemctl restart dsh-web"
|
ssh root@192.168.68.12 "pct exec 112 -- systemctl restart dsh-web"
|
||||||
# dsh-web is Type=simple: restart returns before :3080 is listening. Bounded-poll
|
# dsh-web is Type=simple: restart returns before :3080 is listening. Bounded-poll
|
||||||
# until the socket answers (any status but 000) before asserting the cookie.
|
# until the socket answers (any status but 000) before asserting the cookie.
|
||||||
for i in $(seq 1 60); do
|
for i in $(seq 1 60); do
|
||||||
UP=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
|
UP=$(ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
|
||||||
-H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/")
|
-H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/")
|
||||||
[ "$UP" != "000" ] && break
|
[ "$UP" != "000" ] && break
|
||||||
sleep 2
|
sleep 2
|
||||||
done
|
done
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
|
ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
|
||||||
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
|
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
|
||||||
# Expected: 200 — the pre-restart cookie is still accepted.
|
# Expected: 200 — the pre-restart cookie is still accepted.
|
||||||
# The restart's ExecStartPost (or the 2-minute timer) refreshes the include. A
|
# The restart's ExecStartPost (or the 2-minute timer) refreshes the include. A
|
||||||
# manual run may no-op on the flock, so poll until the include carries a token
|
# manual run may no-op on the flock, so poll until the include carries a token
|
||||||
# the running process accepts (bounded wait) before the mint+reuse check.
|
# the running process accepts (bounded wait) before the mint+reuse check.
|
||||||
for i in $(seq 1 60); do
|
for i in $(seq 1 60); do
|
||||||
TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'")
|
TOKEN=$(ssh root@192.168.68.12 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'")
|
||||||
CODE=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
|
CODE=$(ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
|
||||||
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'")
|
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'")
|
||||||
[ "$CODE" = "303" ] && break
|
[ "$CODE" = "303" ] && break
|
||||||
sleep 2
|
sleep 2
|
||||||
done
|
done
|
||||||
# Expected: 303 — the include now holds the token the running process accepts.
|
# Expected: 303 — the include now holds the token the running process accepts.
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh-new.jar -o /dev/null \
|
ssh root@192.168.68.12 "pct exec 112 -- curl -s -c /tmp/dsh-new.jar -o /dev/null \
|
||||||
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
|
-H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
|
||||||
ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null \
|
ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null \
|
||||||
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
|
-w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
|
||||||
# Expected: 200 — the refreshed token minted a fresh cookie.
|
# Expected: 200 — the refreshed token minted a fresh cookie.
|
||||||
```
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user