fix: add hostname resolution warning and fix dmsetup field documentation

Add warning that bare PVE hostnames (acerpve, amdpve, etc.) resolve to VPS
via *.dns.sysloggh.net wildcard, not to actual nodes. List IP addresses:
- acerpve 192.168.68.9
- amdpve 192.168.68.15
- storepve 192.168.68.6
- minipve 192.168.68.12
- ocupve 192.168.68.5

Update acerpve example in backup preflight to include address (192.168.68.9).

Fix dmsetup comment to show full field order:
=start =length =thin-pool =transaction-id
=metadata_used/metadata_total =data_used/data_total
remaining fields are flags

Make it clear lvs command is the primary source for percentages, dmsetup is only for error-state check.

Incidents now include addresses: acerpve (192.168.68.9) and amdpve (192.168.68.15).

infrastructure-control.prose.md

@@ -364,6 +364,79 @@ For docker-vm specifically:
 - No PBS backup in 48h → fail
 ```
 
+### Backup Safety Preconditions (2026-09-15)
+
+#### Background & Rationale
+
+Two incidents from 2026-09-13/14 demonstrate that backup operations can catastrophically fail when storage conditions are not verified first:
+
+1. **acerpve thin-pool VM 101** (acerpve, 192.168.68.9, 2026-09-13): A snapshot-mode vzdump of VM 101 on acerpve filled the LVM thin pool. `dmsetup status pve-data-tpool` showed `thin-pool Error` (then `Fail`), the host root remounted `emergency_ro`, ordinary commands failed with I/O errors, LVM tools returned nothing and VM 101 (the RTX 3090 host) went unreachable while the host still answered ping and ssh. It happened TWICE in one day with different modes: snapshot at 13:39Z and a `--mode stop` cold run at 18:52Z. Both times a reboot rolled the failed transaction back and the pool returned rw (~30% data, ~1.2% metadata). Pool capacity was NOT the obvious explanation - ~816G with ~572G free - which is why the metadata/snapshot-pressure hypothesis stands unproven. A full or errored thin pool fails EVERY volume on the VG at once, including the host root.
+
+2. **amdpve 0700 tmpdir** (amdpve, 192.168.68.15, 2026-09-14): A custom vzdump `tmpdir` created with mode 0700 broke a whole night of container backups: `fstat "<dir>/vzdumptmp<n>_<ct>//." failed - EACCES`, because the archive step runs through an unprivileged user namespace and could not traverse a root-owned 0700 directory. Fixed with `chmod 1777` (match /var/tmp) and proved with a real backup.
+
+3. **acerpve GPU-host fact**: VM 101 (llm-gpu) and VM 103 (ocu-llm) are in NO scheduled job, so their only coverage is one-off runs - and for VM 101 that is deliberate until the thin-pool is understood.
+
+> ⚠️ **Hostname Resolution Warning (2026-09-15)**: The PVE node hostnames (acerpve, amdpve, minipve, storepve, ocupve) all resolve to the VPS (72.61.0.17, the Netbird VPS at srv1079750.hstgr.cloud) via the wildcard `*.dns.sysloggh.net` record, NOT to the actual nodes. So `ssh acerpve` lands on the VPS. **Nodes must be addressed by IP**: acerpve 192.168.68.9, amdpve 192.168.68.15, storepve 192.168.68.6, minipve 192.168.68.12, ocupve 192.168.68.5. Guest CTs are reached through their node (`pct exec`). Guest hostnames that resolve on the LAN (e.g. kagentz = 192.168.68.14) are fine. (The DNS address records are a separate decision — row: dag-daemon-node-hostnames-resolve-to-the-vps-20260915.)
+
+#### PREFLIGHT Preconditions (Before ANY snapshot-mode backup on thin-pool hosts)
+
+Before starting ANY snapshot-mode vzdump on a host whose storage is an LVM thin pool, the following checks MUST pass:
+
+```bash
+# Check 1: Pool headroom (PRIMARY - yields percentages directly)
+# Run on the NODE (e.g. ssh root@192.168.68.9 for acerpve) — NOT by bare hostname, see warning above
+lvs -o lv_name,data_percent,metadata_percent,lv_size pve/data
+# Example output (acerpve, 192.168.68.9):
+#   LV   Data%  Meta%  LSize
+#   data 29.95  1.22   <816.21g
+# Required thresholds (documented minimum):
+# data_percent < 90%  (80% recommended for safety margin)
+# metadata_percent < 70%  (metadata fills faster than data)
+
+# Check 2: Verify pool is not in error state (dmsetup shows the raw DM device)
+# Run on the NODE (e.g. ssh root@192.168.68.9 for acerpve) — NOT by bare hostname
+dmsetup status pve-data-tpool | grep -q "Error\|Fail" && exit 1
+# dmsetup status pve-data-tpool field order (verified on 192.168.68.9):
+#   $1=start  $2=length  $3="thin-pool"  $4=transaction-id
+#   $5=metadata_used/metadata_total (blocks)  $6=data_used/data_total (sectors)
+#   remaining fields are flags ("-", "rw", "discard_passdown", "queue_if_no_space", ...)
+# This is only used for the ERROR-STATE check; use the lvs command above for percentages.
+#   metadata_percent = $5 / ($5 split by /) [second number in pair]
+```
+
+**Minimum thresholds**: If either `data_percent >= 90%` or `metadata_percent >= 70%`, the backup MUST NOT start. State explicitly that these are hard stops, not warnings.
+
+**Why this is a precondition**: A full or errored thin pool fails EVERY volume on the VG at once, including the host root. This is not a soft failure - it takes down the entire Proxmox host.
+
+#### Staging Directory Requirement (2026-09-14 incident)
+
+Any custom vzdump `tmpdir` MUST be world-traversable and writable exactly like `/var/tmp` (mode 1777). The archive step of vzdump runs in an unprivileged user namespace and cannot traverse a root-owned 0700 directory.
+
+**Symptom to recognize**: `fstat "<dir>/vzdumptmp<n>_<ct>//." failed - EACCES` on every container in the backup run.
+
+**Fix**: `chmod 1777 <custom-tmpdir>` before starting vzdump.
+
+#### Task Start Rule for Truncating Shells
+
+When starting a backup task from a shell that may truncate output (e.g., pipes, `head`), always use:
+
+```bash
+pvesh create /storage/backup --output-format json -- ... | head -2
+# ❌ Can kill the backup task ("broken pipe" status)
+```
+
+Instead, capture JSON output without piping to truncating commands:
+
+```bash
+# Use --output-format json and capture to variable
+result=$(pvesh create /storage/backup --output-format json -- ...)
+# Then parse result if needed
+```
+
+#### GPU Host Backup Status (acerpve VM 101)
+
+VM 101 (llm-gpu) and VM 103 (ocu-llm) have NO scheduled backup job. Coverage is manual one-off runs only. This is intentional for VM 101 until the thin-pool failure mechanism is understood and documented.
+
 ## Section 5: Network Services — Monitoring
 
 ### 5.1 Service Inventory

litellm-api-keys.prose.md

@@ -322,10 +322,10 @@ directly call OpenRouter via Python's requests library. Converting would require
 
 - Master key: **Retrieval path (do not trust a literal value in this file — the key rotates)**:
   ```bash
-  # Read at runtime from the container's environment:
+  # PRIMARY (proven, runs on CT 116 with no extra tooling):
   docker exec harness-litellm printenv LITELLM_MASTER_KEY
-  # Or from Infisical vault (project=infrastructure env=prod) - NOTE: --plain is broken on CLI 0.43.110 (prints nothing):
-  infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production | awk '$1=="LITELLM_MASTER_KEY"{print $NF}'
+  # Note: the same value is stored in /opt/inference-harness/.env on CT 116 (verified matching)
+  # The master key is NOT in the Infisical vault (project=infrastructure env=production does not contain it)
   # Prove a key is live with a 200 from /key/list on the CT 116 host (the container has no curl):
   curl -s -H "Authorization: Bearer <key>" http://127.0.0.1:4000/key/list | jq length
   ```

litellm-health.prose.md

@@ -19,7 +19,7 @@ description: >
     Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/.
   - Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver
     firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09.
-  - Prometheus node job covers ALL 6 PVE nodes (.4/.5/.6/.9/.12/.15:9100).
+  - Prometheus node job covers ALL 5 PVE nodes (.5/.6/.9/.12/.15:9100).
 ---
 
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)

pm2-self-heal.prose.md

@@ -2,16 +2,6 @@
 kind: responsibility
 name: pm2-self-heal
 description: >
-  Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner,
-  spoton-service, zulip-watchdog) and auto-restarts any that are stopped or
-  errored. Logs every action to the knowledge graph and alerts the owner via
-  Zulip DM on failures.
-  CRITICAL: Never restart abiba-zulip — it runs this contract.
-  AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative):
-  gpu-monitor is systemd-managed (gpu-monitor.service) — NOT PM2;
-  gpu-watchdog decommissioned (function folded into gpu-monitor.service);
-  gitea-runner KEPT (online in PM2); abiba-zulip KEPT (online 4d+, the
-  2026-07-04 'removed/decommissioned' note was stale and is removed).
 ---
 
 ## Maintains

scripts/litellm-health-check.py

@@ -35,7 +35,12 @@ def run_command(cmd, timeout=15):
         return 1, "", str(e)
 
 def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, follow_redirects=False):
-    """Probe HTTP endpoint and return status code"""
+    """Probe HTTP endpoint and return (status_code, failure_kind)
+    
+    Returns:
+        (code, None) if successful or HTTP response received
+        (000, kind) if connection failed, where kind is 'timeout', 'refused', 'dns', etc.
+    """
     cmd = "curl -s -o /dev/null -w '%{http_code}' -m " + str(timeout)
     if method == "POST":
         cmd += " -X POST"
@@ -47,15 +52,47 @@ def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, foll
         cmd += " -L"
     cmd += " '" + url + "'"
     
-    rc, stdout, stderr = run_command(cmd, timeout)
-    if rc != 0 and "TIMEOUT" not in stderr:
-        return 000  # Connection failed
+    try:
+        rc, stdout, stderr = run_command(cmd, timeout)
+        if rc != 0:
+            # Determine failure kind from curl exit code
+            # curl exit codes: 28=timeout, 7=refused, 6=dns, 35=ssl, 52=empty
+            if rc == 28:
+                return (000, "timeout after " + str(timeout) + "s")
+            elif rc == 7:
+                return (000, "connection refused")
+            elif rc == 6:
+                return (000, "dns failure")
+            elif rc == 35:
+                return (000, "ssl error")
+            elif rc == 52:
+                return (000, "empty response")
+            else:
+                return (000, "curl exit " + str(rc))
+        return (int(stdout), None) if stdout.isdigit() else (000, "unparseable response")
+    except subprocess.TimeoutExpired:
+        return (000, "timeout after " + str(timeout) + "s")
+
+def get_response_body(url, method="POST", bearer_token=None, data=None, timeout=30):
+    """Get response body for 401/403 credential faults (truncated to 200 chars)"""
+    cmd = "curl -s -m " + str(timeout)
+    if method == "POST":
+        cmd += " -X POST"
+    if bearer_token:
+        cmd += " -H 'Authorization: Bearer " + bearer_token + "'"
+    if data:
+        cmd += " -H 'Content-Type: application/json' -d '" + data + "'"
+    cmd += " '" + url + "'"
     
-    return int(stdout) if stdout.isdigit() else 000
+    rc, stdout, stderr = run_command(cmd, timeout)
+    # Return first 200 chars, single line
+    body = stdout.replace('\n', ' ').replace('\t', ' ')[:200] if stdout else ""
+    return body
+
 
 def check_liveliness():
     """Step 1: Liveliness probe"""
-    code = probe_http("http://" + BACKEND_HOST + "/litellm/health/liveliness")
+    code, _ = probe_http("http://" + BACKEND_HOST + "/litellm/health/liveliness")
     return "Liveliness", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/health/liveliness)"
 
 def check_containers():
@@ -79,32 +116,61 @@ def check_model_probes():
     results = []
     
     for model in ["gpu-dense", "gpu-vision", "strix-moe"]:
-        # Single-host aliases: 30s timeout
-        code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
-                         method="POST",
-                         bearer_token=monitor_key,
-                         data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
-                         timeout=30)
+        # Single-host aliases: 30s timeout each
+        # gpu-dense (RTX 3090) may need long warmup/prefill - timeout is acceptable on cold-start
+        code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                       method="POST",
+                                       bearer_token=monitor_key,
+                                       data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
+                                       timeout=30)
         
-        results.append((model, code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
+        if code == 000 and failure_kind:
+            # Report probe failure with kind, do not assert a service verdict
+            results.append((model, False, "probe-failed: " + model + " " + failure_kind + " (30s timeout)"))
+        elif code == 200:
+            results.append((model, True, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
+        elif code in (401, 403):
+            # Credential fault - capture body and key alias
+            body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                    method="POST",
+                                    bearer_token=monitor_key,
+                                    data='{"model":"' + model + '","messages":[{"role":"user","content":"health"}],"max_tokens":4}',
+                                    timeout=10)
+            # Resolve key alias
+            alias = "monitor-20260813"  # Known from /etc/litellm-monitor.env on CT 116
+            results.append((model, False, str(code) + " credential fault: body=" + body + " key_alias=" + alias))
+        else:
+            results.append((model, False, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
     
     # Pool alias (syslog-auto): 60s timeout, retry once on 000
-    code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
-                     method="POST",
-                     bearer_token=monitor_key,
-                     data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
-                     timeout=60)
+    code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                   method="POST",
+                                   bearer_token=monitor_key,
+                                   data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
+                                   timeout=60)
     
-    if code == 000:
+    if code == 000 and failure_kind:
         # Retry once with same timeout
         time.sleep(1)
-        code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
-                         method="POST",
-                         bearer_token=monitor_key,
-                         data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
-                         timeout=60)
-    
-    results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
+        code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                       method="POST",
+                                       bearer_token=monitor_key,
+                                       data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
+                                       timeout=60)
+        if code == 000 and failure_kind:
+            results.append(("syslog-auto", False, "probe-failed: syslog-auto " + failure_kind + " (60s timeout, retry)"))
+        elif code in (401, 403):
+            body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                    method="POST",
+                                    bearer_token=monitor_key,
+                                    data='{"model":"syslog-auto","messages":[{"role":"user","content":"health"}],"max_tokens":4}',
+                                    timeout=10)
+            alias = "monitor-20260813"
+            results.append(("syslog-auto", False, str(code) + " credential fault: body=" + body + " key_alias=" + alias))
+        else:
+            results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
+    else:
+        results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
     
     return results
 
@@ -146,18 +212,18 @@ def check_admin_key_list():
 
 def check_github_status():
     """Step 3: GitHub status - 301 redirect is acceptable for status page"""
-    code = probe_http("https://status.github.com/api/status.json", timeout=15)
+    code, _ = probe_http("https://status.github.com/api/status.json", timeout=15)
     # GitHub status API returns 301 redirect, which is expected behavior
     return "GitHub Status", code == 301, str(code)
 
 def check_prometheus():
     """Step 4: Prometheus health"""
-    code = probe_http("http://" + BACKEND_HOST + ":9090/-/healthy")
+    code, _ = probe_http("http://" + BACKEND_HOST + ":9090/-/healthy")
     return "Prometheus", code == 200, str(code) + " (target: " + BACKEND_HOST + ":9090/-/healthy)"
 
 def check_grafana():
     """Step 9: Grafana health"""
-    code = probe_http("http://" + BACKEND_HOST + ":3001/api/health")
+    code, _ = probe_http("http://" + BACKEND_HOST + ":3001/api/health")
     return "Grafana", code == 200, str(code) + " (target: " + BACKEND_HOST + ":3001/api/health)"
 
 def check_docker_stats():