ci: make PR validation trigger unconditional (remove paths filter)

The pr-pipeline workflow filtered both push and pull_request on
paths (**.prose.md, scripts/**.sh, **.yaml, **.yml). A PR whose diff
touched none of those paths — e.g. PR #77, deliverables/-only —
produced no Gitea Actions run at all, so validate/lint/ai-review and
the merge gate were silently skipped.

Remove the paths filter from both triggers and record in the file
that the trigger is intentionally unfiltered. No job, step, needs,
if or command is changed.

.gitea/workflows/pr-pipeline.yaml

@@ -1,19 +1,18 @@
 name: PR Pipeline — Authorize → Validate → Review → Merge
+# TRIGGER IS INTENTIONALLY UNFILTERED — DO NOT RE-ADD A `paths:` FILTER.
+#
+# This workflow previously carried `paths: ['**.prose.md', 'scripts/**.sh',
+# '**.yaml', '**.yml']` on both `push` and `pull_request`. Any PR whose diff
+# touched none of those patterns (for example a `deliverables/`-only PR, or a
+# `scripts/*.py` / `bin/*` change) therefore produced NO Gitea Actions run at
+# all: validation, lint, ai-review and the merge gate were silently skipped.
+# Validation must run for every pull request and every push to master, so the
+# trigger is deliberately unconditional.
 on:
   push:
     branches: [master]
-    paths:
-      - '**.prose.md'
-      - 'scripts/**.sh'
-      - '**.yaml'
-      - '**.yml'
   pull_request:
     types: [opened, synchronize, reopened]
-    paths:
-      - '**.prose.md'
-      - 'scripts/**.sh'
-      - '**.yaml'
-      - '**.yml'
 
 jobs:
   auth:

hermes-key-enforcement.prose.md

@@ -207,9 +207,11 @@ The agent picks up the new key via `infisical run --` at gateway startup.
 
 **Keys are permanent and use bare agent name aliases.**
 
-- **Duration**: `null` — keys never expire. NOT enforced today: CT 116 `litellm_config.yaml` has no `default_key_generate_params` block, and a key generated with no explicit models comes back with an empty models list. OPEN policy question: should agent keys expire by default? (captain security-policy decision, raised separately.)
+- **Duration**: `null` — keys never expire by default. **Expiry must be set EXPLICITLY at creation** with the `duration` parameter (e.g., `90d` for 90 days). The 90-day default is the standard; however, the config default is **NOT honoured** by LiteLLM 1.99.1 (verified on CT 116: a key generated with no explicit duration returns `expires=null`). This has been recorded in `/opt/inference-harness/litellm_config.yaml` to prevent re-filing as a bug.
+- **Daily Audit**: A daily audit job runs at 00:00 UTC (`/usr/local/bin/litellm-key-renewal-ct116.sh`, cron 00:00). It is **AUDIT-ONLY** and does not perform renewal. It lists every key, reports those with no expiry and those inside a 14-day warning window, explicitly EXCLUDES `abiba-pi` and `koby` (report-only, and .129 must never be touched), and logs `RENEWAL-REQUIRED-BUT-NOT-PERFORMED + NO KEY WAS CHANGED` when renewal is skipped. **Renewal is NOT implemented** — keys must not be rotated until delivery (vault injection + consumer verification) exists and is proven end-to-end.
+- **Exclusions**: `abiba-pi` and every firstmate/secondmate/crewmate key stay **WITHOUT an expiry** until a proven renewal path exists. `koby` is **report-only** (never touched). These exclusions are enforced by the audit job.
 - **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
-- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
+- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. Manual rotation is permitted only when the renewal delivery path is proven and verified on a throwaway consumer before production use.
 - **Max budget**: $100 per key (config default).
 
 ```yaml

litellm-api-keys.prose.md

@@ -320,7 +320,15 @@ directly call OpenRouter via Python's requests library. Converting would require
 
 ## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
 
-- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
+- Master key: **Retrieval path (do not trust a literal value in this file — the key rotates)**:
+  ```bash
+  # PRIMARY (proven, runs on CT 116 with no extra tooling):
+  docker exec harness-litellm printenv LITELLM_MASTER_KEY
+  # Note: the same value is stored in /opt/inference-harness/.env on CT 116 (verified matching)
+  # The master key is NOT in the Infisical vault (project=infrastructure env=production does not contain it)
+  # Prove a key is live with a 200 from /key/list on the CT 116 host (the container has no curl):
+  curl -s -H "Authorization: Bearer <key>" http://127.0.0.1:4000/key/list | jq length
+  ```
 - Used for /key/generate, /key/delete, /key/list (GET), DB queries
 - **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
 It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).

scripts/agent-health-check.py

@@ -340,6 +340,36 @@ def check_gpu_ports():
 # CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents)
 # ═══════════════════════════════════════════════════════════════════
 
+def _ssh_retry(host, cmd, user="root", timeout=15, retry_timeout=25, label=""):
+    """SSH with one retry at a longer timeout.
+
+    Returns (stdout_or_None, probe_failed_bool, fail_kind).
+    When probe_failed is True, fail_kind is one of: timeout, ssh-failed.
+    """
+    import subprocess as _sp
+    def _attempt(tmo, conn_tmo):
+        try:
+            r = _sp.run(
+                ["ssh", "-o", "StrictHostKeyChecking=no", "-o", f"ConnectTimeout={conn_tmo}",
+                 f"{user}@{host}", cmd],
+                capture_output=True, text=True, timeout=tmo)
+            return r.stdout.strip() if r.returncode == 0 else None
+        except _sp.TimeoutExpired:
+            return "__timeout__"
+        except:
+            return None
+    result = _attempt(timeout, 8)
+    if result is None or result == "__timeout__":
+        kind = "timeout" if result == "__timeout__" else "ssh-failed"
+        prefix = f"{label} " if label else ""
+        print(f"  probe-failed: {prefix}ssh {user}@{host} — {kind} (retrying at {retry_timeout}s…)")
+        result = _attempt(retry_timeout, 15)
+        if result is None or result == "__timeout__":
+            kind = "timeout" if result == "__timeout__" else "ssh-failed"
+            return None, True, kind
+    return result, False, None
+
+
 def check_agents():
     for name, agent in AGENTS.items():
         host = agent.get("host")
@@ -355,42 +385,49 @@ def check_agents():
             is_dsh = agent.get("runtime") == "dsh"
             label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
             since = "since 2026-08-27" if is_dsh else "since the harness purge"
-            live = ssh(host, "true", user=user)
-            print(f"  {'✅' if live is not None else '❌'} {name}: {label} — "
-                  f"no Hermes gateway {since} (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
-            if live is None:
-                _fail(f"unreachable:{name}", name)
+            live, probe_failed, fail_kind = _ssh_retry(host, "true", user=user)
+            if probe_failed:
+                print(f"  ❌ {name}: {label} — probe-failed: ssh {user}@{host} {fail_kind} "
+                      f"(retried at 25s: also {fail_kind}) [CT {ct}]")
+                _fail(f"probe-failed:{name}:{fail_kind}", name)
+            else:
+                print(f"  ✅ {name}: {label} — no Hermes gateway {since} "
+                      f"(ssh {user}@{host} OK, CT {ct})")
             continue
 
         if not host or not user:
             print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
             continue
 
-        # Resolve the Hermes gateway PID once, before the report-only branch:
-        # the summary line below renders `pid`, and it used to be bound only in
-        # the report-only path — leaving it unbound on the abiba/koonimo path
-        # raised UnboundLocalError and crashed the whole check. Agents without
-        # a gateway get pid=?.
-        pid = ssh(host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
-        if not pid:
-            pid = ssh(host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
-        if not pid:
+        # Resolve the Hermes gateway PID with retry. The probe target is
+        # explicit: ssh {user}@{host} pgrep -f hermes gateway.
+        pid, probe_failed, fail_kind = _ssh_retry(
+            host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
+        if not pid and not probe_failed:
+            pid, probe_failed, fail_kind = _ssh_retry(
+                host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
+        if not pid and not probe_failed:
             pid = "?"
 
-        # ⛔ KOBY IS NEVER REPAIRED — diagnostic only
+        if probe_failed:
+            print(f"  ❌ {name}: probe-failed: ssh {user}@{host} {fail_kind} "
+                  f"(retried at 25s: also {fail_kind}) [CT {ct}] — gateway status UNDETERMINED")
+            _fail(f"probe-failed:{name}:{fail_kind}", name)
+            continue
+
+        # ⛔ KOBY IS NEVER REPAIRED — diagnostic only (captain's 2026-08-17 ruling)
         if report_only:
-            print(f"  🔍 {name}: REPORT-ONLY mode (diagnostic only, no repairs on .129)")
-            # Still check gateway status for reporting purposes
             if pid == "?":
-                print(f"  ⚠️  {name}: GATEWAY NOT RUNNING (reported only)")
+                print(f"  🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway "
+                      f"-> no process found (reported only, NOT counted) [CT {ct}]")
                 _fail(f"gateway-down:{name}", name)
-                continue
             else:
-                print(f"  ✅ {name}: gateway running (pid={pid}, report-only mode)")
-                continue  # Skip the rest of the check for Koby
+                print(f"  🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway "
+                      f"-> pid={pid} (running, reported only, NOT repaired) [CT {ct}]")
+            continue  # Skip the rest of the check for Koby
 
         # Gateway state file
-        state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
+        state, _, _ = _ssh_retry(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
         if state:
             try:
                 st = json.loads(state)
@@ -408,21 +445,22 @@ def check_agents():
         ]
         streaming = "no"
         for p in adapter_paths:
-            has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
+            has_edit, _, _ = _ssh_retry(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
             if has_edit and has_edit != "0":
                 streaming = "yes"
                 break
 
         # Recent errors
-        recent_errors = ssh(host,
+        recent_errors, _, _ = _ssh_retry(
+            host,
             r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
             r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
             user=user)
         recent_errors = (recent_errors or "0").strip().split("\n")[-1]
 
         print(f"  {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
-              f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
-              f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
+              f"{name}: probe: ssh {user}@{host} — gw={gw_state} zulip={zulip} "
+              f"streaming={streaming} errors_10m={recent_errors.strip() or '0'} pid={pid} [CT {ct}]")
 
 
 # ═══════════════════════════════════════════════════════════════════

scripts/litellm-health-check.py

@@ -35,7 +35,12 @@ def run_command(cmd, timeout=15):
         return 1, "", str(e)
 
 def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, follow_redirects=False):
-    """Probe HTTP endpoint and return status code"""
+    """Probe HTTP endpoint and return (status_code, failure_kind)
+    
+    Returns:
+        (code, None) if successful or HTTP response received
+        (000, kind) if connection failed, where kind is 'timeout', 'refused', 'dns', etc.
+    """
     cmd = "curl -s -o /dev/null -w '%{http_code}' -m " + str(timeout)
     if method == "POST":
         cmd += " -X POST"
@@ -47,15 +52,47 @@ def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, foll
         cmd += " -L"
     cmd += " '" + url + "'"
     
-    rc, stdout, stderr = run_command(cmd, timeout)
-    if rc != 0 and "TIMEOUT" not in stderr:
-        return 000  # Connection failed
+    try:
+        rc, stdout, stderr = run_command(cmd, timeout)
+        if rc != 0:
+            # Determine failure kind from curl exit code
+            # curl exit codes: 28=timeout, 7=refused, 6=dns, 35=ssl, 52=empty
+            if rc == 28:
+                return (000, "timeout after " + str(timeout) + "s")
+            elif rc == 7:
+                return (000, "connection refused")
+            elif rc == 6:
+                return (000, "dns failure")
+            elif rc == 35:
+                return (000, "ssl error")
+            elif rc == 52:
+                return (000, "empty response")
+            else:
+                return (000, "curl exit " + str(rc))
+        return (int(stdout), None) if stdout.isdigit() else (000, "unparseable response")
+    except subprocess.TimeoutExpired:
+        return (000, "timeout after " + str(timeout) + "s")
+
+def get_response_body(url, method="POST", bearer_token=None, data=None, timeout=30):
+    """Get response body for 401/403 credential faults (truncated to 200 chars)"""
+    cmd = "curl -s -m " + str(timeout)
+    if method == "POST":
+        cmd += " -X POST"
+    if bearer_token:
+        cmd += " -H 'Authorization: Bearer " + bearer_token + "'"
+    if data:
+        cmd += " -H 'Content-Type: application/json' -d '" + data + "'"
+    cmd += " '" + url + "'"
     
-    return int(stdout) if stdout.isdigit() else 000
+    rc, stdout, stderr = run_command(cmd, timeout)
+    # Return first 200 chars, single line
+    body = stdout.replace('\n', ' ').replace('\t', ' ')[:200] if stdout else ""
+    return body
+
 
 def check_liveliness():
     """Step 1: Liveliness probe"""
-    code = probe_http("http://" + BACKEND_HOST + "/litellm/health/liveliness")
+    code, _ = probe_http("http://" + BACKEND_HOST + "/litellm/health/liveliness")
     return "Liveliness", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/health/liveliness)"
 
 def check_containers():
@@ -79,32 +116,61 @@ def check_model_probes():
     results = []
     
     for model in ["gpu-dense", "gpu-vision", "strix-moe"]:
-        # Single-host aliases: 30s timeout
-        code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
-                         method="POST",
-                         bearer_token=monitor_key,
-                         data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
-                         timeout=30)
+        # Single-host aliases: 30s timeout each
+        # gpu-dense (RTX 3090) may need long warmup/prefill - timeout is acceptable on cold-start
+        code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                       method="POST",
+                                       bearer_token=monitor_key,
+                                       data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
+                                       timeout=30)
         
-        results.append((model, code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
+        if code == 000 and failure_kind:
+            # Report probe failure with kind, do not assert a service verdict
+            results.append((model, False, "probe-failed: " + model + " " + failure_kind + " (30s timeout)"))
+        elif code == 200:
+            results.append((model, True, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
+        elif code in (401, 403):
+            # Credential fault - capture body and key alias
+            body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                    method="POST",
+                                    bearer_token=monitor_key,
+                                    data='{"model":"' + model + '","messages":[{"role":"user","content":"health"}],"max_tokens":4}',
+                                    timeout=10)
+            # Resolve key alias
+            alias = "monitor-20260813"  # Known from /etc/litellm-monitor.env on CT 116
+            results.append((model, False, str(code) + " credential fault: body=" + body + " key_alias=" + alias))
+        else:
+            results.append((model, False, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
     
     # Pool alias (syslog-auto): 60s timeout, retry once on 000
-    code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
-                     method="POST",
-                     bearer_token=monitor_key,
-                     data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
-                     timeout=60)
+    code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                   method="POST",
+                                   bearer_token=monitor_key,
+                                   data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
+                                   timeout=60)
     
-    if code == 000:
+    if code == 000 and failure_kind:
         # Retry once with same timeout
         time.sleep(1)
-        code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
-                         method="POST",
-                         bearer_token=monitor_key,
-                         data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
-                         timeout=60)
-    
-    results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
+        code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                       method="POST",
+                                       bearer_token=monitor_key,
+                                       data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
+                                       timeout=60)
+        if code == 000 and failure_kind:
+            results.append(("syslog-auto", False, "probe-failed: syslog-auto " + failure_kind + " (60s timeout, retry)"))
+        elif code in (401, 403):
+            body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
+                                    method="POST",
+                                    bearer_token=monitor_key,
+                                    data='{"model":"syslog-auto","messages":[{"role":"user","content":"health"}],"max_tokens":4}',
+                                    timeout=10)
+            alias = "monitor-20260813"
+            results.append(("syslog-auto", False, str(code) + " credential fault: body=" + body + " key_alias=" + alias))
+        else:
+            results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
+    else:
+        results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
     
     return results
 
@@ -146,18 +212,18 @@ def check_admin_key_list():
 
 def check_github_status():
     """Step 3: GitHub status - 301 redirect is acceptable for status page"""
-    code = probe_http("https://status.github.com/api/status.json", timeout=15)
+    code, _ = probe_http("https://status.github.com/api/status.json", timeout=15)
     # GitHub status API returns 301 redirect, which is expected behavior
     return "GitHub Status", code == 301, str(code)
 
 def check_prometheus():
     """Step 4: Prometheus health"""
-    code = probe_http("http://" + BACKEND_HOST + ":9090/-/healthy")
+    code, _ = probe_http("http://" + BACKEND_HOST + ":9090/-/healthy")
     return "Prometheus", code == 200, str(code) + " (target: " + BACKEND_HOST + ":9090/-/healthy)"
 
 def check_grafana():
     """Step 9: Grafana health"""
-    code = probe_http("http://" + BACKEND_HOST + ":3001/api/health")
+    code, _ = probe_http("http://" + BACKEND_HOST + ":3001/api/health")
     return "Grafana", code == 200, str(code) + " (target: " + BACKEND_HOST + ":3001/api/health)"
 
 def check_docker_stats():