From b2a259fa235822e407d519feb1d9796ac39a9665 Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 14:56:57 +0000 Subject: [PATCH 01/11] fix: add dsh-web restart-persistent authentication MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add capture-dsh-token.sh script that captures the dsh-web launch token - Add login endpoint (/dsh-web-login on :8081) that mints 30-day auth cookie - Document the authentication flow in zulip-health.prose.md (Platform B4) - Cookie is authority-bound to 127.0.0.1:3080 with 30-day expiry - After first login, subsequent requests use the cookie — no token required --- scripts/capture-dsh-token.sh | 77 ++++++++++++++++++++++++++++++++++++ zulip-health.prose.md | 37 +++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100755 scripts/capture-dsh-token.sh diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh new file mode 100755 index 0000000..0f1f532 --- /dev/null +++ b/scripts/capture-dsh-token.sh @@ -0,0 +1,77 @@ +#!/bin/bash +# capture-dsh-token.sh — start dsh-web, capture its token, update nginx +# Run on CT112 (tankodhs.sysloggh.net) +# This script: +# 1. Restarts the dsh-web service +# 2. Captures the token URL from the journal +# 3. Extracts the token value +# 4. Writes the token to /etc/dsh-web/launch-token +# 5. Creates an nginx config that exposes a /dsh-web-login endpoint +# 6. Reloads nginx + +set -euo pipefail + +# Kill any existing dsh-web instance first +systemctl stop dsh-web 2>/dev/null || true +sleep 2 + +# Record the time we started the service (for --since filter) +START_TIME="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + +# Start dsh-web +systemctl start dsh-web + +# Wait for the token to appear in the journal (up to 30 seconds) +TOKEN="" +for i in {1..30}; do + TOKEN=$(journalctl -u dsh-web.service --since "$START_TIME" --output=cat 2>/dev/null | grep -m1 "dsh web: http://" | grep -oP "(?<=dsh web: )(https?://[^ ]+)" | head -1 || true) + if [ -n "$TOKEN" ]; then + break + fi + sleep 1 +done + +if [ -z "$TOKEN" ]; then + echo "ERROR: token not captured within 30s" >&2 + exit 1 +fi + +# Extract the token value (everything after "?token=") +TOKEN_VALUE=$(echo "$TOKEN" | grep -oP "(?<=token=)[^ ]+") + +# Write the token to a file +mkdir -p /etc/dsh-web +echo "$TOKEN_VALUE" > /etc/dsh-web/launch-token +echo "Captured token: $TOKEN_VALUE" + +# Create the nginx config with the token (using printf to control expansion) +{ + printf "server {\n" + printf " listen 8081;\n" + printf " server_name _;\n" + printf " \n" + printf " location /dsh-web-login {\n" + printf " proxy_pass http://127.0.0.1:3080/?token=%s;\n" "$TOKEN_VALUE" + printf " proxy_http_version 1.1;\n" + printf " proxy_set_header Host 127.0.0.1:3080;\n" + printf " proxy_set_header X-Real-IP \$remote_addr;\n" + printf " proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n" + printf " }\n" + printf " \n" + printf " location / {\n" + printf " proxy_pass http://127.0.0.1:3080;\n" + printf " proxy_http_version 1.1;\n" + printf " proxy_set_header Host 127.0.0.1:3080;\n" + printf " proxy_set_header X-Real-IP \$remote_addr;\n" + printf " proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n" + printf " }\n" + printf "}\n" +} > /etc/nginx/sites-enabled/dsh.token + +# Reload nginx +nginx -t && /usr/sbin/nginx -s reload || { + echo "ERROR: failed to reload nginx" >&2 + exit 1 +} + +echo "Token captured and nginx reloaded" diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 6ebafd4..cc47dcb 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -260,6 +260,43 @@ logged/reported as a warning — reported, never healed on. | `dsh-web` service not `active` | Restart Tanko via DSH service | | HTTP `:3080` connection refused/timeout (`000`) | Same as above | | HTTP status outside the expected set | Log/report as a warning — reported, never healed on | +**B4: dsh-web Authentication (Tanko — restart-persistent login)** + +The dsh-web UI is token-gated. Each dsh-web process generates a unique +launch token printed to the journal at startup. The token is used to mint +a 30-day authentication cookie. After the first authenticated login, +subsequent requests use the cookie — no token required. + +**Login endpoint**: `http://127.0.0.1:8081/dsh-web-login` (inside CT 112) + +**Token capture script**: `/opt/deepseek-harness/capture-dsh-token.sh` (CT 112) + +The script: +1. Restarts the dsh-web service +2. Captures the token URL from the journal +3. Extracts the token value +4. Writes the token to `/etc/dsh-web/launch-token` +5. Creates an nginx config that exposes the `/dsh-web-login` endpoint on port 8081 +6. Reloads nginx + +**Authentication flow**: +1. Access `http://127.0.0.1:8081/dsh-web-login` → 303 redirect +2. The redirect includes a `Set-Cookie` header with the `dsh-auth-*` cookie +3. The cookie has a 30-day expiry and is authority-bound to `127.0.0.1:3080` +4. Subsequent requests to `http://127.0.0.1:3080/` use the cookie for authentication +5. After 30 days, the cookie expires and a new token exchange is required + +**Verification**: +```bash +# Check if the login endpoint is working: +ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8081/dsh-web-login" +# Expected: 303 + +# Check if the cookie works: +ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' -b 'dsh-auth-*' http://127.0.0.1:3080/" +# Expected: 200 (after the cookie has been set) +``` + ### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14) -- 2.54.0 From 85ea1f4f3d962f319cc9312a08a387c3c95bbd26 Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 15:22:41 +0000 Subject: [PATCH 02/11] no-mistakes(review): harden dsh token capture: loopback bind, atomic nginx config --- scripts/capture-dsh-token.sh | 57 +++++++++++++++++++++++++++++------- zulip-health.prose.md | 11 +++++-- 2 files changed, 54 insertions(+), 14 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index 0f1f532..d0a87e9 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -10,6 +10,7 @@ # 6. Reloads nginx set -euo pipefail +umask 077 # Kill any existing dsh-web instance first systemctl stop dsh-web 2>/dev/null || true @@ -39,18 +40,36 @@ fi # Extract the token value (everything after "?token=") TOKEN_VALUE=$(echo "$TOKEN" | grep -oP "(?<=token=)[^ ]+") -# Write the token to a file +# Reject tokens that could break nginx config or the request URI +if ! printf '%s' "$TOKEN_VALUE" | grep -qE '^[A-Za-z0-9._~+/=%:@-]+$'; then + echo "ERROR: token contains unsupported characters" >&2 + exit 1 +fi + +# Write the token to a restricted file mkdir -p /etc/dsh-web -echo "$TOKEN_VALUE" > /etc/dsh-web/launch-token -echo "Captured token: $TOKEN_VALUE" +printf '%s\n' "$TOKEN_VALUE" > /etc/dsh-web/launch-token +chmod 600 /etc/dsh-web/launch-token +echo "Captured dsh-web launch token" # Create the nginx config with the token (using printf to control expansion) +NGINX_ENABLED="/etc/nginx/sites-enabled/dsh.token" +NGINX_STAGE_DIR="/etc/nginx/sites-available" +mkdir -p "$NGINX_STAGE_DIR" + +TMP_CONFIG="$(mktemp "$NGINX_STAGE_DIR/dsh.token.XXXXXX")" +BACKUP="" +if [ -f "$NGINX_ENABLED" ]; then + BACKUP="$(mktemp "$NGINX_STAGE_DIR/dsh.token.bak.XXXXXX")" + cp -p "$NGINX_ENABLED" "$BACKUP" +fi + { printf "server {\n" - printf " listen 8081;\n" + printf " listen 127.0.0.1:8081;\n" printf " server_name _;\n" printf " \n" - printf " location /dsh-web-login {\n" + printf " location = /dsh-web-login {\n" printf " proxy_pass http://127.0.0.1:3080/?token=%s;\n" "$TOKEN_VALUE" printf " proxy_http_version 1.1;\n" printf " proxy_set_header Host 127.0.0.1:3080;\n" @@ -66,12 +85,28 @@ echo "Captured token: $TOKEN_VALUE" printf " proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n" printf " }\n" printf "}\n" -} > /etc/nginx/sites-enabled/dsh.token +} > "$TMP_CONFIG" +chmod 600 "$TMP_CONFIG" -# Reload nginx -nginx -t && /usr/sbin/nginx -s reload || { - echo "ERROR: failed to reload nginx" >&2 - exit 1 +mv "$TMP_CONFIG" "$NGINX_ENABLED" +CONFIG_APPLIED=1 +restore_on_exit() { + if [ "$CONFIG_APPLIED" -eq 1 ]; then + if [ -n "$BACKUP" ]; then + if cp -p "$BACKUP" "$NGINX_ENABLED" 2>/dev/null; then rm -f "$BACKUP"; fi + else + rm -f "$NGINX_ENABLED" + fi + fi } +trap restore_on_exit EXIT -echo "Token captured and nginx reloaded" +if nginx -t; then + /usr/sbin/nginx -s reload + CONFIG_APPLIED=0 + if [ -n "$BACKUP" ]; then rm -f "$BACKUP"; fi + echo "Token captured and nginx reloaded" +else + echo "ERROR: nginx config test failed; rolling back" >&2 + exit 1 +fi diff --git a/zulip-health.prose.md b/zulip-health.prose.md index cc47dcb..c6dff2a 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -292,9 +292,14 @@ The script: ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8081/dsh-web-login" # Expected: 303 -# Check if the cookie works: -ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' -b 'dsh-auth-*' http://127.0.0.1:3080/" -# Expected: 200 (after the cookie has been set) +# Mint the 30-day cookie from the login endpoint: +ssh root@192.168.68.15 "pct exec 112 -- rm -f /tmp/dsh.jar" +ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null -w '%{http_code}' http://127.0.0.1:8081/dsh-web-login" +# Expected: 303 + +# Check if the stored cookie authenticates against dsh-web: +ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" +# Expected: 200 ``` -- 2.54.0 From 266fa1f835900ff1fb9c91efa6596a8ece5c038b Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 16:52:45 +0000 Subject: [PATCH 03/11] fix(dsh-web-auth): Authentik-gated :80 login + non-disruptive token capture Correct the dsh-web authentication fix after parent correction: - Remove the unauthenticated :8081 endpoint (0.0.0.0 bind with no auth_request = full Authentik bypass for the LAN). The script now removes /etc/nginx/sites-enabled/dsh.token automatically if it reappears. - Put the login path inside the Authentik-gated :80 server block as location = /dsh-web-login; proxy to dsh-web with Host = tankodhs.sysloggh.net so the 30-day cookie is bound to the public authority, never to 127.0.0.1:3080. - Isolate the rotating token in a generated include /etc/dsh-web/nginx-login.conf; reload nginx only when it changes. - Replace the disruptive capture (systemctl stop/start dsh-web) with a non-disruptive read of the running service's journal, scoped to the current systemd invocation so a restarted process's stale token is never reused while the new banner is still pending. - Keep x-dsh-task-board-proxy-token and Host $ak_origin_host intact in '/'. - Document the corrected design (B4) in zulip-health.prose.md, v3.2.0. Live-verified 2026-09-11: no auth bypass (302), :8081 refused (000), a cookie minted before two dsh-web restarts still returns 200, the refreshed token mints a fresh cookie, and the systemd ExecStartPost/timer refreshes the token automatically without touching dsh-web. --- scripts/capture-dsh-token.sh | 207 ++++++++++++++++++++--------------- zulip-health.prose.md | 147 ++++++++++++++++++------- 2 files changed, 227 insertions(+), 127 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index d0a87e9..7b49bc8 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -1,112 +1,141 @@ -#!/bin/bash -# capture-dsh-token.sh — start dsh-web, capture its token, update nginx -# Run on CT112 (tankodhs.sysloggh.net) +#!/usr/bin/env bash +# capture-dsh-token.sh — refresh the dsh-web login token WITHOUT restarting dsh-web. +# +# Context (CT 112 / tankodhs.sysloggh.net) +# ---------------------------------------- +# The dsh-web UI (systemd unit `dsh-web.service`, 127.0.0.1:3080) prints a random +# launch token to the journal on every start: +# +# dsh web: http://127.0.0.1:3080/?token= +# +# That token is the only way to bootstrap the authority-bound 30-day browser +# cookie. It rotates on every dsh-web start, so the Authentik-gated +# `location = /dsh-web-login` in /etc/nginx/sites-available/dsh must always +# reference the token of the RUNNING process. +# # This script: -# 1. Restarts the dsh-web service -# 2. Captures the token URL from the journal -# 3. Extracts the token value -# 4. Writes the token to /etc/dsh-web/launch-token -# 5. Creates an nginx config that exposes a /dsh-web-login endpoint -# 6. Reloads nginx - +# 1. reads the LATEST launch token from the running service's journal — it +# NEVER stops or starts dsh-web, +# 2. records it in /etc/dsh-web/launch-token, +# 3. regenerates the nginx include /etc/dsh-web/nginx-login.conf (the +# `proxy_pass ...?token=` line consumed by /dsh-web-login), +# 4. validates with `nginx -t` and reloads ONLY when the token changed, +# rolling the include back if validation fails, +# 5. removes the legacy unauthenticated :8081 endpoint if it ever reappears. +# +# Idempotent and safe to run at any time (systemd ExecStartPost or timer). set -euo pipefail umask 077 +PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" -# Kill any existing dsh-web instance first -systemctl stop dsh-web 2>/dev/null || true -sleep 2 +JOURNAL_UNIT="dsh-web.service" +TOKEN_FILE="/etc/dsh-web/launch-token" +INCLUDE_FILE="/etc/dsh-web/nginx-login.conf" +SITE_ENABLED="/etc/nginx/sites-enabled/dsh" +LEGACY_8081="/etc/nginx/sites-enabled/dsh.token" +STASH_DIR="/etc/nginx/sites-available" -# Record the time we started the service (for --since filter) -START_TIME="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +log() { printf 'capture-dsh-token: %s\n' "$*" >&2; } +die() { printf 'capture-dsh-token: ERROR: %s\n' "$*" >&2; exit 1; } -# Start dsh-web -systemctl start dsh-web +[ "$(id -u)" -eq 0 ] || die "must run as root" -# Wait for the token to appear in the journal (up to 30 seconds) -TOKEN="" -for i in {1..30}; do - TOKEN=$(journalctl -u dsh-web.service --since "$START_TIME" --output=cat 2>/dev/null | grep -m1 "dsh web: http://" | grep -oP "(?<=dsh web: )(https?://[^ ]+)" | head -1 || true) - if [ -n "$TOKEN" ]; then - break +# ── 0. Remove the legacy unauthenticated :8081 endpoint, if present ───────── +# It bypassed Authentik entirely (listened on 0.0.0.0:8081 with no auth_request) +# and must never come back. Stash it rather than delete so it is auditable. +if [ -e "$LEGACY_8081" ] || [ -L "$LEGACY_8081" ]; then + STAMP="$(date -u +%Y%m%dT%H%M%SZ)" + STASHED="$STASH_DIR/dsh.token.disabled-$STAMP" + mv "$LEGACY_8081" "$STASHED" + if nginx -t >/dev/null 2>&1; then + nginx -s reload + log "removed legacy :8081 endpoint -> $STASHED" + else + mv "$STASHED" "$LEGACY_8081" + die "nginx config test failed after removing $LEGACY_8081; restored it" fi +fi + +# ── 1. Read the latest launch token from the RUNNING service ──────────────── +# Scope the journal to the service's CURRENT invocation. While a restarted +# process is still booting (~25s before it prints the banner), the newest token +# in the journal still belongs to the PREVIOUS process; without this filter an +# ExecStartPost run would silently keep the stale token. Never stop/start dsh-web. +INVOCATION="$(systemctl show -p InvocationID --value "$JOURNAL_UNIT" 2>/dev/null || true)" +JOURNAL_ARGS=(-u "$JOURNAL_UNIT") +if [ -n "$INVOCATION" ] && [ "$INVOCATION" != "n/a" ]; then + JOURNAL_ARGS+=("_SYSTEMD_INVOCATION_ID=$INVOCATION") +else + log "WARNING: no invocation id for $JOURNAL_UNIT; using latest journal token" +fi + +extract_token() { + grep -oE 'dsh web: https?://[^[:space:]]+[?&]token=[^[:space:]]+' \ + | tail -n1 | sed -E 's/.*[?&]token=//' || true +} + +TOKEN="" +for _ in $(seq 1 60); do + TOKEN="$(journalctl "${JOURNAL_ARGS[@]}" --no-pager -o cat 2>/dev/null | extract_token)" + [ -n "$TOKEN" ] && break sleep 1 done +# Fallback: the current invocation's start banner may have been rotated out of +# the journal; the newest matching line overall is then the best available. if [ -z "$TOKEN" ]; then - echo "ERROR: token not captured within 30s" >&2 - exit 1 + log "WARNING: no token for the current invocation; falling back to newest journal token" + TOKEN="$(journalctl -u "$JOURNAL_UNIT" --no-pager -o cat 2>/dev/null | extract_token)" +fi +[ -n "$TOKEN" ] || die "no launch token found in the $JOURNAL_UNIT journal" + +# The token must be safe to embed in a URI and in the nginx config. +printf '%s' "$TOKEN" | grep -qE '^[A-Za-z0-9._~+/=:@-]+$' \ + || die "captured token contains unsupported characters" + +# ── 2. Record the token (atomic, private) ────────────────────────────────── +mkdir -p "$(dirname "$TOKEN_FILE")" +if ! printf '%s\n' "$TOKEN" | cmp -s - "$TOKEN_FILE" 2>/dev/null; then + printf '%s\n' "$TOKEN" > "$TOKEN_FILE.tmp" + chmod 600 "$TOKEN_FILE.tmp" + mv "$TOKEN_FILE.tmp" "$TOKEN_FILE" + log "recorded new launch token in $TOKEN_FILE" fi -# Extract the token value (everything after "?token=") -TOKEN_VALUE=$(echo "$TOKEN" | grep -oP "(?<=token=)[^ ]+") +# ── 3. Regenerate the nginx login include (reload only when it changes) ──── +NEW_INCLUDE="$(mktemp "$INCLUDE_FILE.XXXXXX")" +printf 'proxy_pass http://127.0.0.1:3080/?token=%s;\n' "$TOKEN" > "$NEW_INCLUDE" +chmod 600 "$NEW_INCLUDE" -# Reject tokens that could break nginx config or the request URI -if ! printf '%s' "$TOKEN_VALUE" | grep -qE '^[A-Za-z0-9._~+/=%:@-]+$'; then - echo "ERROR: token contains unsupported characters" >&2 - exit 1 +if [ -f "$INCLUDE_FILE" ] && cmp -s "$NEW_INCLUDE" "$INCLUDE_FILE"; then + rm -f "$NEW_INCLUDE" + log "token unchanged; nginx not reloaded" + exit 0 fi -# Write the token to a restricted file -mkdir -p /etc/dsh-web -printf '%s\n' "$TOKEN_VALUE" > /etc/dsh-web/launch-token -chmod 600 /etc/dsh-web/launch-token -echo "Captured dsh-web launch token" +[ -e "$SITE_ENABLED" ] || { rm -f "$NEW_INCLUDE"; die "$SITE_ENABLED missing; refusing to reload"; } -# Create the nginx config with the token (using printf to control expansion) -NGINX_ENABLED="/etc/nginx/sites-enabled/dsh.token" -NGINX_STAGE_DIR="/etc/nginx/sites-available" -mkdir -p "$NGINX_STAGE_DIR" - -TMP_CONFIG="$(mktemp "$NGINX_STAGE_DIR/dsh.token.XXXXXX")" -BACKUP="" -if [ -f "$NGINX_ENABLED" ]; then - BACKUP="$(mktemp "$NGINX_STAGE_DIR/dsh.token.bak.XXXXXX")" - cp -p "$NGINX_ENABLED" "$BACKUP" +RESTORE="" +if [ -f "$INCLUDE_FILE" ]; then + RESTORE="$(mktemp "$INCLUDE_FILE.bak.XXXXXX")" + cp -p "$INCLUDE_FILE" "$RESTORE" fi -{ - printf "server {\n" - printf " listen 127.0.0.1:8081;\n" - printf " server_name _;\n" - printf " \n" - printf " location = /dsh-web-login {\n" - printf " proxy_pass http://127.0.0.1:3080/?token=%s;\n" "$TOKEN_VALUE" - printf " proxy_http_version 1.1;\n" - printf " proxy_set_header Host 127.0.0.1:3080;\n" - printf " proxy_set_header X-Real-IP \$remote_addr;\n" - printf " proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n" - printf " }\n" - printf " \n" - printf " location / {\n" - printf " proxy_pass http://127.0.0.1:3080;\n" - printf " proxy_http_version 1.1;\n" - printf " proxy_set_header Host 127.0.0.1:3080;\n" - printf " proxy_set_header X-Real-IP \$remote_addr;\n" - printf " proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n" - printf " }\n" - printf "}\n" -} > "$TMP_CONFIG" -chmod 600 "$TMP_CONFIG" +mv "$NEW_INCLUDE" "$INCLUDE_FILE" +chmod 600 "$INCLUDE_FILE" -mv "$TMP_CONFIG" "$NGINX_ENABLED" -CONFIG_APPLIED=1 -restore_on_exit() { - if [ "$CONFIG_APPLIED" -eq 1 ]; then - if [ -n "$BACKUP" ]; then - if cp -p "$BACKUP" "$NGINX_ENABLED" 2>/dev/null; then rm -f "$BACKUP"; fi - else - rm -f "$NGINX_ENABLED" - fi +if ! nginx -t >/dev/null 2>&1; then + if [ -n "$RESTORE" ]; then + mv "$RESTORE" "$INCLUDE_FILE" + else + rm -f "$INCLUDE_FILE" fi -} -trap restore_on_exit EXIT - -if nginx -t; then - /usr/sbin/nginx -s reload - CONFIG_APPLIED=0 - if [ -n "$BACKUP" ]; then rm -f "$BACKUP"; fi - echo "Token captured and nginx reloaded" -else - echo "ERROR: nginx config test failed; rolling back" >&2 - exit 1 + die "nginx config test failed; previous include restored" fi +if [ -n "$RESTORE" ]; then + rm -f "$RESTORE" +fi + +nginx -s reload +log "token changed; nginx reloaded" +log "login endpoint: https://tankodhs.sysloggh.net/dsh-web-login (Authentik-gated)" diff --git a/zulip-health.prose.md b/zulip-health.prose.md index c6dff2a..e3392b6 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -3,7 +3,7 @@ kind: responsibility name: zulip-health description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (pi/Abiba Zulip bridge), Platform B (Tanko on DSH), and Platform C (Agent Zero Docker). Verifies bot registration, DM delivery, and cross-platform connectivity. Mumuni is no longer monitored from this host — she runs on her own container (kagentz CT 105 on minipve, .14) and is monitored on her side. title: Zulip Mesh Health Monitor — Multi-Platform -version: 3.1.0 +version: 3.2.0 runtime_contract: 2 agent: abiba report_only_agents: @@ -262,45 +262,116 @@ logged/reported as a warning — reported, never healed on. | HTTP status outside the expected set | Log/report as a warning — reported, never healed on | **B4: dsh-web Authentication (Tanko — restart-persistent login)** -The dsh-web UI is token-gated. Each dsh-web process generates a unique -launch token printed to the journal at startup. The token is used to mint -a 30-day authentication cookie. After the first authenticated login, -subsequent requests use the cookie — no token required. +The dsh-web UI is token-gated. On every start the process prints a random +launch token to the journal: -**Login endpoint**: `http://127.0.0.1:8081/dsh-web-login` (inside CT 112) - -**Token capture script**: `/opt/deepseek-harness/capture-dsh-token.sh` (CT 112) - -The script: -1. Restarts the dsh-web service -2. Captures the token URL from the journal -3. Extracts the token value -4. Writes the token to `/etc/dsh-web/launch-token` -5. Creates an nginx config that exposes the `/dsh-web-login` endpoint on port 8081 -6. Reloads nginx - -**Authentication flow**: -1. Access `http://127.0.0.1:8081/dsh-web-login` → 303 redirect -2. The redirect includes a `Set-Cookie` header with the `dsh-auth-*` cookie -3. The cookie has a 30-day expiry and is authority-bound to `127.0.0.1:3080` -4. Subsequent requests to `http://127.0.0.1:3080/` use the cookie for authentication -5. After 30 days, the cookie expires and a new token exchange is required - -**Verification**: -```bash -# Check if the login endpoint is working: -ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8081/dsh-web-login" -# Expected: 303 - -# Mint the 30-day cookie from the login endpoint: -ssh root@192.168.68.15 "pct exec 112 -- rm -f /tmp/dsh.jar" -ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null -w '%{http_code}' http://127.0.0.1:8081/dsh-web-login" -# Expected: 303 - -# Check if the stored cookie authenticates against dsh-web: -ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" -# Expected: 200 ``` +dsh web: http://127.0.0.1:3080/?token= +``` + +The token only bootstraps an authority-bound, HMAC-signed browser cookie with a +30-day lifetime. The signing secret is durable in +`/root/.dsh/.credentials.yaml` (key `client-connection/browser-session`), so a +cookie minted once keeps working across `dsh-web` restarts; the launch token +itself rotates on every restart. + +**Login endpoint (public, Authentik-gated):** +`https://tankodhs.sysloggh.net/dsh-web-login` + +It lives inside the Authentik-gated `:80` server block +(`/etc/nginx/sites-available/dsh`, symlinked from +`/etc/nginx/sites-enabled/dsh`) as `location = /dsh-web-login`, guarded by +`auth_request /outpost.goauthentik.io/auth/nginx`. It proxies to dsh-web with +`Host: tankodhs.sysloggh.net`, so the minted cookie is bound to the public +authority — never to `127.0.0.1:3080`. The token-dependent line is isolated in +the generated include `/etc/dsh-web/nginx-login.conf`: + +``` +proxy_pass http://127.0.0.1:3080/?token=; +``` + +**Token refresh (non-disruptive):** +`/opt/deepseek-harness/capture-dsh-token.sh` (source: +`scripts/capture-dsh-token.sh`) reads the latest launch token from the journal +**of the service's current invocation**, writes `/etc/dsh-web/launch-token` and +regenerates `/etc/dsh-web/nginx-login.conf`, reloading nginx only when the token +changed (`nginx -t` guards the reload, with rollback). It **never stops or +starts `dsh-web`**. It is triggered by the `dsh-web.service` drop-in +`/etc/systemd/system/dsh-web.service.d/20-token-refresh.conf` +(`ExecStartPost=/bin/systemctl --no-block start dsh-web-token.service`) and by +`dsh-web-token.timer` every 2 minutes for reconciliation. + +
Installed systemd wiring (CT 112) + +```ini +# /etc/systemd/system/dsh-web-token.service +[Unit] +Description=Refresh the dsh-web launch token for the nginx login endpoint +After=dsh-web.service +[Service] +Type=oneshot +ExecStart=/opt/deepseek-harness/capture-dsh-token.sh + +# /etc/systemd/system/dsh-web-token.timer +[Unit] +Description=Periodically refresh the dsh-web login token +[Timer] +OnBootSec=90s +OnUnitActiveSec=120s +AccuracySec=10s +Persistent=true +[Install] +WantedBy=timers.target + +# /etc/systemd/system/dsh-web.service.d/20-token-refresh.conf +[Service] +ExecStartPost=/bin/systemctl --no-block start dsh-web-token.service +``` + +
+ +> **Do NOT reintroduce the `:8081` endpoint.** It listened on `0.0.0.0:8081` +> with no `auth_request` and was a full Authentik bypass for anyone on the LAN. +> The script now removes `/etc/nginx/sites-enabled/dsh.token` automatically if +> it ever reappears. + +**Authentication flow:** +1. `GET https://tankodhs.sysloggh.net/dsh-web-login` +2. Unauthenticated → Authentik sign-in; once authenticated the request reaches + dsh-web with `Host: tankodhs.sysloggh.net`. +3. dsh-web accepts the launch token on `GET /`, writes the + `dsh-auth-` cookie (30 days, `HttpOnly`, `SameSite=Strict`) + and returns `303` to `/`. +4. Every later request through `/` presents that cookie; the token is not needed + again until the cookie expires or a new browser is used. + +**Verification** (amdpve vantage): +```bash +# 1. Login endpoint is Authentik-gated: unauthenticated -> 302 (not 200/303). +ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}\n' \ + -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1/dsh-web-login" +# Expected: 302 + +# 2. Legacy :8081 endpoint is gone (connection refused -> 000). +ssh root@192.168.68.15 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \ + -w '%{http_code}\n' http://192.168.68.122:8081/" +# Expected: 000 + +# 3. Backend cookie mint + reuse (exactly what /dsh-web-login proxies to). +TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token") +ssh root@192.168.68.15 "pct exec 112 -- curl -s -D - -o /dev/null \ + -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'" +# Expected: HTTP/1.1 303 + set-cookie: dsh-auth-... (authority tankodhs.sysloggh.net) + +# 4. Token refresh is non-disruptive and idempotent. +ssh root@192.168.68.15 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh" +# Expected: "token unchanged; nginx not reloaded" when nothing changed +``` + +**Restart durability (acceptance):** after `systemctl restart dsh-web`, (a) a +cookie minted before the restart still returns `200` on `/`, and (b) the +refreshed `/etc/dsh-web/nginx-login.conf` carries the new token and mints a +fresh cookie. Both verified live 2026-09-11. ### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14) -- 2.54.0 From b80d3142aaadff5622c46aaad47886958c1d29e5 Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 16:59:55 +0000 Subject: [PATCH 04/11] no-mistakes(review): harden dsh token reload retry, legacy bypass, cookie verification --- scripts/capture-dsh-token.sh | 55 +++++++++++++++++++++++++++--------- zulip-health.prose.md | 28 +++++++++++++++--- 2 files changed, 65 insertions(+), 18 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index 7b49bc8..7a9c1b6 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -19,8 +19,9 @@ # 2. records it in /etc/dsh-web/launch-token, # 3. regenerates the nginx include /etc/dsh-web/nginx-login.conf (the # `proxy_pass ...?token=` line consumed by /dsh-web-login), -# 4. validates with `nginx -t` and reloads ONLY when the token changed, -# rolling the include back if validation fails, +# 4. reloads nginx ONLY when the token differs from the token nginx actually +# loaded (tracked in an applied-state stamp written only after a successful +# reload), rolling the include back on failure so the next run retries, # 5. removes the legacy unauthenticated :8081 endpoint if it ever reappears. # # Idempotent and safe to run at any time (systemd ExecStartPost or timer). @@ -31,6 +32,7 @@ PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" JOURNAL_UNIT="dsh-web.service" TOKEN_FILE="/etc/dsh-web/launch-token" INCLUDE_FILE="/etc/dsh-web/nginx-login.conf" +STAMP_FILE="/etc/dsh-web/nginx-login.conf.applied" SITE_ENABLED="/etc/nginx/sites-enabled/dsh" LEGACY_8081="/etc/nginx/sites-enabled/dsh.token" STASH_DIR="/etc/nginx/sites-available" @@ -44,16 +46,16 @@ die() { printf 'capture-dsh-token: ERROR: %s\n' "$*" >&2; exit 1; } # It bypassed Authentik entirely (listened on 0.0.0.0:8081 with no auth_request) # and must never come back. Stash it rather than delete so it is auditable. if [ -e "$LEGACY_8081" ] || [ -L "$LEGACY_8081" ]; then - STAMP="$(date -u +%Y%m%dT%H%M%SZ)" - STASHED="$STASH_DIR/dsh.token.disabled-$STAMP" + TS="$(date -u +%Y%m%dT%H%M%SZ)" + STASHED="$STASH_DIR/dsh.token.disabled-$TS" mv "$LEGACY_8081" "$STASHED" - if nginx -t >/dev/null 2>&1; then - nginx -s reload - log "removed legacy :8081 endpoint -> $STASHED" - else - mv "$STASHED" "$LEGACY_8081" - die "nginx config test failed after removing $LEGACY_8081; restored it" + if ! nginx -t >/dev/null 2>&1; then + die "nginx config test failed after disabling $LEGACY_8081 (kept disabled at $STASHED). Fix the nginx config and rerun; the legacy :8081 endpoint will NOT be restored." fi + if ! nginx -s reload; then + die "nginx reload failed after disabling $LEGACY_8081 (kept disabled at $STASHED). Fix nginx and rerun; the legacy :8081 endpoint will NOT be restored." + fi + log "removed legacy :8081 endpoint -> $STASHED" fi # ── 1. Read the latest launch token from the RUNNING service ──────────────── @@ -107,9 +109,21 @@ NEW_INCLUDE="$(mktemp "$INCLUDE_FILE.XXXXXX")" printf 'proxy_pass http://127.0.0.1:3080/?token=%s;\n' "$TOKEN" > "$NEW_INCLUDE" chmod 600 "$NEW_INCLUDE" -if [ -f "$INCLUDE_FILE" ] && cmp -s "$NEW_INCLUDE" "$INCLUDE_FILE"; then - rm -f "$NEW_INCLUDE" - log "token unchanged; nginx not reloaded" +# The stamp records the token nginx actually loaded. Comparing against it (not +# the on-disk include) means a failed or interrupted reload is retried on the +# next run instead of being mistaken for success. +APPLIED="" +[ -f "$STAMP_FILE" ] && APPLIED="$(cat "$STAMP_FILE" 2>/dev/null || true)" + +if [ "$APPLIED" = "$TOKEN" ]; then + if [ -f "$INCLUDE_FILE" ] && cmp -s "$NEW_INCLUDE" "$INCLUDE_FILE"; then + rm -f "$NEW_INCLUDE" + log "token unchanged; nginx not reloaded" + exit 0 + fi + mv "$NEW_INCLUDE" "$INCLUDE_FILE" + chmod 600 "$INCLUDE_FILE" + log "include file repaired to match the token nginx already serves" exit 0 fi @@ -132,10 +146,23 @@ if ! nginx -t >/dev/null 2>&1; then fi die "nginx config test failed; previous include restored" fi + +if ! nginx -s reload; then + if [ -n "$RESTORE" ]; then + mv "$RESTORE" "$INCLUDE_FILE" + else + rm -f "$INCLUDE_FILE" + fi + die "nginx reload failed; previous include restored; will retry next run" +fi + if [ -n "$RESTORE" ]; then rm -f "$RESTORE" fi -nginx -s reload +printf '%s\n' "$TOKEN" > "$STAMP_FILE.tmp" +chmod 600 "$STAMP_FILE.tmp" +mv "$STAMP_FILE.tmp" "$STAMP_FILE" + log "token changed; nginx reloaded" log "login endpoint: https://tankodhs.sysloggh.net/dsh-web-login (Authentik-gated)" diff --git a/zulip-health.prose.md b/zulip-health.prose.md index e3392b6..349b439 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -295,7 +295,9 @@ proxy_pass http://127.0.0.1:3080/?token=; `scripts/capture-dsh-token.sh`) reads the latest launch token from the journal **of the service's current invocation**, writes `/etc/dsh-web/launch-token` and regenerates `/etc/dsh-web/nginx-login.conf`, reloading nginx only when the token -changed (`nginx -t` guards the reload, with rollback). It **never stops or +differs from the token nginx actually loaded (`nginx -t` guards the reload, and +the applied-state stamp is written only after a successful `nginx -s reload`, so +a failed or interrupted reload is retried on the next run). It **never stops or starts `dsh-web`**. It is triggered by the `dsh-web.service` drop-in `/etc/systemd/system/dsh-web.service.d/20-token-refresh.conf` (`ExecStartPost=/bin/systemctl --no-block start dsh-web-token.service`) and by @@ -359,20 +361,38 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \ # 3. Backend cookie mint + reuse (exactly what /dsh-web-login proxies to). TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token") -ssh root@192.168.68.15 "pct exec 112 -- curl -s -D - -o /dev/null \ +ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \ -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'" -# Expected: HTTP/1.1 303 + set-cookie: dsh-auth-... (authority tankodhs.sysloggh.net) +ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \ + -w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/" +# Expected: 200 — the minted dsh-auth-... cookie (authority +# tankodhs.sysloggh.net) is replayed on the next request and accepted. # 4. Token refresh is non-disruptive and idempotent. ssh root@192.168.68.15 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh" # Expected: "token unchanged; nginx not reloaded" when nothing changed ``` -**Restart durability (acceptance):** after `systemctl restart dsh-web`, (a) a +**Restart durability (acceptance):** after `systemctl restart dsh-web`, (a) the cookie minted before the restart still returns `200` on `/`, and (b) the refreshed `/etc/dsh-web/nginx-login.conf` carries the new token and mints a fresh cookie. Both verified live 2026-09-11. +```bash +# 5. Cookie survives a dsh-web restart, and the new token mints a new cookie. +ssh root@192.168.68.15 "pct exec 112 -- systemctl restart dsh-web" +ssh root@192.168.68.15 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh" +ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \ + -w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/" +# Expected: 200 — the pre-restart cookie is still accepted. +TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token") +ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh-new.jar -o /dev/null \ + -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'" +ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null \ + -w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/" +# Expected: 200 — the refreshed token minted a fresh cookie. +``` + ### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14) -- 2.54.0 From c66671dbee00fdf86ced030097a448db1a41158d Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 17:07:00 +0000 Subject: [PATCH 05/11] no-mistakes(review): simplify dsh token selection and reload state machine --- scripts/capture-dsh-token.sh | 107 ++++++++++++++++++----------------- zulip-health.prose.md | 21 ++++--- 2 files changed, 70 insertions(+), 58 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index 7a9c1b6..d1ce52c 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -14,14 +14,15 @@ # reference the token of the RUNNING process. # # This script: -# 1. reads the LATEST launch token from the running service's journal — it -# NEVER stops or starts dsh-web, +# 1. selects the launch token the RUNNING service actually accepts — it NEVER +# stops or starts dsh-web, # 2. records it in /etc/dsh-web/launch-token, # 3. regenerates the nginx include /etc/dsh-web/nginx-login.conf (the # `proxy_pass ...?token=` line consumed by /dsh-web-login), -# 4. reloads nginx ONLY when the token differs from the token nginx actually -# loaded (tracked in an applied-state stamp written only after a successful -# reload), rolling the include back on failure so the next run retries, +# 4. reloads nginx ONLY when the on-disk include differs from the generated +# one or the applied-state stamp does not match the token (the stamp is +# written only after a successful reload), rolling the include back on +# failure so the next run retries, # 5. removes the legacy unauthenticated :8081 endpoint if it ever reappears. # # Idempotent and safe to run at any time (systemd ExecStartPost or timer). @@ -36,13 +37,21 @@ STAMP_FILE="/etc/dsh-web/nginx-login.conf.applied" SITE_ENABLED="/etc/nginx/sites-enabled/dsh" LEGACY_8081="/etc/nginx/sites-enabled/dsh.token" STASH_DIR="/etc/nginx/sites-available" +LOCK_FILE="/run/capture-dsh-token.lock" +LOGIN_HOST="tankodhs.sysloggh.net" +LOGIN_UPSTREAM="http://127.0.0.1:3080" +TOKEN_WAIT=120 log() { printf 'capture-dsh-token: %s\n' "$*" >&2; } die() { printf 'capture-dsh-token: ERROR: %s\n' "$*" >&2; exit 1; } [ "$(id -u)" -eq 0 ] || die "must run as root" -# ── 0. Remove the legacy unauthenticated :8081 endpoint, if present ───────── +# ── 0. Serialize runs so timer/ExecStartPost/manual runs cannot interleave ── +exec 9>"$LOCK_FILE" +flock -n 9 || { log "another capture-dsh-token run holds $LOCK_FILE; exiting"; exit 0; } + +# ── 1. Remove the legacy unauthenticated :8081 endpoint, if present ───────── # It bypassed Authentik entirely (listened on 0.0.0.0:8081 with no auth_request) # and must never come back. Stash it rather than delete so it is auditable. if [ -e "$LEGACY_8081" ] || [ -L "$LEGACY_8081" ]; then @@ -58,72 +67,68 @@ if [ -e "$LEGACY_8081" ] || [ -L "$LEGACY_8081" ]; then log "removed legacy :8081 endpoint -> $STASHED" fi -# ── 1. Read the latest launch token from the RUNNING service ──────────────── -# Scope the journal to the service's CURRENT invocation. While a restarted -# process is still booting (~25s before it prints the banner), the newest token -# in the journal still belongs to the PREVIOUS process; without this filter an -# ExecStartPost run would silently keep the stale token. Never stop/start dsh-web. -INVOCATION="$(systemctl show -p InvocationID --value "$JOURNAL_UNIT" 2>/dev/null || true)" -JOURNAL_ARGS=(-u "$JOURNAL_UNIT") -if [ -n "$INVOCATION" ] && [ "$INVOCATION" != "n/a" ]; then - JOURNAL_ARGS+=("_SYSTEMD_INVOCATION_ID=$INVOCATION") -else - log "WARNING: no invocation id for $JOURNAL_UNIT; using latest journal token" -fi - -extract_token() { - grep -oE 'dsh web: https?://[^[:space:]]+[?&]token=[^[:space:]]+' \ - | tail -n1 | sed -E 's/.*[?&]token=//' || true +# ── 2. Select the token the RUNNING service actually accepts ──────────────── +# Functionally verify each journal candidate against the local dsh-web using the +# public authority, exactly as the /dsh-web-login proxy does. A token from a +# previous invocation is rejected (never 303) and can never be selected, so no +# systemd invocation scoping or newest-overall fallback is needed. Candidates +# are tried newest-first and re-read from the journal each pass until one is +# accepted or the wait elapses. +collect_tokens() { + journalctl -u "$JOURNAL_UNIT" --no-pager -o cat 2>/dev/null \ + | grep -oE 'dsh web: https?://[^[:space:]]+[?&]token=[^[:space:]]+' \ + | sed -E 's/.*[?&]token=//' \ + | grep -E '^[A-Za-z0-9._~+/=:@-]+$' \ + | tac | awk '!seen[$0]++' || true } TOKEN="" -for _ in $(seq 1 60); do - TOKEN="$(journalctl "${JOURNAL_ARGS[@]}" --no-pager -o cat 2>/dev/null | extract_token)" +TRIED=" " +DEADLINE=$((SECONDS + TOKEN_WAIT)) +while [ -z "$TOKEN" ] && [ "$SECONDS" -lt "$DEADLINE" ]; do + for cand in $(collect_tokens); do + case "$TRIED" in *" $cand "*) continue ;; esac + TRIED="$TRIED$cand " + code="$(curl -s -o /dev/null --max-time 5 -w '%{http_code}' \ + -H "Host: $LOGIN_HOST" "$LOGIN_UPSTREAM/?token=$cand" || true)" + if [ "$code" = "303" ]; then + TOKEN="$cand" + break + fi + done [ -n "$TOKEN" ] && break - sleep 1 + sleep 2 done -# Fallback: the current invocation's start banner may have been rotated out of -# the journal; the newest matching line overall is then the best available. if [ -z "$TOKEN" ]; then - log "WARNING: no token for the current invocation; falling back to newest journal token" - TOKEN="$(journalctl -u "$JOURNAL_UNIT" --no-pager -o cat 2>/dev/null | extract_token)" + log "no dsh-web launch token accepted within ${TOKEN_WAIT}s; leaving the include untouched for the next run" + exit 0 fi -[ -n "$TOKEN" ] || die "no launch token found in the $JOURNAL_UNIT journal" -# The token must be safe to embed in a URI and in the nginx config. -printf '%s' "$TOKEN" | grep -qE '^[A-Za-z0-9._~+/=:@-]+$' \ - || die "captured token contains unsupported characters" - -# ── 2. Record the token (atomic, private) ────────────────────────────────── +# ── 3. Record the token (atomic, private) ────────────────────────────────── mkdir -p "$(dirname "$TOKEN_FILE")" if ! printf '%s\n' "$TOKEN" | cmp -s - "$TOKEN_FILE" 2>/dev/null; then printf '%s\n' "$TOKEN" > "$TOKEN_FILE.tmp" chmod 600 "$TOKEN_FILE.tmp" mv "$TOKEN_FILE.tmp" "$TOKEN_FILE" - log "recorded new launch token in $TOKEN_FILE" + log "recorded live launch token in $TOKEN_FILE" fi -# ── 3. Regenerate the nginx login include (reload only when it changes) ──── +# ── 4. Regenerate the nginx login include (reload only when it changes) ──── NEW_INCLUDE="$(mktemp "$INCLUDE_FILE.XXXXXX")" -printf 'proxy_pass http://127.0.0.1:3080/?token=%s;\n' "$TOKEN" > "$NEW_INCLUDE" +printf 'proxy_pass %s/?token=%s;\n' "$LOGIN_UPSTREAM" "$TOKEN" > "$NEW_INCLUDE" chmod 600 "$NEW_INCLUDE" -# The stamp records the token nginx actually loaded. Comparing against it (not -# the on-disk include) means a failed or interrupted reload is retried on the -# next run instead of being mistaken for success. +# The stamp records the token nginx actually loaded. It is written only after a +# successful reload, so the early exit is safe only when both the stamp and the +# on-disk include agree with the live token; anything else falls through to the +# reload path so the include can never silently diverge from what nginx serves. APPLIED="" [ -f "$STAMP_FILE" ] && APPLIED="$(cat "$STAMP_FILE" 2>/dev/null || true)" -if [ "$APPLIED" = "$TOKEN" ]; then - if [ -f "$INCLUDE_FILE" ] && cmp -s "$NEW_INCLUDE" "$INCLUDE_FILE"; then - rm -f "$NEW_INCLUDE" - log "token unchanged; nginx not reloaded" - exit 0 - fi - mv "$NEW_INCLUDE" "$INCLUDE_FILE" - chmod 600 "$INCLUDE_FILE" - log "include file repaired to match the token nginx already serves" +if [ "$APPLIED" = "$TOKEN" ] && [ -f "$INCLUDE_FILE" ] && cmp -s "$NEW_INCLUDE" "$INCLUDE_FILE"; then + rm -f "$NEW_INCLUDE" + log "token unchanged; nginx not reloaded" exit 0 fi @@ -165,4 +170,4 @@ chmod 600 "$STAMP_FILE.tmp" mv "$STAMP_FILE.tmp" "$STAMP_FILE" log "token changed; nginx reloaded" -log "login endpoint: https://tankodhs.sysloggh.net/dsh-web-login (Authentik-gated)" +log "login endpoint: https://$LOGIN_HOST/dsh-web-login (Authentik-gated)" diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 349b439..0de8755 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -292,13 +292,19 @@ proxy_pass http://127.0.0.1:3080/?token=; **Token refresh (non-disruptive):** `/opt/deepseek-harness/capture-dsh-token.sh` (source: -`scripts/capture-dsh-token.sh`) reads the latest launch token from the journal -**of the service's current invocation**, writes `/etc/dsh-web/launch-token` and -regenerates `/etc/dsh-web/nginx-login.conf`, reloading nginx only when the token -differs from the token nginx actually loaded (`nginx -t` guards the reload, and -the applied-state stamp is written only after a successful `nginx -s reload`, so -a failed or interrupted reload is retried on the next run). It **never stops or -starts `dsh-web`**. It is triggered by the `dsh-web.service` drop-in +`scripts/capture-dsh-token.sh`) selects the live launch token by functionally +verifying journal candidates against dsh-web with `Host: tankodhs.sysloggh.net` +and using the first one the running process accepts with `303`; a stale token +from a previous invocation is rejected and can never be selected. It waits up to +120s for a restarted process to accept a token, and if none is accepted it +leaves the include untouched and exits `0` so the timer retries. It writes +`/etc/dsh-web/launch-token` and regenerates `/etc/dsh-web/nginx-login.conf`, +reloading nginx only when the on-disk include differs from the generated one or +the applied-state stamp does not match the token (`nginx -t` guards the reload, +and the stamp is written only after a successful `nginx -s reload`, so a failed +or interrupted reload is retried on the next run). Runs are serialized with +`flock` on `/run/capture-dsh-token.lock`. It **never stops or starts `dsh-web`**. +It is triggered by the `dsh-web.service` drop-in `/etc/systemd/system/dsh-web.service.d/20-token-refresh.conf` (`ExecStartPost=/bin/systemctl --no-block start dsh-web-token.service`) and by `dsh-web-token.timer` every 2 minutes for reconciliation. @@ -312,6 +318,7 @@ Description=Refresh the dsh-web launch token for the nginx login endpoint After=dsh-web.service [Service] Type=oneshot +TimeoutStartSec=180 ExecStart=/opt/deepseek-harness/capture-dsh-token.sh # /etc/systemd/system/dsh-web-token.timer -- 2.54.0 From 288f74cf8405fb95dc6afc55ede5b3d454bd647f Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 17:12:11 +0000 Subject: [PATCH 06/11] no-mistakes(review): reprobe dsh tokens; persist pending nginx reload on failure --- scripts/capture-dsh-token.sh | 19 ++++++++++++------- zulip-health.prose.md | 11 ++++++++--- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index d1ce52c..89c609c 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -34,6 +34,7 @@ JOURNAL_UNIT="dsh-web.service" TOKEN_FILE="/etc/dsh-web/launch-token" INCLUDE_FILE="/etc/dsh-web/nginx-login.conf" STAMP_FILE="/etc/dsh-web/nginx-login.conf.applied" +PENDING_FILE="/etc/dsh-web/nginx-reload.pending" SITE_ENABLED="/etc/nginx/sites-enabled/dsh" LEGACY_8081="/etc/nginx/sites-enabled/dsh.token" STASH_DIR="/etc/nginx/sites-available" @@ -50,6 +51,7 @@ die() { printf 'capture-dsh-token: ERROR: %s\n' "$*" >&2; exit 1; } # ── 0. Serialize runs so timer/ExecStartPost/manual runs cannot interleave ── exec 9>"$LOCK_FILE" flock -n 9 || { log "another capture-dsh-token run holds $LOCK_FILE; exiting"; exit 0; } +mkdir -p "$(dirname "$PENDING_FILE")" # ── 1. Remove the legacy unauthenticated :8081 endpoint, if present ───────── # It bypassed Authentik entirely (listened on 0.0.0.0:8081 with no auth_request) @@ -58,12 +60,14 @@ if [ -e "$LEGACY_8081" ] || [ -L "$LEGACY_8081" ]; then TS="$(date -u +%Y%m%dT%H%M%SZ)" STASHED="$STASH_DIR/dsh.token.disabled-$TS" mv "$LEGACY_8081" "$STASHED" + touch "$PENDING_FILE" if ! nginx -t >/dev/null 2>&1; then - die "nginx config test failed after disabling $LEGACY_8081 (kept disabled at $STASHED). Fix the nginx config and rerun; the legacy :8081 endpoint will NOT be restored." + die "nginx config test failed after disabling $LEGACY_8081 (kept disabled at $STASHED); a pending reload is recorded so running nginx is reloaded once the config is fixed. The legacy :8081 endpoint will NOT be restored." fi if ! nginx -s reload; then - die "nginx reload failed after disabling $LEGACY_8081 (kept disabled at $STASHED). Fix nginx and rerun; the legacy :8081 endpoint will NOT be restored." + die "nginx reload failed after disabling $LEGACY_8081 (kept disabled at $STASHED); a pending reload is recorded so running nginx is reloaded on the next run. The legacy :8081 endpoint will NOT be restored." fi + rm -f "$PENDING_FILE" log "removed legacy :8081 endpoint -> $STASHED" fi @@ -83,12 +87,9 @@ collect_tokens() { } TOKEN="" -TRIED=" " DEADLINE=$((SECONDS + TOKEN_WAIT)) while [ -z "$TOKEN" ] && [ "$SECONDS" -lt "$DEADLINE" ]; do for cand in $(collect_tokens); do - case "$TRIED" in *" $cand "*) continue ;; esac - TRIED="$TRIED$cand " code="$(curl -s -o /dev/null --max-time 5 -w '%{http_code}' \ -H "Host: $LOGIN_HOST" "$LOGIN_UPSTREAM/?token=$cand" || true)" if [ "$code" = "303" ]; then @@ -126,7 +127,8 @@ chmod 600 "$NEW_INCLUDE" APPLIED="" [ -f "$STAMP_FILE" ] && APPLIED="$(cat "$STAMP_FILE" 2>/dev/null || true)" -if [ "$APPLIED" = "$TOKEN" ] && [ -f "$INCLUDE_FILE" ] && cmp -s "$NEW_INCLUDE" "$INCLUDE_FILE"; then +if [ "$APPLIED" = "$TOKEN" ] && [ -f "$INCLUDE_FILE" ] && cmp -s "$NEW_INCLUDE" "$INCLUDE_FILE" \ + && [ ! -e "$PENDING_FILE" ]; then rm -f "$NEW_INCLUDE" log "token unchanged; nginx not reloaded" exit 0 @@ -158,13 +160,16 @@ if ! nginx -s reload; then else rm -f "$INCLUDE_FILE" fi - die "nginx reload failed; previous include restored; will retry next run" + touch "$PENDING_FILE" + die "nginx reload failed; previous include restored; a pending reload is recorded so the next run retries" fi if [ -n "$RESTORE" ]; then rm -f "$RESTORE" fi +rm -f "$PENDING_FILE" + printf '%s\n' "$TOKEN" > "$STAMP_FILE.tmp" chmod 600 "$STAMP_FILE.tmp" mv "$STAMP_FILE.tmp" "$STAMP_FILE" diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 0de8755..e4ee53c 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -296,13 +296,18 @@ proxy_pass http://127.0.0.1:3080/?token=; verifying journal candidates against dsh-web with `Host: tankodhs.sysloggh.net` and using the first one the running process accepts with `303`; a stale token from a previous invocation is rejected and can never be selected. It waits up to -120s for a restarted process to accept a token, and if none is accepted it -leaves the include untouched and exits `0` so the timer retries. It writes +120s for a restarted process to accept a token; every distinct candidate is +re-probed on each pass, so a token that briefly returns `000` while the service +is still starting is not disqualified. If none is accepted it leaves the include +untouched and exits `0` so the timer retries. It writes `/etc/dsh-web/launch-token` and regenerates `/etc/dsh-web/nginx-login.conf`, reloading nginx only when the on-disk include differs from the generated one or the applied-state stamp does not match the token (`nginx -t` guards the reload, and the stamp is written only after a successful `nginx -s reload`, so a failed -or interrupted reload is retried on the next run). Runs are serialized with +or interrupted reload is retried on the next run). Any failed reload records a +pending-reload marker under `/etc/dsh-web/` that forces the next run through the +reload path even when the token is unchanged, so a disabled legacy `:8081` file +can never leave the running nginx unreloaded. Runs are serialized with `flock` on `/run/capture-dsh-token.lock`. It **never stops or starts `dsh-web`**. It is triggered by the `dsh-web.service` drop-in `/etc/systemd/system/dsh-web.service.d/20-token-refresh.conf` -- 2.54.0 From aeb66ea22d7328617003dd3360d73cb9f4bfb79c Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 17:17:26 +0000 Subject: [PATCH 07/11] no-mistakes(review): fix pending-reload path, token modes, restart check --- scripts/capture-dsh-token.sh | 19 +++++++++++++++++++ zulip-health.prose.md | 22 ++++++++++++++++------ 2 files changed, 35 insertions(+), 6 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index 89c609c..273fe6d 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -71,6 +71,21 @@ if [ -e "$LEGACY_8081" ] || [ -L "$LEGACY_8081" ]; then log "removed legacy :8081 endpoint -> $STASHED" fi +# ── 1b. Honor a recorded pending reload regardless of token selection ─────── +# A failed reload leaves PENDING_FILE set so a stashed legacy :8081 file can +# never remain loaded in the running nginx while dsh-web is down or not yet +# answering. Reconcile it before the token wait. +if [ -e "$PENDING_FILE" ]; then + if ! nginx -t >/dev/null 2>&1; then + die "pending nginx reload recorded but 'nginx -t' fails; fix the config and rerun" + fi + if ! nginx -s reload; then + die "pending nginx reload recorded but 'nginx -s reload' failed; will retry next run" + fi + rm -f "$PENDING_FILE" + log "completed pending nginx reload" +fi + # ── 2. Select the token the RUNNING service actually accepts ──────────────── # Functionally verify each journal candidate against the local dsh-web using the # public authority, exactly as the /dsh-web-login proxy does. A token from a @@ -114,6 +129,7 @@ if ! printf '%s\n' "$TOKEN" | cmp -s - "$TOKEN_FILE" 2>/dev/null; then mv "$TOKEN_FILE.tmp" "$TOKEN_FILE" log "recorded live launch token in $TOKEN_FILE" fi +chmod 600 "$TOKEN_FILE" # ── 4. Regenerate the nginx login include (reload only when it changes) ──── NEW_INCLUDE="$(mktemp "$INCLUDE_FILE.XXXXXX")" @@ -126,6 +142,8 @@ chmod 600 "$NEW_INCLUDE" # reload path so the include can never silently diverge from what nginx serves. APPLIED="" [ -f "$STAMP_FILE" ] && APPLIED="$(cat "$STAMP_FILE" 2>/dev/null || true)" +[ -f "$INCLUDE_FILE" ] && chmod 600 "$INCLUDE_FILE" +[ -f "$STAMP_FILE" ] && chmod 600 "$STAMP_FILE" if [ "$APPLIED" = "$TOKEN" ] && [ -f "$INCLUDE_FILE" ] && cmp -s "$NEW_INCLUDE" "$INCLUDE_FILE" \ && [ ! -e "$PENDING_FILE" ]; then @@ -140,6 +158,7 @@ RESTORE="" if [ -f "$INCLUDE_FILE" ]; then RESTORE="$(mktemp "$INCLUDE_FILE.bak.XXXXXX")" cp -p "$INCLUDE_FILE" "$RESTORE" + chmod 600 "$RESTORE" fi mv "$NEW_INCLUDE" "$INCLUDE_FILE" diff --git a/zulip-health.prose.md b/zulip-health.prose.md index e4ee53c..ac31867 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -305,10 +305,11 @@ reloading nginx only when the on-disk include differs from the generated one or the applied-state stamp does not match the token (`nginx -t` guards the reload, and the stamp is written only after a successful `nginx -s reload`, so a failed or interrupted reload is retried on the next run). Any failed reload records a -pending-reload marker under `/etc/dsh-web/` that forces the next run through the -reload path even when the token is unchanged, so a disabled legacy `:8081` file -can never leave the running nginx unreloaded. Runs are serialized with -`flock` on `/run/capture-dsh-token.lock`. It **never stops or starts `dsh-web`**. +pending-reload marker under `/etc/dsh-web/`; the next run honors it before the +token wait, reloading nginx and clearing the marker regardless of token state, +so a disabled legacy `:8081` file can never leave the running nginx unreloaded. +Runs are serialized with `flock` on `/run/capture-dsh-token.lock`. It **never +stops or starts `dsh-web`**. It is triggered by the `dsh-web.service` drop-in `/etc/systemd/system/dsh-web.service.d/20-token-refresh.conf` (`ExecStartPost=/bin/systemctl --no-block start dsh-web-token.service`) and by @@ -393,11 +394,20 @@ fresh cookie. Both verified live 2026-09-11. ```bash # 5. Cookie survives a dsh-web restart, and the new token mints a new cookie. ssh root@192.168.68.15 "pct exec 112 -- systemctl restart dsh-web" -ssh root@192.168.68.15 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh" ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \ -w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/" # Expected: 200 — the pre-restart cookie is still accepted. -TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token") +# The restart's ExecStartPost (or the 2-minute timer) refreshes the include. A +# manual run may no-op on the flock, so poll until the include carries a token +# the running process accepts (bounded wait) before the mint+reuse check. +for i in $(seq 1 60); do + TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'") + CODE=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \ + -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'") + [ "$CODE" = "303" ] && break + sleep 2 +done +# Expected: 303 — the include now holds the token the running process accepts. ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh-new.jar -o /dev/null \ -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'" ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null \ -- 2.54.0 From b9adf353ee52d8e27121c9166994569286dad2f7 Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 17:24:12 +0000 Subject: [PATCH 08/11] no-mistakes(review): guard missing include, non-fatal pending reload, chmod stash --- scripts/capture-dsh-token.sh | 30 ++++++++++++++++++++++++------ zulip-health.prose.md | 19 +++++++++++++++---- 2 files changed, 39 insertions(+), 10 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index 273fe6d..bcce34f 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -53,6 +53,22 @@ exec 9>"$LOCK_FILE" flock -n 9 || { log "another capture-dsh-token run holds $LOCK_FILE; exiting"; exit 0; } mkdir -p "$(dirname "$PENDING_FILE")" +# ── 0b. Guarantee the generated include exists before any `nginx -t` ────── +# The :80 site includes /etc/dsh-web/nginx-login.conf by literal path, so a +# missing include makes every `nginx -t` fail and can wedge recovery. Seed it +# from the last known token (or a placeholder); step 4 replaces it. +if [ ! -f "$INCLUDE_FILE" ]; then + SEED="placeholder" + if [ -f "$TOKEN_FILE" ]; then + SEED="$(cat "$TOKEN_FILE" 2>/dev/null || true)" + [ -n "$SEED" ] || SEED="placeholder" + fi + printf '%s' "$SEED" | grep -qE '^[A-Za-z0-9._~+/=:@-]+$' || SEED="placeholder" + printf 'proxy_pass %s/?token=%s;\n' "$LOGIN_UPSTREAM" "$SEED" > "$INCLUDE_FILE" + chmod 600 "$INCLUDE_FILE" + log "created missing $INCLUDE_FILE" +fi + # ── 1. Remove the legacy unauthenticated :8081 endpoint, if present ───────── # It bypassed Authentik entirely (listened on 0.0.0.0:8081 with no auth_request) # and must never come back. Stash it rather than delete so it is auditable. @@ -60,6 +76,7 @@ if [ -e "$LEGACY_8081" ] || [ -L "$LEGACY_8081" ]; then TS="$(date -u +%Y%m%dT%H%M%SZ)" STASHED="$STASH_DIR/dsh.token.disabled-$TS" mv "$LEGACY_8081" "$STASHED" + chmod 600 "$STASHED" 2>/dev/null || true touch "$PENDING_FILE" if ! nginx -t >/dev/null 2>&1; then die "nginx config test failed after disabling $LEGACY_8081 (kept disabled at $STASHED); a pending reload is recorded so running nginx is reloaded once the config is fixed. The legacy :8081 endpoint will NOT be restored." @@ -77,13 +94,13 @@ fi # answering. Reconcile it before the token wait. if [ -e "$PENDING_FILE" ]; then if ! nginx -t >/dev/null 2>&1; then - die "pending nginx reload recorded but 'nginx -t' fails; fix the config and rerun" + log "WARNING: pending nginx reload recorded but 'nginx -t' fails; continuing so the include can be regenerated; will retry next run" + elif ! nginx -s reload; then + log "WARNING: pending nginx reload recorded but 'nginx -s reload' failed; will retry next run" + else + rm -f "$PENDING_FILE" + log "completed pending nginx reload" fi - if ! nginx -s reload; then - die "pending nginx reload recorded but 'nginx -s reload' failed; will retry next run" - fi - rm -f "$PENDING_FILE" - log "completed pending nginx reload" fi # ── 2. Select the token the RUNNING service actually accepts ──────────────── @@ -118,6 +135,7 @@ done if [ -z "$TOKEN" ]; then log "no dsh-web launch token accepted within ${TOKEN_WAIT}s; leaving the include untouched for the next run" + [ -e "$PENDING_FILE" ] && die "pending nginx reload could not be completed; will retry next run" exit 0 fi diff --git a/zulip-health.prose.md b/zulip-health.prose.md index ac31867..14a3666 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -299,15 +299,18 @@ from a previous invocation is rejected and can never be selected. It waits up to 120s for a restarted process to accept a token; every distinct candidate is re-probed on each pass, so a token that briefly returns `000` while the service is still starting is not disqualified. If none is accepted it leaves the include -untouched and exits `0` so the timer retries. It writes +untouched and exits so the timer retries (exiting non-zero when a pending reload +is still outstanding). It writes `/etc/dsh-web/launch-token` and regenerates `/etc/dsh-web/nginx-login.conf`, reloading nginx only when the on-disk include differs from the generated one or the applied-state stamp does not match the token (`nginx -t` guards the reload, and the stamp is written only after a successful `nginx -s reload`, so a failed or interrupted reload is retried on the next run). Any failed reload records a -pending-reload marker under `/etc/dsh-web/`; the next run honors it before the -token wait, reloading nginx and clearing the marker regardless of token state, -so a disabled legacy `:8081` file can never leave the running nginx unreloaded. +pending-reload marker under `/etc/dsh-web/`; the next run attempts the reload +before the token wait, independent of token state, and clears the marker only +once the reload succeeds, so a disabled legacy `:8081` file can never leave the +running nginx unreloaded. The generated include is recreated before any +`nginx -t` if it is missing, so a failed run cannot wedge recovery. Runs are serialized with `flock` on `/run/capture-dsh-token.lock`. It **never stops or starts `dsh-web`**. It is triggered by the `dsh-web.service` drop-in @@ -394,6 +397,14 @@ fresh cookie. Both verified live 2026-09-11. ```bash # 5. Cookie survives a dsh-web restart, and the new token mints a new cookie. ssh root@192.168.68.15 "pct exec 112 -- systemctl restart dsh-web" +# dsh-web is Type=simple: restart returns before :3080 is listening. Bounded-poll +# until the socket answers (any status but 000) before asserting the cookie. +for i in $(seq 1 60); do + UP=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \ + -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/") + [ "$UP" != "000" ] && break + sleep 2 +done ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \ -w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/" # Expected: 200 — the pre-restart cookie is still accepted. -- 2.54.0 From 55f1208eb89bcbb1ff501c998d4a3927779c0313 Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 17:30:33 +0000 Subject: [PATCH 09/11] no-mistakes(review): scope dsh token to invocation; log nginx diagnostics --- scripts/capture-dsh-token.sh | 41 +++++++++++++++++++++++------------- zulip-health.prose.md | 15 +++++++------ 2 files changed, 35 insertions(+), 21 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index bcce34f..342275a 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -14,8 +14,8 @@ # reference the token of the RUNNING process. # # This script: -# 1. selects the launch token the RUNNING service actually accepts — it NEVER -# stops or starts dsh-web, +# 1. selects the launch token the RUNNING service actually accepts from the +# current systemd invocation — it NEVER stops or starts dsh-web, # 2. records it in /etc/dsh-web/launch-token, # 3. regenerates the nginx include /etc/dsh-web/nginx-login.conf (the # `proxy_pass ...?token=` line consumed by /dsh-web-login), @@ -78,8 +78,8 @@ if [ -e "$LEGACY_8081" ] || [ -L "$LEGACY_8081" ]; then mv "$LEGACY_8081" "$STASHED" chmod 600 "$STASHED" 2>/dev/null || true touch "$PENDING_FILE" - if ! nginx -t >/dev/null 2>&1; then - die "nginx config test failed after disabling $LEGACY_8081 (kept disabled at $STASHED); a pending reload is recorded so running nginx is reloaded once the config is fixed. The legacy :8081 endpoint will NOT be restored." + if ! NGINX_TEST_OUT="$(nginx -t 2>&1)"; then + die "nginx config test failed after disabling $LEGACY_8081 (kept disabled at $STASHED): $NGINX_TEST_OUT; a pending reload is recorded so running nginx is reloaded once the config is fixed. The legacy :8081 endpoint will NOT be restored." fi if ! nginx -s reload; then die "nginx reload failed after disabling $LEGACY_8081 (kept disabled at $STASHED); a pending reload is recorded so running nginx is reloaded on the next run. The legacy :8081 endpoint will NOT be restored." @@ -93,8 +93,8 @@ fi # never remain loaded in the running nginx while dsh-web is down or not yet # answering. Reconcile it before the token wait. if [ -e "$PENDING_FILE" ]; then - if ! nginx -t >/dev/null 2>&1; then - log "WARNING: pending nginx reload recorded but 'nginx -t' fails; continuing so the include can be regenerated; will retry next run" + if ! NGINX_TEST_OUT="$(nginx -t 2>&1)"; then + log "WARNING: pending nginx reload recorded but 'nginx -t' fails: $NGINX_TEST_OUT; continuing so the include can be regenerated; will retry next run" elif ! nginx -s reload; then log "WARNING: pending nginx reload recorded but 'nginx -s reload' failed; will retry next run" else @@ -104,14 +104,25 @@ if [ -e "$PENDING_FILE" ]; then fi # ── 2. Select the token the RUNNING service actually accepts ──────────────── -# Functionally verify each journal candidate against the local dsh-web using the -# public authority, exactly as the /dsh-web-login proxy does. A token from a -# previous invocation is rejected (never 303) and can never be selected, so no -# systemd invocation scoping or newest-overall fallback is needed. Candidates -# are tried newest-first and re-read from the journal each pass until one is +# Read candidates ONLY from the service's current systemd invocation so a +# restarted process's stale token is never considered while its new startup +# banner is still pending; there is no whole-journal or cross-invocation +# fallback. Each candidate is then functionally verified against the local +# dsh-web using the public authority, exactly as the /dsh-web-login proxy does, +# and the first that answers 303 is the live token. Candidates are re-probed +# newest-first on each pass (connection failures stay eligible) until one is # accepted or the wait elapses. +INVOCATION="$(systemctl show -p InvocationID --value "$JOURNAL_UNIT" 2>/dev/null || true)" +JOURNAL_ARGS=(-u "$JOURNAL_UNIT") +if [ -n "$INVOCATION" ] && [ "$INVOCATION" != "n/a" ]; then + JOURNAL_ARGS+=("_SYSTEMD_INVOCATION_ID=$INVOCATION") +else + log "WARNING: no invocation id for $JOURNAL_UNIT; no token can be selected this run" +fi + collect_tokens() { - journalctl -u "$JOURNAL_UNIT" --no-pager -o cat 2>/dev/null \ + [ "${#JOURNAL_ARGS[@]}" -ge 3 ] || return 0 + journalctl "${JOURNAL_ARGS[@]}" --no-pager -o cat 2>/dev/null \ | grep -oE 'dsh web: https?://[^[:space:]]+[?&]token=[^[:space:]]+' \ | sed -E 's/.*[?&]token=//' \ | grep -E '^[A-Za-z0-9._~+/=:@-]+$' \ @@ -134,7 +145,7 @@ while [ -z "$TOKEN" ] && [ "$SECONDS" -lt "$DEADLINE" ]; do done if [ -z "$TOKEN" ]; then - log "no dsh-web launch token accepted within ${TOKEN_WAIT}s; leaving the include untouched for the next run" + log "no accepted launch token in the current invocation within ${TOKEN_WAIT}s; leaving the include untouched for the next run" [ -e "$PENDING_FILE" ] && die "pending nginx reload could not be completed; will retry next run" exit 0 fi @@ -182,13 +193,13 @@ fi mv "$NEW_INCLUDE" "$INCLUDE_FILE" chmod 600 "$INCLUDE_FILE" -if ! nginx -t >/dev/null 2>&1; then +if ! NGINX_TEST_OUT="$(nginx -t 2>&1)"; then if [ -n "$RESTORE" ]; then mv "$RESTORE" "$INCLUDE_FILE" else rm -f "$INCLUDE_FILE" fi - die "nginx config test failed; previous include restored" + die "nginx config test failed: $NGINX_TEST_OUT; previous include restored" fi if ! nginx -s reload; then diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 14a3666..7b8e0e6 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -292,12 +292,15 @@ proxy_pass http://127.0.0.1:3080/?token=; **Token refresh (non-disruptive):** `/opt/deepseek-harness/capture-dsh-token.sh` (source: -`scripts/capture-dsh-token.sh`) selects the live launch token by functionally -verifying journal candidates against dsh-web with `Host: tankodhs.sysloggh.net` -and using the first one the running process accepts with `303`; a stale token -from a previous invocation is rejected and can never be selected. It waits up to -120s for a restarted process to accept a token; every distinct candidate is -re-probed on each pass, so a token that briefly returns `000` while the service +`scripts/capture-dsh-token.sh`) reads candidate launch tokens from the journal +**scoped to the service's current systemd invocation** +(`systemctl show -p InvocationID` + `_SYSTEMD_INVOCATION_ID=`), so a restarted +process's stale token is never considered while its new startup banner is still +pending; there is no whole-journal or cross-invocation fallback. Each candidate +is then functionally verified against dsh-web with `Host: tankodhs.sysloggh.net`, +using the first the running process accepts with `303`. It waits up to 120s for +a restarted process to accept a token and re-probes every current-invocation +candidate on each pass, so a token that briefly returns `000` while the service is still starting is not disqualified. If none is accepted it leaves the include untouched and exits so the timer retries (exiting non-zero when a pending reload is still outstanding). It writes -- 2.54.0 From e97145c88f3355065cb904517cf3165febceeabe Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 17:34:53 +0000 Subject: [PATCH 10/11] no-mistakes(review): re-sample systemd invocation each pass during token wait --- scripts/capture-dsh-token.sh | 41 ++++++++++++++++++------------------ zulip-health.prose.md | 8 ++++--- 2 files changed, 26 insertions(+), 23 deletions(-) diff --git a/scripts/capture-dsh-token.sh b/scripts/capture-dsh-token.sh index 342275a..4d7fe59 100755 --- a/scripts/capture-dsh-token.sh +++ b/scripts/capture-dsh-token.sh @@ -104,25 +104,16 @@ if [ -e "$PENDING_FILE" ]; then fi # ── 2. Select the token the RUNNING service actually accepts ──────────────── -# Read candidates ONLY from the service's current systemd invocation so a -# restarted process's stale token is never considered while its new startup -# banner is still pending; there is no whole-journal or cross-invocation -# fallback. Each candidate is then functionally verified against the local -# dsh-web using the public authority, exactly as the /dsh-web-login proxy does, -# and the first that answers 303 is the live token. Candidates are re-probed -# newest-first on each pass (connection failures stay eligible) until one is -# accepted or the wait elapses. -INVOCATION="$(systemctl show -p InvocationID --value "$JOURNAL_UNIT" 2>/dev/null || true)" -JOURNAL_ARGS=(-u "$JOURNAL_UNIT") -if [ -n "$INVOCATION" ] && [ "$INVOCATION" != "n/a" ]; then - JOURNAL_ARGS+=("_SYSTEMD_INVOCATION_ID=$INVOCATION") -else - log "WARNING: no invocation id for $JOURNAL_UNIT; no token can be selected this run" -fi - -collect_tokens() { - [ "${#JOURNAL_ARGS[@]}" -ge 3 ] || return 0 - journalctl "${JOURNAL_ARGS[@]}" --no-pager -o cat 2>/dev/null \ +# Re-sample the service's CURRENT systemd invocation on every pass and read +# candidates only from it, so a restart that lands during the wait immediately +# switches to the new invocation; there is no whole-journal or cross-invocation +# fallback, and an empty/unknown invocation just waits. Each candidate is then +# functionally verified against the local dsh-web using the public authority, +# exactly as the /dsh-web-login proxy does, and the first that answers 303 is +# the live token. Candidates are re-probed newest-first on each pass (connection +# failures stay eligible) until one is accepted or the wait elapses. +journal_tokens() { + journalctl -u "$JOURNAL_UNIT" "_SYSTEMD_INVOCATION_ID=$1" --no-pager -o cat 2>/dev/null \ | grep -oE 'dsh web: https?://[^[:space:]]+[?&]token=[^[:space:]]+' \ | sed -E 's/.*[?&]token=//' \ | grep -E '^[A-Za-z0-9._~+/=:@-]+$' \ @@ -131,8 +122,18 @@ collect_tokens() { TOKEN="" DEADLINE=$((SECONDS + TOKEN_WAIT)) +NO_INVOCATION_WARNED=0 while [ -z "$TOKEN" ] && [ "$SECONDS" -lt "$DEADLINE" ]; do - for cand in $(collect_tokens); do + INVOCATION="$(systemctl show -p InvocationID --value "$JOURNAL_UNIT" 2>/dev/null || true)" + if [ -z "$INVOCATION" ] || [ "$INVOCATION" = "n/a" ]; then + if [ "$NO_INVOCATION_WARNED" -eq 0 ]; then + log "WARNING: no invocation id for $JOURNAL_UNIT; waiting for a live invocation" + NO_INVOCATION_WARNED=1 + fi + sleep 2 + continue + fi + for cand in $(journal_tokens "$INVOCATION"); do code="$(curl -s -o /dev/null --max-time 5 -w '%{http_code}' \ -H "Host: $LOGIN_HOST" "$LOGIN_UPSTREAM/?token=$cand" || true)" if [ "$code" = "303" ]; then diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 7b8e0e6..5824a59 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -294,9 +294,11 @@ proxy_pass http://127.0.0.1:3080/?token=; `/opt/deepseek-harness/capture-dsh-token.sh` (source: `scripts/capture-dsh-token.sh`) reads candidate launch tokens from the journal **scoped to the service's current systemd invocation** -(`systemctl show -p InvocationID` + `_SYSTEMD_INVOCATION_ID=`), so a restarted -process's stale token is never considered while its new startup banner is still -pending; there is no whole-journal or cross-invocation fallback. Each candidate +(`systemctl show -p InvocationID` + `_SYSTEMD_INVOCATION_ID=`), re-sampling the +invocation on every pass so a restart that lands during the wait switches to the +new invocation; a restarted process's stale token is never considered while its +new startup banner is still pending and there is no whole-journal or +cross-invocation fallback. Each candidate is then functionally verified against dsh-web with `Host: tankodhs.sysloggh.net`, using the first the running process accepts with `303`. It waits up to 120s for a restarted process to accept a token and re-probes every current-invocation -- 2.54.0 From 767bd22d9c5ee40ef8d885c45890097a7bcbc21d Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Fri, 11 Sep 2026 17:49:25 +0000 Subject: [PATCH 11/11] no-mistakes(document): Align zulip-health v3.2.0 registry metadata and B4 formatting --- contract-registry.yaml | 2 +- zulip-health.prose.md | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/contract-registry.yaml b/contract-registry.yaml index fb50e4a..49731af 100644 --- a/contract-registry.yaml +++ b/contract-registry.yaml @@ -628,7 +628,7 @@ contracts: sensitivity: high status: active owner: abiba - version: 3.1.0 + version: 3.2.0 trigger: type: scheduled cadence: '*/15 * * * *' diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 5824a59..4e9cfb6 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -260,6 +260,7 @@ logged/reported as a warning — reported, never healed on. | `dsh-web` service not `active` | Restart Tanko via DSH service | | HTTP `:3080` connection refused/timeout (`000`) | Same as above | | HTTP status outside the expected set | Log/report as a warning — reported, never healed on | + **B4: dsh-web Authentication (Tanko — restart-persistent login)** The dsh-web UI is token-gated. On every start the process prints a random -- 2.54.0