feat(daily-digest): deliver via Zulip DM as an HTML attachment; drop mail entirely

Captain's decision 2026-09-26, clarified the same day: the digest is delivered to his Zulip DM (user id 9) from abiba-bot as an HTML FILE - an attachment, not HTML rendered in the message body and not a Markdown translation of it. Closes daily-digest-mail-transport-20260921; the Google dependency is gone (no SMTP, no EMAIL_PASSWORD, no app password, nothing to rotate). WHAT CHANGES * scripts/daily-infra-report.py: send_email() is replaced by send_zulip(), which writes the styled dashboard to /var/log/daily-infra-report/infra-report-<ts>.html, uploads it via POST /api/v1/user_uploads, then posts a SHORT Markdown pointer to user 9. The message body carries subject, top-line status and the attachment link; it does not reproduce the report. * the 10,000-character cap is irrelevant here - it bounds message TEXT only, and the report travels as a file, so nothing is shrunk to fit. * the key is abiba-bot's, already on the execution host at /root/.pi/agent/extensions/zulip/.env (mode 600). No vault entry was added: under the auth-keys charter that is a captain decision. * daily-health-digest.prose.md -> v2.0.0 and contract-registry.yaml updated: transport, healthy/degraded definitions, and exit codes now match observed behaviour. There is NO degraded delivery leg any more - delivery is the only output path, so a missing or rejected key is a real failure (exit 1). * queued defect folded in: a failed delivery used to print only the transport error while the report body never surfaced. Now the HTML is printed to stdout AND persisted on every failure, and the message names which step failed. EVIDENCE (all against the live stack) * real send: message id 86221 to user 9, attachment 16208 bytes at /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html * the message is type=private, sender abiba-bot@chat.sysloggh.net, recipients [9, 21], body carries the top-line status and the attachment link, and does NOT contain a <table> - i.e. it does not reproduce the report * the attachment fetches HTTP 200, 16208 bytes, content-type text/html, starts with <!DOCTYPE html>, and contains <style>, <table> and 16 class="card" blocks - it opens as a standalone styled document * failure path: a bad key gives 'Delivery FAILED at upload: Malformed API key', EXIT=1, the HTML is printed to stdout and persisted to disk * scheduled path: the run's own output is pasted in the PR prose-lint: PASSED (19 warnings); secret scan clean.
2026-09-26 15:35:36 +00:00
parent 042fd3ccc6
commit de32f54337
3 changed files with 207 additions and 78 deletions
@@ -1965,17 +1965,21 @@ contracts:
    verify_commands:
    - infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-email
    - python3 -m pytest tests/test_daily_infra_report.py -q
-  email_dependency:
-    transport: smtp.gmail.com:587
-    identity: jtabiri@gmail.com
-    secret: EMAIL_PASSWORD (must be a Google app password)
-    status: DEGRADED as of 2026-09-25 - 534 5.7.9 Application-specific password required
-    note: A delivery failure is a credential dependency, not a code defect. Tracked
-      as daily-digest-mail-transport-20260921.
+  delivery:
+    transport: zulip-dm-attachment
+    recipient_user_id: 9
+    sender: abiba-bot@chat.sysloggh.net
+    key_source: abiba-bot Zulip key already on the execution host, read from the
+      600-mode env file /root/.pi/agent/extensions/zulip/.env
+    key_policy: do NOT add a vault entry - that is a captain decision under the auth-keys charter
+    body: short Markdown pointer; the HTML attachment IS the report
+    artifact: /var/log/daily-infra-report/infra-report-<UTCstamp>.html
+    note: Replaced SMTP/mail on 2026-09-26 by captain decision. Removes the Google
+      dependency entirely; closes daily-digest-mail-transport-20260921.
  exit_semantics:
-    '1': missing PVE_TOKEN, unreachable Proxmox probe, or failed email send - raises an alert
-    '0': healthy, or a deliberate DEGRADED leg where the email credential is absent
-      and the report is still produced
+    '1': missing PVE_TOKEN, unreachable Proxmox probe, missing/rejected Zulip
+      credential, or a failed upload/post - raises an alert
+    '0': healthy delivery only - there is no degraded delivery leg any more
  depends_on: []
  last_run: null
  last_status: null
@@ -16,11 +16,12 @@ description: >

  Exit-code semantics (as they actually behave, verified 2026-09-25):
    * missing PVE_TOKEN, or an unreachable Proxmox probe -> exit 1 + alert
-    * missing EMAIL credential -> deliberate DEGRADED leg, exit 0, report still
-      produced
-    * email send failure -> exit 1 (a delivery fault, not a code defect)
+    * missing or rejected Zulip credential -> exit 1 (delivery is the only
+      output path, so it is a real failure, not a degraded leg)
+    * delivery failure -> exit 1, and the report body is printed AND persisted
+      so the content is never swallowed

-version: 1.0.0
+version: 2.0.0
 ---

 ## Purpose
@@ -93,14 +94,13 @@ $ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json
 EXIT=0
 ```

-and in mail mode:
+and in delivery mode:

 ```
-   Sending email...
-   ✅ All legs fully credentialed
-📋 Summary:
-   Proxmox: 5/5 nodes online
-   VMs/CTs: 22/22 running
+   report ready: 16208 chars of HTML (delivered as a file attachment)
+   Sending to the captain's Zulip DM...
+   ✅ Delivered to Zulip DM (user 9), message id 86221, attachment 16208 bytes
+      at /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html
 ```

 Healthy means: every probe reports `ok`, `nodes_online == node_count`, and the
@@ -115,9 +115,8 @@ Verified on 2026-09-25 by running each case deliberately.
 | all probes reachable, email sent | 0 | — | healthy |
 | **missing `PVE_TOKEN`** | **1** | yes | `PROBE FAILURES: proxmox: node list unreachable (PVE_TOKEN missing or API down)`, and `cluster resources unreachable` |
 | **Proxmox probe unreachable** | **1** | yes | same path as above; `pve_probe_status: unreachable` |
-| **missing `EMAIL_PASSWORD`** | **0** | no | deliberate **DEGRADED** leg (`credential-missing: EMAIL_PASSWORD`); the report is still produced |
-| **email send fails** | **1** | yes | e.g. Gmail `534 5.7.9 Application-specific password required` |
-| degraded legs present (non-email) | 0 | no | logged under `⚠️ Degraded legs` |
+| **missing/rejected Zulip credential** | **1** | yes | delivery is the only output path; report printed and persisted |
+| **upload or message post fails** | **1** | yes | report printed and persisted; message names which step failed |

 The distinction is deliberate and must not be flattened:

@@ -130,22 +129,31 @@ The distinction is deliberate and must not be flattened:
 `PROBE_FAILURES` and `DEGRADED_LEGS` are separate lists for exactly this
 reason. Do not merge them.

-## Email-delivery dependency
+## Delivery: Zulip DM carrying the report as an HTML ATTACHMENT

-Delivery is a **credential dependency, not a code path**. The producer
-authenticates to `smtp.gmail.com:587` as `jtabiri@gmail.com` with
-`EMAIL_PASSWORD` from the vault and sends to `jerome@sysloggh.com`.
+Captain's decision 2026-09-26, clarified the same day: the digest is delivered to
+his **Zulip DM (user id 9)** from `abiba-bot@chat.sysloggh.net`, as an **HTML
+FILE** — an attachment, not HTML rendered in the message body and not a Markdown
+translation of it.

-* Since that Google account has two-step verification, `EMAIL_PASSWORD` must be
-  a Google **app password**, not the account password.
-* As of 2026-09-25 delivery is **failing** with
-  `534 5.7.9 Application-specific password required`; the fix is for the
-  captain to generate a fresh app password and place it in Infisical
-  (`infrastructure/production`) as `EMAIL_PASSWORD`.
-* **A delivery failure is not a code defect.** Investigation of a failed send
-  should start at the credential, not the script. Chasing it as a code bug
-  wastes the effort; verify the credential path first with `--test-email`.
-* Tracked separately as `daily-digest-mail-transport-20260921`.
+* the styled dashboard is built exactly as before and written to
+  `/var/log/daily-infra-report/infra-report-<UTCstamp>.html`;
+* it is uploaded through `POST /api/v1/user_uploads`;
+* the **message body stays short Markdown** — subject line, top-line status
+  (nodes online, guests running, any degraded legs), and a link to the
+  attachment. The attachment IS the report; the body does not reproduce it.
+
+This removes the Google dependency entirely: **no SMTP, no `EMAIL_PASSWORD`, no
+app password, nothing to rotate.** `daily-digest-mail-transport-20260921` is
+closed under this option.
+
+The **10,000-character message cap does not apply** — it bounds message TEXT
+only, and the report travels as a file. Do not shrink the report to fit it.
+
+The credential is abiba-bot's Zulip key already on the execution host at
+`/root/.pi/agent/extensions/zulip/.env` (`ABIBA_ZULIP_API_KEY`, mode 600,
+root-readable). **Do not place a new credential in the vault** — under the
+auth-keys charter that is a captain decision.

 ## What counts as a failure

@@ -153,10 +161,18 @@ A run FAILS (exit 1) when the report cannot be trusted or delivered:

 * any probe is unreachable, so a section would silently be empty;
 * `PVE_TOKEN` is missing;
-* the email send fails.
+* the Zulip credential is missing or rejected, or the upload/post fails.

-A run is DEGRADED (exit 0, report still produced) when a non-load-bearing
-credential is absent, currently only `EMAIL_PASSWORD`.
+There is **no degraded delivery leg any more**. Delivery is the only output
+path, so a missing credential is a failure rather than a survivable degradation —
+the previous "missing `EMAIL_PASSWORD` still exits 0" rule is retired with the
+mail transport.
+
+**A delivery failure must never swallow the report.** On failure the script
+prints the report body to stdout *and* leaves the HTML artifact on disk, so the
+content is always recoverable from the run log. That closes the queued defect
+where a failed send printed only the transport error and the report never
+surfaced.

 ## Failure behaviour

@@ -175,7 +191,7 @@ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json \
  | grep -E 'pve_probe_status|node_count|nodes_online'

 # delivery path
-infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-email
+infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-zulip
 ```

 Regression tests: `tests/test_daily_infra_report.py` (7 tests). Four of them
@@ -183,5 +199,5 @@ fail against the pre-fix script, which is what makes them bite.

 ## Maintains

- daily-infra-dashboard: { status: "degraded", reason: "email credential", last_check: timestamp }
+- daily-infra-dashboard: { status: "ok|undelivered", transport: zulip-dm-attachment, last_check: timestamp }
 - pve-probe: { status: "ok|unreachable", last_check: timestamp }
@@ -688,42 +688,152 @@ Proxmox: {r.get('pve_probe_status', 'ok')} ({r['nodes_online']}/{r['node_count']
    return html


-# ── Send Email ──
+# ── Delivery: Zulip DM carrying the report as an HTML ATTACHMENT ──
+#
+# Captain's decision, clarified 2026-09-26: the report is sent as an HTML FILE,
+# i.e. an attachment - NOT HTML rendered in the message body, and NOT a Markdown
+# translation of it. So the styled dashboard is built exactly as before, uploaded
+# through Zulip's file-upload API, and the message body stays short: subject,
+# top-line status, and a pointer to the attachment.
+#
+# This removes the Google dependency entirely (no SMTP, no EMAIL_PASSWORD).
+# The 10,000-character message cap does not apply: it bounds message TEXT only,
+# and the report travels as a file.

-def send_email(html_content, subject_prefix=""):
-    FROM = "abiba@sysloggh.com"
-    TO = "jerome@sysloggh.com"
-    SUBJECT = f"{subject_prefix}{'🏗️ Infrastructure Report — ' + DATE_STR}"
-    
-    msg = MIMEMultipart("alternative")
-    msg["From"] = FROM
-    msg["To"] = TO
-    msg["Subject"] = SUBJECT
-    msg.attach(MIMEText("Infrastructure report in HTML format — enable images to view.", "plain"))
-    msg.attach(MIMEText(html_content, "html"))
-    
+ZULIP_SITE = "https://chat.sysloggh.net"
+ZULIP_BOT_EMAIL = "abiba-bot@chat.sysloggh.net"
+CAPTAIN_USER_ID = 9
+ZULIP_KEY_FILE = "/root/.pi/agent/extensions/zulip/.env"
+REPORT_ARTIFACT_DIR = "/var/log/daily-infra-report"
+
+
+def zulip_key():
+    """abiba-bot's Zulip key, from the env or the on-host 600 file."""
+    key = os.environ.get("ABIBA_ZULIP_API_KEY")
+    if key:
+        return key.strip()
    try:
-        EMAIL_PASSWORD = os.environ.get("EMAIL_PASSWORD") or os.environ.get("SMTP_PASSWORD") or os.environ.get("MAIL_PASSWORD")
-        if not EMAIL_PASSWORD:
-            print("   ⚠️  Degraded leg: credential-missing: EMAIL_PASSWORD (or SMTP_PASSWORD/MAIL_PASSWORD)", file=sys.stderr)
-            DEGRADED_LEGS.append("credential-missing: EMAIL_PASSWORD")
-            return True, "✅ Email leg degraded (no credential) — report still produced"
-        GMAIL_EMAIL = "jtabiri@gmail.com"
-        
-        server = smtplib.SMTP("smtp.gmail.com", 587)
-        server.starttls()
-        server.login(GMAIL_EMAIL, EMAIL_PASSWORD)
-        server.sendmail(FROM, [TO], msg.as_string())
-        server.quit()
-        return True, "✅ Email sent to jerome@sysloggh.com"
-    except Exception as e:
-        return False, f"❌ Email failed: {e}"
+        with open(ZULIP_KEY_FILE) as fh:
+            for line in fh:
+                if line.startswith("ABIBA_ZULIP_API_KEY="):
+                    return line.split("=", 1)[1].strip()
+    except OSError:
+        return None
+    return None
+
+
+def build_summary(r, filename, test=False):
+    """Short Markdown body: subject, top-line status, pointer to the attachment.
+
+    Deliberately NOT a reproduction of the report - the attachment is the report.
+    """
+    nodes = f"{r.get('nodes_online', 0)}/{r.get('node_count', 0)} nodes online"
+    guests = f"{r.get('running_vms', 0)}/{r.get('total_vms', 0)} guests running"
+    lines = [
+        ("\U0001F9EA **TEST — **" if test else "") + "\U0001F3D7\uFE0F **Infrastructure Report — " + DATE_STR + "**",
+        f"**{nodes}** \u00b7 **{guests}** \u00b7 generated {TIME_STR}",
+    ]
+    problems = []
+    if r.get("pve_probe_status") != "ok":
+        problems.append(f"\u274c Proxmox probe: {r.get('pve_probe_status')}")
+    if r.get("resources_probe_status") != "ok":
+        problems.append(f"\u274c Resources probe: {r.get('resources_probe_status')}")
+    lit = r.get("litellm", {}) or {}
+    checks = lit.get("checks", []) or []
+    if checks:
+        passed = sum(1 for c in checks if c.get("status") == "pass")
+        if passed != len(checks):
+            problems.append(f"\u274c LiteLLM: {passed}/{len(checks)} checks pass")
+    if not (r.get("zulip_ext", {}) or {}).get("connected"):
+        problems.append("\u274c Zulip extension: not connected")
+    for leg in DEGRADED_LEGS:
+        problems.append(f"\u26a0\uFE0F degraded: {leg}")
+
+    lines.append("\n".join(problems) if problems else "\u2705 All monitored services healthy")
+    lines.append(f"\U0001F4CE **Full report attached:** `{filename}`")
+    return "\n\n".join(lines)
+
+
+def _curl(args, timeout=60):
+    r = subprocess.run(["curl", "-s", "-m", str(timeout)] + args,
+                       capture_output=True, text=True)
+    try:
+        return json.loads(r.stdout or "{}"), r.stdout
+    except json.JSONDecodeError:
+        return {}, r.stdout
+
+
+def _curl_json(args, timeout=90):
+    r = subprocess.run(["curl", "-s", "-m", str(timeout)] + args,
+                       capture_output=True, text=True)
+    try:
+        return json.loads(r.stdout or "{}"), r.stdout
+    except json.JSONDecodeError:
+        return {}, r.stdout
+
+
+def send_zulip(html_content, report, test=False):
+    """Upload the styled HTML and post a short pointer to the captain's DM.
+
+    Returns (ok, message). On ANY failure the report body is also printed to
+    stdout and persisted to disk, so a delivery failure can never swallow the
+    content - the defect this folds in.
+    """
+    os.makedirs(REPORT_ARTIFACT_DIR, exist_ok=True)
+    stamp = NOW.strftime("%Y%m%d-%H%M%S")
+    filename = f"infra-report-{stamp}.html"
+    html_path = os.path.join(REPORT_ARTIFACT_DIR, filename)
+    try:
+        with open(html_path, "w") as fh:
+            fh.write(html_content)
+    except OSError as e:
+        print(f"   \u26a0\uFE0F  could not persist report artifact: {e}", file=sys.stderr)
+
+    key = zulip_key()
+    if not key:
+        print(html_content)  # never swallow the content
+        return False, ("\u274c Delivery FAILED: no Zulip credential "
+                       "(ABIBA_ZULIP_API_KEY unset and "
+                       f"{ZULIP_KEY_FILE} unreadable). Report persisted to {html_path}")
+
+    auth = ["-u", f"{ZULIP_BOT_EMAIL}:{key}"]
+
+    # 1. Upload the report as a file.
+    up, up_raw = _curl_json(auth + [
+        "-X", "POST", f"{ZULIP_SITE}/api/v1/user_uploads",
+        "-F", f"file=@{html_path};type=text/html",
+    ])
+    if up.get("result") != "success" or not up.get("uri"):
+        print(html_content)
+        return False, (f"\u274c Delivery FAILED at upload: {up.get('msg') or up_raw[:160]} "
+                       f"(report persisted to {html_path})")
+
+    uri = up["uri"]
+    size = os.path.getsize(html_path)
+
+    # 2. Post a short message pointing at it.
+    body = build_summary(report, filename, test=test)
+    link = f"[{filename}]({uri})"
+    body = body.replace(f"`{filename}`", link)
+    payload, raw = _curl_json(auth + [
+        "-X", "POST", f"{ZULIP_SITE}/api/v1/messages",
+        "-d", "type=private",
+        "-d", f"to=[{CAPTAIN_USER_ID}]",
+        "--data-urlencode", f"content={body}",
+    ])
+    if payload.get("result") == "success":
+        return True, (f"\u2705 Delivered to Zulip DM (user {CAPTAIN_USER_ID}), "
+                      f"message id {payload.get('id')}, attachment {size} bytes at {uri}")
+
+    print(html_content)
+    return False, (f"\u274c Delivery FAILED at message post: {payload.get('msg') or raw[:160]} "
+                   f"(uploaded {uri}; report persisted to {html_path})")


 # ── Main ──

 if __name__ == "__main__":
-    is_test = "--test-email" in sys.argv
+    is_test = ("--test-email" in sys.argv) or ("--test-zulip" in sys.argv)
    
    print(f"{'🧪 TEST MODE' if is_test else '📊'} Collecting infrastructure data...")
    report = collect()
@@ -738,15 +848,14 @@ if __name__ == "__main__":
    
    print("   Building dashboard...")
    html = build_html(report)
-    
+    print(f"   report ready: {len(html)} chars of HTML (delivered as a file attachment)")
+
    if is_test:
-        prefix = "🧪 TEST — "
-        print("   Sending test email...")
+        print("   Sending TEST message to the captain's Zulip DM...")
    else:
-        prefix = ""
-        print("   Sending email...")
-    
-    ok, msg = send_email(html, subject_prefix=prefix)
+        print("   Sending to the captain's Zulip DM...")
+
+    ok, msg = send_zulip(html, report, test=is_test)
    print(f"   {msg}")
    
    # Show summary