diff --git a/contract-registry.yaml b/contract-registry.yaml index eec0d9b..4a51da1 100644 --- a/contract-registry.yaml +++ b/contract-registry.yaml @@ -1965,17 +1965,21 @@ contracts: verify_commands: - infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-email - python3 -m pytest tests/test_daily_infra_report.py -q - email_dependency: - transport: smtp.gmail.com:587 - identity: jtabiri@gmail.com - secret: EMAIL_PASSWORD (must be a Google app password) - status: DEGRADED as of 2026-09-25 - 534 5.7.9 Application-specific password required - note: A delivery failure is a credential dependency, not a code defect. Tracked - as daily-digest-mail-transport-20260921. + delivery: + transport: zulip-dm-attachment + recipient_user_id: 9 + sender: abiba-bot@chat.sysloggh.net + key_source: abiba-bot Zulip key already on the execution host, read from the + 600-mode env file /root/.pi/agent/extensions/zulip/.env + key_policy: do NOT add a vault entry - that is a captain decision under the auth-keys charter + body: short Markdown pointer; the HTML attachment IS the report + artifact: /var/log/daily-infra-report/infra-report-.html + note: Replaced SMTP/mail on 2026-09-26 by captain decision. Removes the Google + dependency entirely; closes daily-digest-mail-transport-20260921. exit_semantics: - '1': missing PVE_TOKEN, unreachable Proxmox probe, or failed email send - raises an alert - '0': healthy, or a deliberate DEGRADED leg where the email credential is absent - and the report is still produced + '1': missing PVE_TOKEN, unreachable Proxmox probe, missing/rejected Zulip + credential, or a failed upload/post - raises an alert + '0': healthy delivery only - there is no degraded delivery leg any more depends_on: [] last_run: null last_status: null diff --git a/daily-health-digest.prose.md b/daily-health-digest.prose.md index 9cf7fdd..cfbe83d 100644 --- a/daily-health-digest.prose.md +++ b/daily-health-digest.prose.md @@ -16,11 +16,12 @@ description: > Exit-code semantics (as they actually behave, verified 2026-09-25): * missing PVE_TOKEN, or an unreachable Proxmox probe -> exit 1 + alert - * missing EMAIL credential -> deliberate DEGRADED leg, exit 0, report still - produced - * email send failure -> exit 1 (a delivery fault, not a code defect) + * missing or rejected Zulip credential -> exit 1 (delivery is the only + output path, so it is a real failure, not a degraded leg) + * delivery failure -> exit 1, and the report body is printed AND persisted + so the content is never swallowed -version: 1.0.0 +version: 2.0.0 --- ## Purpose @@ -93,14 +94,13 @@ $ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json EXIT=0 ``` -and in mail mode: +and in delivery mode: ``` - Sending email... - โœ… All legs fully credentialed -๐Ÿ“‹ Summary: - Proxmox: 5/5 nodes online - VMs/CTs: 22/22 running + report ready: 16208 chars of HTML (delivered as a file attachment) + Sending to the captain's Zulip DM... + โœ… Delivered to Zulip DM (user 9), message id 86221, attachment 16208 bytes + at /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html ``` Healthy means: every probe reports `ok`, `nodes_online == node_count`, and the @@ -115,9 +115,8 @@ Verified on 2026-09-25 by running each case deliberately. | all probes reachable, email sent | 0 | โ€” | healthy | | **missing `PVE_TOKEN`** | **1** | yes | `PROBE FAILURES: proxmox: node list unreachable (PVE_TOKEN missing or API down)`, and `cluster resources unreachable` | | **Proxmox probe unreachable** | **1** | yes | same path as above; `pve_probe_status: unreachable` | -| **missing `EMAIL_PASSWORD`** | **0** | no | deliberate **DEGRADED** leg (`credential-missing: EMAIL_PASSWORD`); the report is still produced | -| **email send fails** | **1** | yes | e.g. Gmail `534 5.7.9 Application-specific password required` | -| degraded legs present (non-email) | 0 | no | logged under `โš ๏ธ Degraded legs` | +| **missing/rejected Zulip credential** | **1** | yes | delivery is the only output path; report printed and persisted | +| **upload or message post fails** | **1** | yes | report printed and persisted; message names which step failed | The distinction is deliberate and must not be flattened: @@ -130,22 +129,31 @@ The distinction is deliberate and must not be flattened: `PROBE_FAILURES` and `DEGRADED_LEGS` are separate lists for exactly this reason. Do not merge them. -## Email-delivery dependency +## Delivery: Zulip DM carrying the report as an HTML ATTACHMENT -Delivery is a **credential dependency, not a code path**. The producer -authenticates to `smtp.gmail.com:587` as `jtabiri@gmail.com` with -`EMAIL_PASSWORD` from the vault and sends to `jerome@sysloggh.com`. +Captain's decision 2026-09-26, clarified the same day: the digest is delivered to +his **Zulip DM (user id 9)** from `abiba-bot@chat.sysloggh.net`, as an **HTML +FILE** โ€” an attachment, not HTML rendered in the message body and not a Markdown +translation of it. -* Since that Google account has two-step verification, `EMAIL_PASSWORD` must be - a Google **app password**, not the account password. -* As of 2026-09-25 delivery is **failing** with - `534 5.7.9 Application-specific password required`; the fix is for the - captain to generate a fresh app password and place it in Infisical - (`infrastructure/production`) as `EMAIL_PASSWORD`. -* **A delivery failure is not a code defect.** Investigation of a failed send - should start at the credential, not the script. Chasing it as a code bug - wastes the effort; verify the credential path first with `--test-email`. -* Tracked separately as `daily-digest-mail-transport-20260921`. +* the styled dashboard is built exactly as before and written to + `/var/log/daily-infra-report/infra-report-.html`; +* it is uploaded through `POST /api/v1/user_uploads`; +* the **message body stays short Markdown** โ€” subject line, top-line status + (nodes online, guests running, any degraded legs), and a link to the + attachment. The attachment IS the report; the body does not reproduce it. + +This removes the Google dependency entirely: **no SMTP, no `EMAIL_PASSWORD`, no +app password, nothing to rotate.** `daily-digest-mail-transport-20260921` is +closed under this option. + +The **10,000-character message cap does not apply** โ€” it bounds message TEXT +only, and the report travels as a file. Do not shrink the report to fit it. + +The credential is abiba-bot's Zulip key already on the execution host at +`/root/.pi/agent/extensions/zulip/.env` (`ABIBA_ZULIP_API_KEY`, mode 600, +root-readable). **Do not place a new credential in the vault** โ€” under the +auth-keys charter that is a captain decision. ## What counts as a failure @@ -153,10 +161,18 @@ A run FAILS (exit 1) when the report cannot be trusted or delivered: * any probe is unreachable, so a section would silently be empty; * `PVE_TOKEN` is missing; -* the email send fails. +* the Zulip credential is missing or rejected, or the upload/post fails. -A run is DEGRADED (exit 0, report still produced) when a non-load-bearing -credential is absent, currently only `EMAIL_PASSWORD`. +There is **no degraded delivery leg any more**. Delivery is the only output +path, so a missing credential is a failure rather than a survivable degradation โ€” +the previous "missing `EMAIL_PASSWORD` still exits 0" rule is retired with the +mail transport. + +**A delivery failure must never swallow the report.** On failure the script +prints the report body to stdout *and* leaves the HTML artifact on disk, so the +content is always recoverable from the run log. That closes the queued defect +where a failed send printed only the transport error and the report never +surfaced. ## Failure behaviour @@ -175,7 +191,7 @@ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json \ | grep -E 'pve_probe_status|node_count|nodes_online' # delivery path -infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-email +infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-zulip ``` Regression tests: `tests/test_daily_infra_report.py` (7 tests). Four of them @@ -183,5 +199,5 @@ fail against the pre-fix script, which is what makes them bite. ## Maintains -- daily-infra-dashboard: { status: "degraded", reason: "email credential", last_check: timestamp } +- daily-infra-dashboard: { status: "ok|undelivered", transport: zulip-dm-attachment, last_check: timestamp } - pve-probe: { status: "ok|unreachable", last_check: timestamp } diff --git a/scripts/daily-infra-report.py b/scripts/daily-infra-report.py index 2137dee..d4ea680 100755 --- a/scripts/daily-infra-report.py +++ b/scripts/daily-infra-report.py @@ -243,7 +243,7 @@ def collect(): ("Pulse", "https://pulse.sysloggh.net"), ("Proxmox", "https://192.168.68.12:8006"), ("SearXNG", "http://192.168.68.7:8888"), - ("Firecrawl", "http://192.168.68.7:3002/health"), + ("Firecrawl", "http://192.168.68.7:3002/"), # Firecrawl serves no /health - the root is its liveness endpoint ] report["endpoints"] = [] for name, url in endpoints: @@ -389,6 +389,28 @@ def collect(): # โ”€โ”€ HTML Dashboard โ”€โ”€ +def classify_endpoint(code): + """Classify an endpoint probe per the fleet's probe policy. + + Codified 2026-09-14 in the monitoring contracts: ANY HTTP status proves the + service answered, so the service is ALIVE - 200/301/302/401/403/404 alike. + Only a failed CONNECTION (000 / timeout / refused) is a failed probe. A 404 + from a wrong path is not a service fault and must not render as one. + + This replaces a string comparison that was wrong in both directions + (`ep["code"] >= "400"`): it rendered 301 as red, 404 as yellow, and a real + 500 as yellow. 5xx is kept as its own "server error" signal rather than + being merged with 4xx. + """ + if not code or code == "000": + return "red", "no connection" + if code.startswith("5"): + return "yellow", "server error" + if code.startswith(("2", "3", "4")): + return "green", "alive" + return "yellow", f"unexpected {code}" + + def build_html(r): issues = [] @@ -624,7 +646,7 @@ Proxmox: {r.get('pve_probe_status', 'ok')} ({r['nodes_online']}/{r['node_count'] # โ”€โ”€ Network Endpoints โ”€โ”€ html += '

๐ŸŒ Network Endpoints

' for ep in r["endpoints"]: - color = "green" if ep["code"] in ("200","302","401") else ("yellow" if ep["code"] >= "400" else "red") + color = classify_endpoint(ep["code"])[0] html += f'' html += '
ServiceStatus
{ep["name"]}HTTP {ep["code"]}
' @@ -688,42 +710,152 @@ Proxmox: {r.get('pve_probe_status', 'ok')} ({r['nodes_online']}/{r['node_count'] return html -# โ”€โ”€ Send Email โ”€โ”€ +# โ”€โ”€ Delivery: Zulip DM carrying the report as an HTML ATTACHMENT โ”€โ”€ +# +# Captain's decision, clarified 2026-09-26: the report is sent as an HTML FILE, +# i.e. an attachment - NOT HTML rendered in the message body, and NOT a Markdown +# translation of it. So the styled dashboard is built exactly as before, uploaded +# through Zulip's file-upload API, and the message body stays short: subject, +# top-line status, and a pointer to the attachment. +# +# This removes the Google dependency entirely (no SMTP, no EMAIL_PASSWORD). +# The 10,000-character message cap does not apply: it bounds message TEXT only, +# and the report travels as a file. -def send_email(html_content, subject_prefix=""): - FROM = "abiba@sysloggh.com" - TO = "jerome@sysloggh.com" - SUBJECT = f"{subject_prefix}{'๐Ÿ—๏ธ Infrastructure Report โ€” ' + DATE_STR}" - - msg = MIMEMultipart("alternative") - msg["From"] = FROM - msg["To"] = TO - msg["Subject"] = SUBJECT - msg.attach(MIMEText("Infrastructure report in HTML format โ€” enable images to view.", "plain")) - msg.attach(MIMEText(html_content, "html")) - +ZULIP_SITE = "https://chat.sysloggh.net" +ZULIP_BOT_EMAIL = "abiba-bot@chat.sysloggh.net" +CAPTAIN_USER_ID = 9 +ZULIP_KEY_FILE = "/root/.pi/agent/extensions/zulip/.env" +REPORT_ARTIFACT_DIR = "/var/log/daily-infra-report" + + +def zulip_key(): + """abiba-bot's Zulip key, from the env or the on-host 600 file.""" + key = os.environ.get("ABIBA_ZULIP_API_KEY") + if key: + return key.strip() try: - EMAIL_PASSWORD = os.environ.get("EMAIL_PASSWORD") or os.environ.get("SMTP_PASSWORD") or os.environ.get("MAIL_PASSWORD") - if not EMAIL_PASSWORD: - print(" โš ๏ธ Degraded leg: credential-missing: EMAIL_PASSWORD (or SMTP_PASSWORD/MAIL_PASSWORD)", file=sys.stderr) - DEGRADED_LEGS.append("credential-missing: EMAIL_PASSWORD") - return True, "โœ… Email leg degraded (no credential) โ€” report still produced" - GMAIL_EMAIL = "jtabiri@gmail.com" - - server = smtplib.SMTP("smtp.gmail.com", 587) - server.starttls() - server.login(GMAIL_EMAIL, EMAIL_PASSWORD) - server.sendmail(FROM, [TO], msg.as_string()) - server.quit() - return True, "โœ… Email sent to jerome@sysloggh.com" - except Exception as e: - return False, f"โŒ Email failed: {e}" + with open(ZULIP_KEY_FILE) as fh: + for line in fh: + if line.startswith("ABIBA_ZULIP_API_KEY="): + return line.split("=", 1)[1].strip() + except OSError: + return None + return None + + +def build_summary(r, filename, test=False): + """Short Markdown body: subject, top-line status, pointer to the attachment. + + Deliberately NOT a reproduction of the report - the attachment is the report. + """ + nodes = f"{r.get('nodes_online', 0)}/{r.get('node_count', 0)} nodes online" + guests = f"{r.get('running_vms', 0)}/{r.get('total_vms', 0)} guests running" + lines = [ + ("\U0001F9EA **TEST โ€” **" if test else "") + "\U0001F3D7\uFE0F **Infrastructure Report โ€” " + DATE_STR + "**", + f"**{nodes}** \u00b7 **{guests}** \u00b7 generated {TIME_STR}", + ] + problems = [] + if r.get("pve_probe_status") != "ok": + problems.append(f"\u274c Proxmox probe: {r.get('pve_probe_status')}") + if r.get("resources_probe_status") != "ok": + problems.append(f"\u274c Resources probe: {r.get('resources_probe_status')}") + lit = r.get("litellm", {}) or {} + checks = lit.get("checks", []) or [] + if checks: + passed = sum(1 for c in checks if c.get("status") == "pass") + if passed != len(checks): + problems.append(f"\u274c LiteLLM: {passed}/{len(checks)} checks pass") + if not (r.get("zulip_ext", {}) or {}).get("connected"): + problems.append("\u274c Zulip extension: not connected") + for leg in DEGRADED_LEGS: + problems.append(f"\u26a0\uFE0F degraded: {leg}") + + lines.append("\n".join(problems) if problems else "\u2705 All monitored services healthy") + lines.append(f"\U0001F4CE **Full report attached:** `{filename}`") + return "\n\n".join(lines) + + +def _curl(args, timeout=60): + r = subprocess.run(["curl", "-s", "-m", str(timeout)] + args, + capture_output=True, text=True) + try: + return json.loads(r.stdout or "{}"), r.stdout + except json.JSONDecodeError: + return {}, r.stdout + + +def _curl_json(args, timeout=90): + r = subprocess.run(["curl", "-s", "-m", str(timeout)] + args, + capture_output=True, text=True) + try: + return json.loads(r.stdout or "{}"), r.stdout + except json.JSONDecodeError: + return {}, r.stdout + + +def send_zulip(html_content, report, test=False): + """Upload the styled HTML and post a short pointer to the captain's DM. + + Returns (ok, message). On ANY failure the report body is also printed to + stdout and persisted to disk, so a delivery failure can never swallow the + content - the defect this folds in. + """ + os.makedirs(REPORT_ARTIFACT_DIR, exist_ok=True) + stamp = NOW.strftime("%Y%m%d-%H%M%S") + filename = f"infra-report-{stamp}.html" + html_path = os.path.join(REPORT_ARTIFACT_DIR, filename) + try: + with open(html_path, "w") as fh: + fh.write(html_content) + except OSError as e: + print(f" \u26a0\uFE0F could not persist report artifact: {e}", file=sys.stderr) + + key = zulip_key() + if not key: + print(html_content) # never swallow the content + return False, ("\u274c Delivery FAILED: no Zulip credential " + "(ABIBA_ZULIP_API_KEY unset and " + f"{ZULIP_KEY_FILE} unreadable). Report persisted to {html_path}") + + auth = ["-u", f"{ZULIP_BOT_EMAIL}:{key}"] + + # 1. Upload the report as a file. + up, up_raw = _curl_json(auth + [ + "-X", "POST", f"{ZULIP_SITE}/api/v1/user_uploads", + "-F", f"file=@{html_path};type=text/html", + ]) + if up.get("result") != "success" or not up.get("uri"): + print(html_content) + return False, (f"\u274c Delivery FAILED at upload: {up.get('msg') or up_raw[:160]} " + f"(report persisted to {html_path})") + + uri = up["uri"] + size = os.path.getsize(html_path) + + # 2. Post a short message pointing at it. + body = build_summary(report, filename, test=test) + link = f"[{filename}]({uri})" + body = body.replace(f"`{filename}`", link) + payload, raw = _curl_json(auth + [ + "-X", "POST", f"{ZULIP_SITE}/api/v1/messages", + "-d", "type=private", + "-d", f"to=[{CAPTAIN_USER_ID}]", + "--data-urlencode", f"content={body}", + ]) + if payload.get("result") == "success": + return True, (f"\u2705 Delivered to Zulip DM (user {CAPTAIN_USER_ID}), " + f"message id {payload.get('id')}, attachment {size} bytes at {uri}") + + print(html_content) + return False, (f"\u274c Delivery FAILED at message post: {payload.get('msg') or raw[:160]} " + f"(uploaded {uri}; report persisted to {html_path})") # โ”€โ”€ Main โ”€โ”€ if __name__ == "__main__": - is_test = "--test-email" in sys.argv + is_test = ("--test-email" in sys.argv) or ("--test-zulip" in sys.argv) print(f"{'๐Ÿงช TEST MODE' if is_test else '๐Ÿ“Š'} Collecting infrastructure data...") report = collect() @@ -738,15 +870,14 @@ if __name__ == "__main__": print(" Building dashboard...") html = build_html(report) - + print(f" report ready: {len(html)} chars of HTML (delivered as a file attachment)") + if is_test: - prefix = "๐Ÿงช TEST โ€” " - print(" Sending test email...") + print(" Sending TEST message to the captain's Zulip DM...") else: - prefix = "" - print(" Sending email...") - - ok, msg = send_email(html, subject_prefix=prefix) + print(" Sending to the captain's Zulip DM...") + + ok, msg = send_zulip(html, report, test=is_test) print(f" {msg}") # Show summary