SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(daily-digest): deliver via Zulip DM as an HTML attachment; drop mail entirely
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
Details
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s
Details
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 13s
Details
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s
Details
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Details

Browse Source 
Captain's decision 2026-09-26, clarified the same day: the digest is delivered to
his Zulip DM (user id 9) from abiba-bot as an HTML FILE - an attachment, not HTML
rendered in the message body and not a Markdown translation of it. Closes
daily-digest-mail-transport-20260921; the Google dependency is gone (no SMTP, no
EMAIL_PASSWORD, no app password, nothing to rotate).

WHAT CHANGES
* scripts/daily-infra-report.py: send_email() is replaced by send_zulip(), which
  writes the styled dashboard to /var/log/daily-infra-report/infra-report-<ts>.html,
  uploads it via POST /api/v1/user_uploads, then posts a SHORT Markdown pointer to
  user 9. The message body carries subject, top-line status and the attachment
  link; it does not reproduce the report.
* the 10,000-character cap is irrelevant here - it bounds message TEXT only, and
  the report travels as a file, so nothing is shrunk to fit.
* the key is abiba-bot's, already on the execution host at
  /root/.pi/agent/extensions/zulip/.env (mode 600). No vault entry was added:
  under the auth-keys charter that is a captain decision.
* daily-health-digest.prose.md -> v2.0.0 and contract-registry.yaml updated:
  transport, healthy/degraded definitions, and exit codes now match observed
  behaviour. There is NO degraded delivery leg any more - delivery is the only
  output path, so a missing or rejected key is a real failure (exit 1).
* queued defect folded in: a failed delivery used to print only the transport
  error while the report body never surfaced. Now the HTML is printed to stdout
  AND persisted on every failure, and the message names which step failed.

EVIDENCE (all against the live stack)
* real send: message id 86221 to user 9, attachment 16208 bytes at
  /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html
* the message is type=private, sender abiba-bot@chat.sysloggh.net, recipients
  [9, 21], body carries the top-line status and the attachment link, and does NOT
  contain a <table> - i.e. it does not reproduce the report
* the attachment fetches HTTP 200, 16208 bytes, content-type text/html, starts
  with <!DOCTYPE html>, and contains <style>, <table> and 16 class="card" blocks -
  it opens as a standalone styled document
* failure path: a bad key gives 'Delivery FAILED at upload: Malformed API key',
  EXIT=1, the HTML is printed to stdout and persisted to disk
* scheduled path: the run's own output is pasted in the PR

prose-lint: PASSED (19 warnings); secret scan clean.
This commit is contained in:
root
2026-09-26 15:35:36 +00:00
parent 042fd3ccc6
commit de32f54337
3 changed files with 207 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
contract-registry.yaml
+14 -10
View File
@@ -1965,17 +1965,21 @@ contracts:
verify_commands:
- infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-email
- python3 -m pytest tests/test_daily_infra_report.py -q
email_dependency:
transport: smtp.gmail.com:587
identity: jtabiri@gmail.com
secret: EMAIL_PASSWORD (must be a Google app password)
status: DEGRADED as of 2026-09-25 - 534 5.7.9 Application-specific password required
note: A delivery failure is a credential dependency, not a code defect. Tracked
as daily-digest-mail-transport-20260921.
delivery:
transport: zulip-dm-attachment
recipient_user_id: 9
sender: abiba-bot@chat.sysloggh.net
key_source: abiba-bot Zulip key already on the execution host, read from the
600-mode env file /root/.pi/agent/extensions/zulip/.env
key_policy: do NOT add a vault entry - that is a captain decision under the auth-keys charter
body: short Markdown pointer; the HTML attachment IS the report
artifact: /var/log/daily-infra-report/infra-report-<UTCstamp>.html
note: Replaced SMTP/mail on 2026-09-26 by captain decision. Removes the Google
dependency entirely; closes daily-digest-mail-transport-20260921.
exit_semantics:
'1': missing PVE_TOKEN, unreachable Proxmox probe, or failed email send - raises an alert
'0': healthy, or a deliberate DEGRADED leg where the email credential is absent
and the report is still produced
'1': missing PVE_TOKEN, unreachable Proxmox probe, missing/rejected Zulip
credential, or a failed upload/post - raises an alert
'0': healthy delivery only - there is no degraded delivery leg any more
depends_on: []
last_run: null
last_status: null
daily-health-digest.prose.md
+48 -32
View File
@@ -16,11 +16,12 @@ description: >
Exit-code semantics (as they actually behave, verified 2026-09-25):
* missing PVE_TOKEN, or an unreachable Proxmox probe -> exit 1 + alert
* missing EMAIL credential -> deliberate DEGRADED leg, exit 0, report still
produced
* email send failure -> exit 1 (a delivery fault, not a code defect)
* missing or rejected Zulip credential -> exit 1 (delivery is the only
output path, so it is a real failure, not a degraded leg)
* delivery failure -> exit 1, and the report body is printed AND persisted
so the content is never swallowed
version: 1.0.0
version: 2.0.0
---
## Purpose
@@ -93,14 +94,13 @@ $ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json
EXIT=0
```
and in mail mode:
and in delivery mode:
```
Sending email...
✅ All legs fully credentialed
📋 Summary:
Proxmox: 5/5 nodes online
VMs/CTs: 22/22 running
report ready: 16208 chars of HTML (delivered as a file attachment)
Sending to the captain's Zulip DM...
✅ Delivered to Zulip DM (user 9), message id 86221, attachment 16208 bytes
at /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html
```
Healthy means: every probe reports `ok`, `nodes_online == node_count`, and the
@@ -115,9 +115,8 @@ Verified on 2026-09-25 by running each case deliberately.
| all probes reachable, email sent | 0 | — | healthy |
| **missing `PVE_TOKEN`** | **1** | yes | `PROBE FAILURES: proxmox: node list unreachable (PVE_TOKEN missing or API down)`, and `cluster resources unreachable` |
| **Proxmox probe unreachable** | **1** | yes | same path as above; `pve_probe_status: unreachable` |
| **missing `EMAIL_PASSWORD`** | **0** | no | deliberate **DEGRADED** leg (`credential-missing: EMAIL_PASSWORD`); the report is still produced |
| **email send fails** | **1** | yes | e.g. Gmail `534 5.7.9 Application-specific password required` |
| degraded legs present (non-email) | 0 | no | logged under `⚠️ Degraded legs` |
| **missing/rejected Zulip credential** | **1** | yes | delivery is the only output path; report printed and persisted |
| **upload or message post fails** | **1** | yes | report printed and persisted; message names which step failed |
The distinction is deliberate and must not be flattened:
@@ -130,22 +129,31 @@ The distinction is deliberate and must not be flattened:
`PROBE_FAILURES` and `DEGRADED_LEGS` are separate lists for exactly this
reason. Do not merge them.
## Email-delivery dependency
## Delivery: Zulip DM carrying the report as an HTML ATTACHMENT
Delivery is a **credential dependency, not a code path**. The producer
authenticates to `smtp.gmail.com:587` as `jtabiri@gmail.com` with
`EMAIL_PASSWORD` from the vault and sends to `jerome@sysloggh.com`.
Captain's decision 2026-09-26, clarified the same day: the digest is delivered to
his **Zulip DM (user id 9)** from `abiba-bot@chat.sysloggh.net`, as an **HTML
FILE** — an attachment, not HTML rendered in the message body and not a Markdown
translation of it.
* Since that Google account has two-step verification, `EMAIL_PASSWORD` must be
a Google **app password**, not the account password.
* As of 2026-09-25 delivery is **failing** with
`534 5.7.9 Application-specific password required`; the fix is for the
captain to generate a fresh app password and place it in Infisical
(`infrastructure/production`) as `EMAIL_PASSWORD`.
* **A delivery failure is not a code defect.** Investigation of a failed send
should start at the credential, not the script. Chasing it as a code bug
wastes the effort; verify the credential path first with `--test-email`.
* Tracked separately as `daily-digest-mail-transport-20260921`.
* the styled dashboard is built exactly as before and written to
`/var/log/daily-infra-report/infra-report-<UTCstamp>.html`;
* it is uploaded through `POST /api/v1/user_uploads`;
* the **message body stays short Markdown** — subject line, top-line status
(nodes online, guests running, any degraded legs), and a link to the
attachment. The attachment IS the report; the body does not reproduce it.
This removes the Google dependency entirely: **no SMTP, no `EMAIL_PASSWORD`, no
app password, nothing to rotate.** `daily-digest-mail-transport-20260921` is
closed under this option.
The **10,000-character message cap does not apply** — it bounds message TEXT
only, and the report travels as a file. Do not shrink the report to fit it.
The credential is abiba-bot's Zulip key already on the execution host at
`/root/.pi/agent/extensions/zulip/.env` (`ABIBA_ZULIP_API_KEY`, mode 600,
root-readable). **Do not place a new credential in the vault** — under the
auth-keys charter that is a captain decision.
## What counts as a failure
@@ -153,10 +161,18 @@ A run FAILS (exit 1) when the report cannot be trusted or delivered:
* any probe is unreachable, so a section would silently be empty;
* `PVE_TOKEN` is missing;
* the email send fails.
* the Zulip credential is missing or rejected, or the upload/post fails.
A run is DEGRADED (exit 0, report still produced) when a non-load-bearing
credential is absent, currently only `EMAIL_PASSWORD`.
There is **no degraded delivery leg any more**. Delivery is the only output
path, so a missing credential is a failure rather than a survivable degradation —
the previous "missing `EMAIL_PASSWORD` still exits 0" rule is retired with the
mail transport.
**A delivery failure must never swallow the report.** On failure the script
prints the report body to stdout *and* leaves the HTML artifact on disk, so the
content is always recoverable from the run log. That closes the queued defect
where a failed send printed only the transport error and the report never
surfaced.
## Failure behaviour
@@ -175,7 +191,7 @@ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json \
| grep -E 'pve_probe_status|node_count|nodes_online'
# delivery path
infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-email
infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-zulip
```
Regression tests: `tests/test_daily_infra_report.py` (7 tests). Four of them
@@ -183,5 +199,5 @@ fail against the pre-fix script, which is what makes them bite.
## Maintains
- daily-infra-dashboard: { status: "degraded", reason: "email credential", last_check: timestamp }
- daily-infra-dashboard: { status: "ok|undelivered", transport: zulip-dm-attachment, last_check: timestamp }
- pve-probe: { status: "ok|unreachable", last_check: timestamp }
scripts/daily-infra-report.py
+145 -36
View File
@@ -688,42 +688,152 @@ Proxmox: {r.get('pve_probe_status', 'ok')} ({r['nodes_online']}/{r['node_count']
return html
# ── Send Email ──
# ── Delivery: Zulip DM carrying the report as an HTML ATTACHMENT ──
#
# Captain's decision, clarified 2026-09-26: the report is sent as an HTML FILE,
# i.e. an attachment - NOT HTML rendered in the message body, and NOT a Markdown
# translation of it. So the styled dashboard is built exactly as before, uploaded
# through Zulip's file-upload API, and the message body stays short: subject,
# top-line status, and a pointer to the attachment.
#
# This removes the Google dependency entirely (no SMTP, no EMAIL_PASSWORD).
# The 10,000-character message cap does not apply: it bounds message TEXT only,
# and the report travels as a file.
def send_email(html_content, subject_prefix=""):
FROM = "abiba@sysloggh.com"
TO = "jerome@sysloggh.com"
SUBJECT = f"{subject_prefix}{'🏗️ Infrastructure Report — ' + DATE_STR}"
msg = MIMEMultipart("alternative")
msg["From"] = FROM
msg["To"] = TO
msg["Subject"] = SUBJECT
msg.attach(MIMEText("Infrastructure report in HTML format — enable images to view.", "plain"))
msg.attach(MIMEText(html_content, "html"))
ZULIP_SITE = "https://chat.sysloggh.net"
ZULIP_BOT_EMAIL = "abiba-bot@chat.sysloggh.net"
CAPTAIN_USER_ID = 9
ZULIP_KEY_FILE = "/root/.pi/agent/extensions/zulip/.env"
REPORT_ARTIFACT_DIR = "/var/log/daily-infra-report"
def zulip_key():
"""abiba-bot's Zulip key, from the env or the on-host 600 file."""
key = os.environ.get("ABIBA_ZULIP_API_KEY")
if key:
return key.strip()
try:
EMAIL_PASSWORD = os.environ.get("EMAIL_PASSWORD") or os.environ.get("SMTP_PASSWORD") or os.environ.get("MAIL_PASSWORD")
if not EMAIL_PASSWORD:
print(" ⚠️ Degraded leg: credential-missing: EMAIL_PASSWORD (or SMTP_PASSWORD/MAIL_PASSWORD)", file=sys.stderr)
DEGRADED_LEGS.append("credential-missing: EMAIL_PASSWORD")
return True, "✅ Email leg degraded (no credential) — report still produced"
GMAIL_EMAIL = "jtabiri@gmail.com"
server = smtplib.SMTP("smtp.gmail.com", 587)
server.starttls()
server.login(GMAIL_EMAIL, EMAIL_PASSWORD)
server.sendmail(FROM, [TO], msg.as_string())
server.quit()
return True, "✅ Email sent to jerome@sysloggh.com"
except Exception as e:
return False, f"❌ Email failed: {e}"
with open(ZULIP_KEY_FILE) as fh:
for line in fh:
if line.startswith("ABIBA_ZULIP_API_KEY="):
return line.split("=", 1)[1].strip()
except OSError:
return None
return None
def build_summary(r, filename, test=False):
"""Short Markdown body: subject, top-line status, pointer to the attachment.
Deliberately NOT a reproduction of the report - the attachment is the report.
"""
nodes = f"{r.get('nodes_online', 0)}/{r.get('node_count', 0)} nodes online"
guests = f"{r.get('running_vms', 0)}/{r.get('total_vms', 0)} guests running"
lines = [
("\U0001F9EA **TEST — **" if test else "") + "\U0001F3D7\uFE0F **Infrastructure Report — " + DATE_STR + "**",
f"**{nodes}** \u00b7 **{guests}** \u00b7 generated {TIME_STR}",
]
problems = []
if r.get("pve_probe_status") != "ok":
problems.append(f"\u274c Proxmox probe: {r.get('pve_probe_status')}")
if r.get("resources_probe_status") != "ok":
problems.append(f"\u274c Resources probe: {r.get('resources_probe_status')}")
lit = r.get("litellm", {}) or {}
checks = lit.get("checks", []) or []
if checks:
passed = sum(1 for c in checks if c.get("status") == "pass")
if passed != len(checks):
problems.append(f"\u274c LiteLLM: {passed}/{len(checks)} checks pass")
if not (r.get("zulip_ext", {}) or {}).get("connected"):
problems.append("\u274c Zulip extension: not connected")
for leg in DEGRADED_LEGS:
problems.append(f"\u26a0\uFE0F degraded: {leg}")
lines.append("\n".join(problems) if problems else "\u2705 All monitored services healthy")
lines.append(f"\U0001F4CE **Full report attached:** `{filename}`")
return "\n\n".join(lines)
def _curl(args, timeout=60):
r = subprocess.run(["curl", "-s", "-m", str(timeout)] + args,
capture_output=True, text=True)
try:
return json.loads(r.stdout or "{}"), r.stdout
except json.JSONDecodeError:
return {}, r.stdout
def _curl_json(args, timeout=90):
r = subprocess.run(["curl", "-s", "-m", str(timeout)] + args,
capture_output=True, text=True)
try:
return json.loads(r.stdout or "{}"), r.stdout
except json.JSONDecodeError:
return {}, r.stdout
def send_zulip(html_content, report, test=False):
"""Upload the styled HTML and post a short pointer to the captain's DM.
Returns (ok, message). On ANY failure the report body is also printed to
stdout and persisted to disk, so a delivery failure can never swallow the
content - the defect this folds in.
"""
os.makedirs(REPORT_ARTIFACT_DIR, exist_ok=True)
stamp = NOW.strftime("%Y%m%d-%H%M%S")
filename = f"infra-report-{stamp}.html"
html_path = os.path.join(REPORT_ARTIFACT_DIR, filename)
try:
with open(html_path, "w") as fh:
fh.write(html_content)
except OSError as e:
print(f" \u26a0\uFE0F could not persist report artifact: {e}", file=sys.stderr)
key = zulip_key()
if not key:
print(html_content) # never swallow the content
return False, ("\u274c Delivery FAILED: no Zulip credential "
"(ABIBA_ZULIP_API_KEY unset and "
f"{ZULIP_KEY_FILE} unreadable). Report persisted to {html_path}")
auth = ["-u", f"{ZULIP_BOT_EMAIL}:{key}"]
# 1. Upload the report as a file.
up, up_raw = _curl_json(auth + [
"-X", "POST", f"{ZULIP_SITE}/api/v1/user_uploads",
"-F", f"file=@{html_path};type=text/html",
])
if up.get("result") != "success" or not up.get("uri"):
print(html_content)
return False, (f"\u274c Delivery FAILED at upload: {up.get('msg') or up_raw[:160]} "
f"(report persisted to {html_path})")
uri = up["uri"]
size = os.path.getsize(html_path)
# 2. Post a short message pointing at it.
body = build_summary(report, filename, test=test)
link = f"[{filename}]({uri})"
body = body.replace(f"`{filename}`", link)
payload, raw = _curl_json(auth + [
"-X", "POST", f"{ZULIP_SITE}/api/v1/messages",
"-d", "type=private",
"-d", f"to=[{CAPTAIN_USER_ID}]",
"--data-urlencode", f"content={body}",
])
if payload.get("result") == "success":
return True, (f"\u2705 Delivered to Zulip DM (user {CAPTAIN_USER_ID}), "
f"message id {payload.get('id')}, attachment {size} bytes at {uri}")
print(html_content)
return False, (f"\u274c Delivery FAILED at message post: {payload.get('msg') or raw[:160]} "
f"(uploaded {uri}; report persisted to {html_path})")
# ── Main ──
if __name__ == "__main__":
is_test = "--test-email" in sys.argv
is_test = ("--test-email" in sys.argv) or ("--test-zulip" in sys.argv)
print(f"{'🧪 TEST MODE' if is_test else '📊'} Collecting infrastructure data...")
report = collect()
@@ -738,15 +848,14 @@ if __name__ == "__main__":
print(" Building dashboard...")
html = build_html(report)
print(f" report ready: {len(html)} chars of HTML (delivered as a file attachment)")
if is_test:
prefix = "🧪 TEST — "
print(" Sending test email...")
print(" Sending TEST message to the captain's Zulip DM...")
else:
prefix = ""
print(" Sending email...")
ok, msg = send_email(html, subject_prefix=prefix)
print(" Sending to the captain's Zulip DM...")
ok, msg = send_zulip(html, report, test=is_test)
print(f" {msg}")
# Show summary