---
kind: function
name: daily-health-digest
description: >
  Produces the daily infrastructure dashboard for the whole estate — 5 Proxmox
  nodes, every VM/CT, Zulip, LiteLLM, Docker hosts, storage and NFS — and mails
  it as an HTML report.

  Dispatched by cron on CT 100 (abiba) at 10:30 UTC as a firstmate message to
  the ops lane, which executes the producer below. Until 2026-09-25 this ran
  with NO contract file at all, which is why the choice of execution copy was
  silently the operator's rather than the contract's.

  EXECUTION IS PINNED. The producer must be run from the clone named under
  "Execution pinning" — not from an agent working copy.

  Exit-code semantics (as they actually behave, verified 2026-09-25):
    * missing PVE_TOKEN, or an unreachable Proxmox probe -> exit 1 + alert
    * missing or rejected Zulip credential -> exit 1 (delivery is the only
      output path, so it is a real failure, not a degraded leg)
    * delivery failure -> exit 1, and the report body is printed AND persisted
      so the content is never swallowed

version: 2.0.0
---

## Purpose

Give one daily, machine-collected picture of the estate so drift and outages
are seen the day they happen rather than when something breaks. It is a
*report*, not a repair: it changes nothing.

## Execution pinning

**Pinned execution path:**

```
/root/abiba-workspace/projects/prose-contracts/scripts/daily-infra-report.py
```

**Pinned clone:** `/root/abiba-workspace/projects/prose-contracts`

That is the cron's `FM_HOME` clone and the only stable, non-ephemeral copy.
The treehouse clone (`/root/.treehouse/agent-workspace-*/…/projects/prose-contracts`)
is a **per-agent working copy and must NOT be pinned or executed from** — it
drifts onto feature branches, which is exactly how a stale producer reported a
stale picture and nobody noticed.

See `docs/contract-execution-pinning.md`. Schedule and alerting live in
`/etc/cron.d/contract-runner` on CT 100:

```
30 10 * * * FM_HOME=/root/abiba-workspace /root/abiba-workspace/bin/fm-send.sh ops "run contract: daily-health-digest" > /dev/null 2>&1
```

Invocation (credentials come from the vault; never inline them):

```bash
cd /root/abiba-workspace/projects/prose-contracts
infisical run --env=prod -- python3 scripts/daily-infra-report.py
```

## Output shape

Modes:

| invocation | effect |
| --- | --- |
| *(none)* | collect, build the HTML dashboard, email it |
| `--test-email` | same but with a `🧪 TEST —` subject prefix |
| `--json` | print the collected data as JSON to stdout and **send no email** |

`--json` emits a single object with these top-level keys (observed on a live
run 2026-09-25):

| key | type | meaning |
| --- | --- | --- |
| `nodes` | object (5) | per-node cpu/ram/disk/uptime/status |
| `node_count`, `nodes_online` | int | Proxmox node totals |
| `pve_probe_status`, `resources_probe_status` | `ok`\|`unreachable` | probe outcome |
| `total_vms`, `running_vms`, `stopped_vms`, `vms_by_node` | — | guest inventory |
| `storage`, `nfs` | array | datastore and mount usage |
| `litellm` | object | inference checks |
| `zulip_ext` | object | Zulip queue/serving state |
| `agents` | object | per-agent health |
| `docker_vm`, `docker_syslog`, `docker_netbird`, `endpoints` | object/array | Docker hosts and probed endpoints |

## What a healthy run looks like

```
$ infisical run --env=prod -- python3 scripts/daily-infra-report.py --json
  "nodes_online": 5,  "node_count": 5,  "pve_probe_status": "ok",
  "resources_probe_status": "ok",  "running_vms": 22,  "total_vms": 22
EXIT=0
```

and in delivery mode:

```
   report ready: 16208 chars of HTML (delivered as a file attachment)
   Sending to the captain's Zulip DM...
   ✅ Delivered to Zulip DM (user 9), message id 86221, attachment 16208 bytes
      at /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html
```

Healthy means: every probe reports `ok`, `nodes_online == node_count`, and the
email leg reports a successful send.

## Exit-code semantics — as they actually behave

Verified on 2026-09-25 by running each case deliberately.

| condition | exit | alert | notes |
| --- | --- | --- | --- |
| all probes reachable, email sent | 0 | — | healthy |
| **missing `PVE_TOKEN`** | **1** | yes | `PROBE FAILURES: proxmox: node list unreachable (PVE_TOKEN missing or API down)`, and `cluster resources unreachable` |
| **Proxmox probe unreachable** | **1** | yes | same path as above; `pve_probe_status: unreachable` |
| **missing/rejected Zulip credential** | **1** | yes | delivery is the only output path; report printed and persisted |
| **upload or message post fails** | **1** | yes | report printed and persisted; message names which step failed |

The distinction is deliberate and must not be flattened:

* A **missing PVE token or an unreachable probe is a real failure** — the report
  would otherwise claim zero nodes and still look successful. That was the
 2026-09-25 silent-zero defect (fixed in PR #133); it now exits 1.
* A **missing email credential is survivable** — the report is still produced
  and is still useful. It is a `DEGRADED` leg and exits 0 by design.

`PROBE_FAILURES` and `DEGRADED_LEGS` are separate lists for exactly this
reason. Do not merge them.

## Delivery: Zulip DM carrying the report as an HTML ATTACHMENT

Captain's decision 2026-09-26, clarified the same day: the digest is delivered to
his **Zulip DM (user id 9)** from `abiba-bot@chat.sysloggh.net`, as an **HTML
FILE** — an attachment, not HTML rendered in the message body and not a Markdown
translation of it.

* the styled dashboard is built exactly as before and written to
  `/var/log/daily-infra-report/infra-report-<UTCstamp>.html`;
* it is uploaded through `POST /api/v1/user_uploads`;
* the **message body stays short Markdown** — subject line, top-line status
  (nodes online, guests running, any degraded legs), and a link to the
  attachment. The attachment IS the report; the body does not reproduce it.

This removes the Google dependency entirely: **no SMTP, no `EMAIL_PASSWORD`, no
app password, nothing to rotate.** `daily-digest-mail-transport-20260921` is
closed under this option.

The **10,000-character message cap does not apply** — it bounds message TEXT
only, and the report travels as a file. Do not shrink the report to fit it.

The credential is abiba-bot's Zulip key already on the execution host at
`/root/.pi/agent/extensions/zulip/.env` (`ABIBA_ZULIP_API_KEY`, mode 600,
root-readable). **Do not place a new credential in the vault** — under the
auth-keys charter that is a captain decision.

## What counts as a failure

A run FAILS (exit 1) when the report cannot be trusted or delivered:

* any probe is unreachable, so a section would silently be empty;
* `PVE_TOKEN` is missing;
* the Zulip credential is missing or rejected, or the upload/post fails.

There is **no degraded delivery leg any more**. Delivery is the only output
path, so a missing credential is a failure rather than a survivable degradation —
the previous "missing `EMAIL_PASSWORD` still exits 0" rule is retired with the
mail transport.

**A delivery failure must never swallow the report.** On failure the script
prints the report body to stdout *and* leaves the HTML artifact on disk, so the
content is always recoverable from the run log. That closes the queued defect
where a failed send printed only the transport error and the report never
surfaced.

## Failure behaviour

* Non-zero exit with the alert text above; on the scheduled path the dispatch is
  a firstmate message, so the ops lane sees it and reports it.
* On a probe failure the report must **not** be treated as evidence about the
  estate — a `0/0` Proxmox section means "could not look", not "nothing there".
  That reading is why the 2026-09-25 defect went unnoticed.

## Verification

```bash
# data path, no email
cd /root/abiba-workspace/projects/prose-contracts
infisical run --env=prod -- python3 scripts/daily-infra-report.py --json \
  | grep -E 'pve_probe_status|node_count|nodes_online'

# delivery path
infisical run --env=prod -- python3 scripts/daily-infra-report.py --test-zulip
```

Regression tests: `tests/test_daily_infra_report.py` (7 tests). Four of them
fail against the pre-fix script, which is what makes them bite.

## Maintains

- daily-infra-dashboard: { status: "ok|undelivered", transport: zulip-dm-attachment, last_check: timestamp }
- pve-probe: { status: "ok|unreachable", last_check: timestamp }