fix(daily-infra-report): resolve PVE token from env; make a dead probe non-silent #133
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: SyslogSolution/prose-contracts#133
Reference in New Issue
Block a user
Problem
The Proxmox leg of the daily digest has been reporting nothing while exiting 0.
Root cause: the auth header was a literal placeholder string:
which was sent verbatim. The API rejected it,
pve_get()returnedNone, and the report renderednode_count=0 / nodes_online=0withpve_probe_status=unreachablewhile still exiting 0. A monitoring gap that looks like a healthy run.Also: the vault key is
PVE_TOKEN, notPVE_API_TOKEN, so readingos.environunder the old name would not have found it either.Fix
pve_auth()resolves the token at call time fromPVE_TOKEN(injected byinfisical run --env=prod). Nothing hardcoded; a missing token raises.pve_get()builds the command inside itstryblock so a missing token degrades toNoneinstead of escaping as an unhandled exception.PROBE_FAILURESrecords an unreachable node/resources probe. Deliberately separate fromDEGRADED_LEGS: a missing credential stays exit 0 (existing intent), but a probe with no data now exits 1 in both the report and--jsonpaths.Measured effect (live)
pve_probe_statusunreachableoknode_countnodes_onlinetotal_vmsrunning_vmsNode detail:
amdpve=online, minipve=online, acerpve=online, ocupve=online, storepve=online.Tests
4 new regression tests. All 4 fail against the pre-fix script and pass after; the 3 pre-existing tests still pass (7/7).
Not fixed here (needs the captain)
The email leg fails:
EMAIL_PASSWORDin the vault is not a valid Gmail app password forjtabiri@gmail.com. That is a credential action, not a code change — the report is now generated correctly but cannot be delivered until a fresh app password is placed in Infisical.The Proxmox leg of the daily digest has been reporting NOTHING while exiting 0. Root cause: the auth header was a literal placeholder string, AUTH = "Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»" which was sent verbatim. The API rejected it, pve_get() returned None, and the report rendered node_count=0 / nodes_online=0 with pve_probe_status='unreachable' while still exiting 0. A monitoring gap that looks like a healthy run. Also: the vault key is PVE_TOKEN, not PVE_API_TOKEN, so even reading os.environ by the old name would not have found it. Fixes: * pve_auth() resolves the token at call time from PVE_TOKEN (injected by 'infisical run --env=prod'). Nothing is hardcoded; a missing token raises. * pve_get() builds the command inside its try block, so a missing token degrades to None instead of escaping as an unhandled exception. * PROBE_FAILURES records an unreachable node/resources probe. Probe failures are deliberately separate from DEGRADED_LEGS: a missing credential stays exit 0 (existing intent), but a probe with no data now exits 1 in both the report and --json paths, so it cannot pass unnoticed. Measured effect on the live host: pve_probe_status unreachable -> ok, node_count 0 -> 5, nodes_online 0 -> 5, total_vms 0 -> 22, running_vms 0 -> 22. Tests: 4 new regression tests; all 4 fail against the pre-fix script and pass after, and the 3 pre-existing tests still pass (7/7). Not fixed here (needs the captain): the email leg fails with '534 5.7.9 Application-specific password required' - EMAIL_PASSWORD in the vault is not a valid Gmail app password for jtabiri@gmail.com. That is a credential action, not a code change.