100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
root	ba38efcd75	test(search): quality guard so the ranking layer cannot silently rot ... Extends search-stack-visibility with a ranking assertion: for the fixed query set, no config demote_domains host may appear in the top 3, and the known non-answers (bestbuy.com, merriam-webster.com) must not be returned at all. Without this the layer could rot back to raw engine ordering unnoticed - the same way the endpoint colours silently rotted before 2026-09-26. It reads the demote list from the SAME config the layer uses, so the guard cannot drift from the policy it is guarding. Live: 'ok: no demoted host in the top 3; no banned non-answer returned'; visibility contract still PASSES end to end.	2026-09-26 15:44:10 +00:00
root	0d30091f62	feat(search): agent-consumption layer - dedupe, filter, rerank, extract ... Raw multi-engine aggregation had no dedupe, no filtering and no reranking. Measured 2026-09-26: 'best practices agent context management' returned bestbuy.com and merriam-webster.com, plus 4 content farms, with medium.com twice; 'proxmox thin pool metadata exhaustion recovery' put four SEO blogs ABOVE the real Proxmox forum threads. Identical queries also ranked DIFFERENTLY between runs, which is why the fix is deterministic rather than trusting the engines. scripts/search-agent-consume.py: 1. dedupe by normalised URL (tracking params and fragments stripped) 2. drop non-answers - shopping/dictionary hosts, navigational host roots, search/shopping/cart/login paths and query keys 3. demote content farms and promote primary sources 4. STABLE sort (score desc, then original position) so runs are reproducible 5. extract page text for the top N via Firecrawl POST /v1/scrape under an explicit character budget, so an agent gets usable material in ONE call 6. emit stable JSON with engine provenance and source_type Policy is config, not code: config/search-ranking.yaml holds demote_domains, prefer_domains, non_answer rules and the extraction budget, so it is reviewable and changeable without touching the module. Content farms are DEMOTED rather than dropped so a useful hit is not lost, it just cannot outrank a primary. A '/products/' path rule was REMOVED after the before/after run caught it dropping docs.digitalocean.com/products/inference/... - a legitimate docs page. Shopping is caught by the host list instead, which has no such false positive. Measured: 'best practices...' top 8 becomes anthropic, langchain, jetbrains, blog.jetbrains, docs.langchain, reddit, cursor, reddit - no content farm. 'proxmox thin pool...' moves the forum threads from positions 5-9 to 1-4. Extraction: 5 items, 12000 chars of 12000 budget, 0 failures, 5.28s; whole run 6.4s wall. Contract: search-agent-consumption.prose.md, including the honest reachability gap - the pi MCP search server's shape is not ours to change, so this layer is NOT wired into it.	2026-09-26 15:44:09 +00:00
root	0a41a2d584	fix(daily-digest): probe Firecrawl on its real liveness path and classify endpoints per fleet policy ... Folds into the same branch as the Zulip delivery change, as instructed. FINDING 1 - the Firecrawl probe path was wrong; the service is fine. scripts/daily-infra-report.py probed http://192.168.68.7:3002/health, which Firecrawl does not serve - it 404s. The root answers 200 with {"message":"Firecrawl API",...}. Live 2026-09-26: Firecrawl(/) -> 200 Firecrawl(/health) -> 404 <- what the report was showing The probe is now the root, which is its liveness endpoint. FINDING 2 - the Network Endpoints classification was wrong twice over. It read: color = green if code in (200,302,401) else (yellow if code >= 400 else red). Two defects: (a) it ignored the fleet's own probe policy, codified 2026-09-14 in the monitoring contracts: ANY HTTP status proves the service answered, so the service is ALIVE, and only a failed CONNECTION is a failed probe. A 404 from a wrong path is not a service fault. (b) was a STRING comparison. Reproduced: 301 -> red (a live redirect rendered as a failure), 404 -> yellow, 500 -> yellow (a real server error softened to a warning). Replaced with classify_endpoint(), which returns: any 2xx/3xx/4xx -> green 'alive' (code still shown) 5xx -> yellow 'server error' (kept distinct from 4xx, as asked) 000/no answer -> red 'no connection' Verified against the live endpoints after the change: Gitea 200, Authentik 302, Zulip 302, Pulse 200, Proxmox 200, SearXNG 200, Firecrawl 200 - all green/alive; the only red state is a genuine no-connection. ALSO CHECKED, as asked: scripts/search-stack-check.py does NOT depend on the wrong route. It POSTs to {FIRECRAWL_URL}/v1/scrape with formats=[markdown], and that path really works - live POST returned HTTP 200 and 180 chars of markdown for https://example.com. It was never using /health. prose-lint: PASSED.	2026-09-26 15:36:38 +00:00
root	de32f54337	feat(daily-digest): deliver via Zulip DM as an HTML attachment; drop mail entirely ... Captain's decision 2026-09-26, clarified the same day: the digest is delivered to his Zulip DM (user id 9) from abiba-bot as an HTML FILE - an attachment, not HTML rendered in the message body and not a Markdown translation of it. Closes daily-digest-mail-transport-20260921; the Google dependency is gone (no SMTP, no EMAIL_PASSWORD, no app password, nothing to rotate). WHAT CHANGES * scripts/daily-infra-report.py: send_email() is replaced by send_zulip(), which writes the styled dashboard to /var/log/daily-infra-report/infra-report-<ts>.html, uploads it via POST /api/v1/user_uploads, then posts a SHORT Markdown pointer to user 9. The message body carries subject, top-line status and the attachment link; it does not reproduce the report. * the 10,000-character cap is irrelevant here - it bounds message TEXT only, and the report travels as a file, so nothing is shrunk to fit. * the key is abiba-bot's, already on the execution host at /root/.pi/agent/extensions/zulip/.env (mode 600). No vault entry was added: under the auth-keys charter that is a captain decision. * daily-health-digest.prose.md -> v2.0.0 and contract-registry.yaml updated: transport, healthy/degraded definitions, and exit codes now match observed behaviour. There is NO degraded delivery leg any more - delivery is the only output path, so a missing or rejected key is a real failure (exit 1). * queued defect folded in: a failed delivery used to print only the transport error while the report body never surfaced. Now the HTML is printed to stdout AND persisted on every failure, and the message names which step failed. EVIDENCE (all against the live stack) * real send: message id 86221 to user 9, attachment 16208 bytes at /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html * the message is type=private, sender abiba-bot@chat.sysloggh.net, recipients [9, 21], body carries the top-line status and the attachment link, and does NOT contain a <table> - i.e. it does not reproduce the report * the attachment fetches HTTP 200, 16208 bytes, content-type text/html, starts with <!DOCTYPE html>, and contains <style>, <table> and 16 class="card" blocks - it opens as a standalone styled document * failure path: a bad key gives 'Delivery FAILED at upload: Malformed API key', EXIT=1, the HTML is printed to stdout and persisted to disk * scheduled path: the run's own output is pasted in the PR prose-lint: PASSED (19 warnings); secret scan clean.	2026-09-26 15:35:36 +00:00
root	226f2ad55e	fix: reconcile the health-logs self-heal contracts, and add a dead-man's-switch ... relay-785 / self-heal-contract-stale-script-pointers-20260924. gpu/ had been silent since 2026-09-14T18:02:03Z (12 days) and pm2/ had never posted a run. FINDING - the gpu-self-heal executor was never lost, only its schedule was. The brief concluded the mechanism was gone. It is not: on CT 116 /opt/inference-harness/scripts/gpu-self-heal.py exists (mtime 2026-09-21 11:43), is self-posting via gitea-logger.sh, and runs cleanly - I ran it and it posted health-logs/gpu/gpu-self-heal-20260926-144618.json. What was missing was the cron entry, dropped during a CT 116 /etc/cron.d rework on 2026-09-21. So this is a schedule restoration, not a resurrection. DECISIONS * gpu-self-heal -> RESTORE. Schedule re-added as /etc/cron.d/gpu-self-heal on CT 116, matching the observed historical cadence (:02 past 0/6/12/18). Proven with a REAL unattended cron fire (temporary */1 entry, removed after): gpu-self-heal-20260926-144802.json committed 14:48:03Z. The contract now names the executor, schedule, log and posting, which it previously did not. * pm2 health-logs posting -> RETIRE the claim. pm2-self-heal.prose.md required appending to health-logs/pm2/ as a 'hard rule', but scripts/pm2-self-heal.sh contains no Gitea or push code and never did; the directory has held only its init commit since 2026-07-28. Corrected to point at the contract-runner's durable per-run logs and failure note instead of adding a second, redundant posting path. DEAD-MAN'S-SWITCH - scripts/health-log-freshness.py + contract. Absence of logs must raise an alarm, and the producer cannot raise it, so this runs on CT 100, a different host from the producers, and fails when the newest health-logs/gpu entry is older than 12h (litellm 18h). Verified it would have caught the real gap: evaluated at 2026-09-20 the newest gpu/ entry was 126h old against a 12h limit -> STALE. Bugs found and fixed while testing, each caught by a test that bit: * the documented HEALTH_LOG_MAX_AGE_* override was never implemented; * a Gitea password held under GITEA_TOKEN was sent as an API token -> HTTP 401; now every candidate auth is tried and the first that works is used; * the ~/.git-credentials fallback filtered on GITEA_URL's host, which missed whenever GITEA_URL pointed at the internal IP -> zero candidates. Exit codes: 0 fresh, 1 stale-or-unreadable, 2 cannot run. A directory that cannot be read is a failure, never a skip. Evidence: live PASS; stale override names the directory and producer; no credential -> exit 2; whole thing runs green through contract-run.sh. prose-lint: PASSED.	2026-09-26 14:50:52 +00:00
root	778424acd4	Merge remote-tracking branch 'origin/master' into fix/land-revision-preflight-guard-20260925	2026-09-25 11:29:30 +00:00
root	c5a6dcd42a	fix(revision-preflight): default to warn, name the refusal class, bound the fetch ... Bounded correction round on PR #134 after a PASS-WITH-FINDINGS review whose Finding 4 is High. The guard's purpose and its fail-closed fix stand; the problem was that with 'enforce' as the default it gates EVERY scheduled contract, and three legitimate states produce a refusal - a clone legitimately ahead of origin/master mid-review, a detached HEAD, and an offline or failed fetch - so any of them would turn the fleet's monitoring into withheld verdicts. That risk outweighs the staleness the guard catches. 1. DEFAULT IS NOW 'warn'. 'enforce' remains available and documented. The criteria for flipping the default later are written into the doc as a decision with evidence - a sustained window (30 days / 200+ runs) with zero mismatch:* and zero cannot-verify:* refusals, no fetch blips, and a pinned clone demonstrably kept current - explicitly as its own change, not a silent flip. 2. 'COULD NOT CHECK' IS NOW DISTINGUISHABLE FROM 'THIS COPY IS WRONG'. Every non-zero exit prints a machine-readable REASON=<class> line: cannot-verify:fetch-failed | cannot-verify:ref-unresolvable (exit 2) mismatch:path-absent | mismatch:content mismatch:detached-head | mismatch:clone-ahead (exit 1) detached-head and clone-ahead are named separately because they are legitimate states, far less alarming than a hand-edited file. clone-ahead requires HEAD to be STRICTLY ahead; an uncommitted edit on a commit that IS the ref is a plain content mismatch (my own first cut got this wrong and the new test 7d caught it). 3. THE DEFAULT FETCH IS BOUNDED: --fetch-timeout, default 20s, 0 = unbounded, and a missing 'timeout' binary is itself a cannot-verify rather than an unbounded fetch inside a scheduled contract. 4. TEST COVERAGE ADDED for every new class: fetch failure, fetch timeout (asserted to return promptly under a 1s bound), unresolvable ref, detached HEAD, clone-ahead, genuine content mismatch, and the contract-run.sh default. The pre-fix draft fixture comparisons are kept: 31 passed, 0 failed. 5. MERGE-TIME SEQUENCE documented: fast-forward /opt/contract-runner, confirm clean, prove a contract runs and reports. Baseline recorded as of today - firstmate has already fast-forwarded it to 9faffe4 - with the note that an untracked file blocks a fast-forward even when byte-identical. Live behaviour re-verified on the real runner path: default: REASON=mismatch:clone-ahead -> 'continuing because ...=warn' -> VERDICT: PASS, exit 0 enforce: REASON=mismatch:clone-ahead -> 'VERDICT WITHHELD: mismatch:clone-ahead', exit 2 MANDATORY CHECKS (master went red once from a credential-SHAPED string, so these are now run on every shippable branch): bash scripts/prose-lint.sh -> LINT PASSED (18 warning(s)) secret scan -> secret scan clean (tree; 34 allowlisted, 24 inert value(s) ignored); No committed credentials shellcheck revision-preflight.sh -> clean shellcheck test_revision_preflight.sh -> clean shellcheck contract-run.sh -> SC2034 x1, SC2086 x2 - byte-identical on master, i.e. pre-existing, none introduced tests/test_probe_drift.py::test_prose_lint_accepts_report_format_with_provenance fails both before and after this branch (it runs prose-lint from a temp CWD and cannot find its sibling secret-scan.sh). Pre-existing, unrelated, not fixed here.	2026-09-25 11:29:15 +00:00
root	cd00cd0475	feat: add the missing daily-health-digest contract and register it ... Backlog row daily-health-digest-contract-missing-20260925. The digest has been dispatched on a schedule with NO contract file at all - no *daily*.prose.md, absent from contract-registry.yaml, the only reference anywhere being the CT100 cron line. That absence is why the choice of execution copy was silently the operator's, and how a stale clone could run the check unnoticed. New daily-health-digest.prose.md states: * the PINNED execution path /root/abiba-workspace/projects/prose-contracts/ scripts/daily-infra-report.py and the pinned clone - the cron's FM_HOME clone, the only stable non-ephemeral copy; the treehouse clone is a per-agent working copy and must NOT be pinned; * the output shape (all 18 top-level --json keys) and what a healthy run is; * the exit-code semantics AS THEY ACTUALLY BEHAVE, verified case by case: missing PVE_TOKEN or an unreachable probe exits 1 and raises an alert, while a missing EMAIL credential is a deliberate DEGRADED leg that still exits 0 and still produces the report. PROBE_FAILURES and DEGRADED_LEGS are separate lists on purpose and must not be merged; * the email-delivery dependency, that EMAIL_PASSWORD must be a Google app password, that it is failing with 534 5.7.9 as of 2026-09-25, and that a delivery failure is a credential dependency rather than a code defect; * what counts as a failure versus degraded. Registered in contract-registry.yaml (contracts entry plus index.by_category .monitoring and index.by_domain.infrastructure). Verified: YAML parses, 31 contracts, exactly one daily-health-digest entry.	2026-09-25 11:08:28 +00:00
root	819cd53bce	fix(lint): restore the repo secret scan, which PR #133 broke ... Master's lint is RED right now, and it is my doing. at 483a66b (before #133): prose-lint -> LINT PASSED at 9d64b0b (after #133): prose-lint -> LINT FAILED, 4 credential-shaped strings The regression came from #133's new pve_auth() work. The secret scanner flags the literal header shape PVEAPIToken=<value> (rule proxmox-token), and three occurrences landed in the tree: scripts/daily-infra-report.py the f-string building the auth header tests/test_daily_infra_report.py a docstring quoting the old placeholder tests/test_daily_infra_report.py a synthetic token in an assertion Fixed without allowlisting anything, because none of these is a credential: * the header prefix becomes PVE_AUTH_HEADER = "PVEAPIToken=", a constant ending at '=' so the scanner's pattern (which needs a character after '=') cannot match, and the f-string no longer contains the literal; * the test docstring no longer reproduces the old placeholder verbatim; * the test builds its expected value from the constant plus a local sample variable instead of embedding a credential-shaped literal. Behaviour is unchanged and re-verified: with PVE_TOKEN unset the script still exits 1 with the probe failure, and with the vault token it still reports pve_probe_status ok / node_count 5 / nodes_online 5. before: LINT FAILED — 4 credential-shaped strings after: LINT PASSED (18 warnings)	2026-09-25 11:08:28 +00:00
root	574cb99d76	feat: land the revision-preflight guard, fixed and wired into contract execution ... A contract verdict is only meaningful if it came from the merged copy. The fleet has been bitten three times on 2026-09-25 (a clone parked on a merged feature branch while executing from another clone; a script copied into the runner clone by hand; a stale local origin/master making an ancestry check report unlanded work). The control for this existed as an untracked draft and protected nobody, because it was entirely fail-open. Defect in the draft, preserved verbatim as tests/fixtures/revision-preflight.prefix.sh: git -C "$CLONE" show "origin/master:$(basename "$SCRIPT")" basename drops the scripts/ prefix, so for any script under scripts/ it queried the repo root, failed, took the "warn but don't block" branch and exited 0 - passing a script that exists in no revision at all. Reproduced: pre-fix + scripts/demo.sh under scripts/ -> 'could not resolve', EXIT=0 pre-fix + a script in no revision -> EXIT=0 Fixed guard (scripts/revision-preflight.sh): * resolves the repo-relative path inside the clone, so scripts/ paths resolve; * FAILS CLOSED - a path absent from the ref, an unresolvable ref, or a failed fetch is a failure, never a warning; * fetches the remote by default, because a stale local ref would otherwise pass a stale script as current; --no-fetch states the assumption instead of hiding it. Wiring (scripts/contract-run.sh): before executing, the wrapper runs the guard against the clone it lives in. Default CONTRACT_REVISION_PREFLIGHT=enforce withholds the verdict, alerts and exits 2 on mismatch; =warn logs and continues; =off skips. Verified live: match -> contract proceeds and PASSes; mismatch -> 'VERDICT WITHHELD', exit 2; =warn -> continues. Pinning (docs/contract-execution-pinning.md): every contract pins the clone contract-run.sh lives in - the deployed runner being /opt/contract-runner on CT 100. Documented that daily-health-digest has no contract file at all, which is why its execution copy was silently operator-chosen. Tests: tests/test_revision_preflight.sh, 15 assertions over a throwaway clone with a real bare remote. It runs the pre-fix draft against the same cases and shows it passing a ghost script, so the tests provably bite. shellcheck: scripts/revision-preflight.sh and the new test are clean. The three findings remaining in contract-run.sh (SC2086 x2, SC2034) are pre-existing and byte-identical on master.	2026-09-25 11:04:29 +00:00
root	cdc7ad2c79	fix(daily-infra-report): resolve PVE token from env; make a dead probe non-silent ... The Proxmox leg of the daily digest has been reporting NOTHING while exiting 0. Root cause: the auth header was a literal placeholder string, AUTH = "Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»" which was sent verbatim. The API rejected it, pve_get() returned None, and the report rendered node_count=0 / nodes_online=0 with pve_probe_status='unreachable' while still exiting 0. A monitoring gap that looks like a healthy run. Also: the vault key is PVE_TOKEN, not PVE_API_TOKEN, so even reading os.environ by the old name would not have found it. Fixes: * pve_auth() resolves the token at call time from PVE_TOKEN (injected by 'infisical run --env=prod'). Nothing is hardcoded; a missing token raises. * pve_get() builds the command inside its try block, so a missing token degrades to None instead of escaping as an unhandled exception. * PROBE_FAILURES records an unreachable node/resources probe. Probe failures are deliberately separate from DEGRADED_LEGS: a missing credential stays exit 0 (existing intent), but a probe with no data now exits 1 in both the report and --json paths, so it cannot pass unnoticed. Measured effect on the live host: pve_probe_status unreachable -> ok, node_count 0 -> 5, nodes_online 0 -> 5, total_vms 0 -> 22, running_vms 0 -> 22. Tests: 4 new regression tests; all 4 fail against the pre-fix script and pass after, and the 3 pre-existing tests still pass (7/7). Not fixed here (needs the captain): the email leg fails with '534 5.7.9 Application-specific password required' - EMAIL_PASSWORD in the vault is not a valid Gmail app password for jtabiri@gmail.com. That is a credential action, not a code change.	2026-09-25 10:34:07 +00:00
root	8b2eba4f7a	docs+check: correct DDG status, explain silent-zero semantics, credit google cse ... Follow-up corrections after review: 1. DuckDuckGo is NOT fixed. The VPS fallback egress has since been flagged by DuckDuckGo too (HTTP 202 + challenge markers), so it reports CAPTCHA on both paths. The contract and script docstring now say so instead of claiming a fix that had already expired. It stays enabled as best-effort coverage so a recovery shows up as a contribution. 2. The relationship between silent zeros and the verdict is now explicit in both the script output and the contract: an enabled expected engine that contributes zero with no error is REPORTED, not fatal. Only the <SEARCH_CHECK_MIN_ENGINES> floor and the extraction leg fail the run. This is deliberate - de-duplication and query-shape make a zero non-probative. 3. Recorded that 'google cse' uses a THIRD PARTY's public search-engine id hardcoded in the SearXNG build, not a key we own; its quota and availability are outside our control, and our own free key would need a wrapper (not built). VPS forward proxy is now a real service: /opt/fwd-proxy docker compose with restart: unless-stopped, a healthy healthcheck, and Docker enabled at boot.	2026-09-25 01:19:27 +00:00
root	040fecef3e	feat: multi-engine search stack + visibility check ... Search stack (192.168.68.7) was effectively Bing-only: google served a JS shell, duckduckgo CAPTCHA'd from the house egress, and every other shipped engine returned a silent zero. Upgraded SearXNG to 2026.9.23 (same pinned digest as the image already pulled by other hosts) which uses browser impersonation, and routed DuckDuckGo through a VPS forward proxy over the existing WireGuard tunnel via a per-engine 'network'. Live result: bing, google cse, brave and yandex contribute on every query; duckduckgo is best-effort via the datacenter egress. Adds the visibility leg so a future regression cannot be silent: scripts/search-stack-check.py * two fixed queries; FAILS when fewer than two engines contribute, printing contributing engines and every unresponsive_engines entry * FAILS when Firecrawl extraction returns empty markdown or errors * reports silent-zero engines explicitly scripts/contract-run.sh * maps search-stack-visibility -> search-stack-check.py search-stack-visibility.prose.md * contract text, execution model, pass/fail shapes, residual risk Scheduled hourly at :15 on CT 100 via /etc/cron.d/contract-runner.	2026-09-25 01:14:03 +00:00
root	c0d04a2c02	fix: three wrapper holes in contract-run.sh ... 1. Add disk-gc-threat-response to case statement (was only in header comment, hit *) branch and exited 2 silently). Mapped to scripts/disk-gc-scan.py per contract's Execution section. 2. Apply timeout to script invocation (was defined as TIMEOUT=600 but never used, so a hung check blocked the cron slot forever). Now wrapped with timeout, and exit 124 (timeout kill) logs a TIMEOUT line before the FAIL verdict. 3. Send alert on exit-2 paths (unknown contract and missing script). Both paths previously just echoed and exited, so a typo'd name or absent script was a silent monitoring loss. Now they send the same Zulip DM as a failed check. Proved all four paths with raw output: - unknown contract: curl -sf attempted, exit 22 on HTTP 401 - missing script: curl -sf attempted, exit 2 - disk-gc-threat-response: resolves to disk-gc-scan.py, runs, PASS - stub sleep > timeout: TIMEOUT line logged, exit 1, alert failure recorded	2026-09-24 06:01:01 +00:00
root	fda6c844ff	fix: add -f to curl to fail on HTTP >= 400 ... Without -f, a rejected credential (HTTP 401) returns curl exit 0, making a failed alert indistinguishable from a successful one. With -f, curl exits non-zero on HTTP >= 400, so DM_EXIT and STREAM_EXIT correctly capture the transmission failure and the run log records it.	2026-09-24 05:27:28 +00:00
root	748ea389be	fix: correct execution headings, variableize LOG_DIR, fix dead alert path ... 1. Replace '(## Execution' + ')' with '## Execution' in 6 contract files 2. Make LOG_DIR honor CONTRACT_RUN_LOG_DIR env var (default: /var/log/contract-runs) 3. Fix Zulip alert: correct URL (https://chat.sysloggh.net/api/v1), user (abiba-bot@chat.sysloggh.net), take ZULIP_API_KEY from environment, and write alert failures to run log	2026-09-24 05:17:49 +00:00
root	f16a890d0e	fix: make test_pbs_gc_states.sh self-contained with inline SSH replacement	2026-09-24 05:12:44 +00:00
root	c666d3e15c	feat: implement PBS GC four-state logic and update contracts ... - Implement four-state PBS GC logic in proxmox-monitor.sh: - probe-failed: unparseable JSON, store not found, or empty body → FAIL - running: collection in progress (last-run-endtime absent, upid present) → DO NOT FAIL - stale: no completed run within 48h → FAIL, naming last completed run age - healthy: completed within 48h → PASS, naming endtime and pending bytes - Add tests/test_pbs_gc_states.sh covering all four states - Proves the test bites on the pre-fix version (5/6 tests fail) - All 6 tests pass against the fixed version - Update contract-run.sh to map disk-gc-threat-response -> scripts/disk-gc-scan.py - Update Execution sections of host-scheduled contracts: infrastructure-monitoring, zulip-health, litellm-health, agent-health-check, disk-gc-threat-response, pm2-self-heal Adding note that execution is host-scheduled via cron, not agent session ack.	2026-09-24 04:53:46 +00:00
root	c07aa5e382	Add contract-run.sh for machine scheduler execution and fix PBS GC leg ... - Add scripts/contract-run.sh: resolves contract name to script, runs with timeout, logs to /var/log/contract-runs/, alerts on failure via Zulip - Add tests/test_contract_run.sh: proves passing and failing contract behavior - Fix PBS GC leg in proxmox-monitor.sh: simplify logic to check last-run-endtime, use absolute paths for pct and proxmox-backup-manager to avoid PATH issues Part of Task: contract-execution-host-scheduler-20260924	2026-09-24 01:22:24 +00:00
root	bb1b65340e	Merge pull request #128 from decisions-2026-08-03-rebased	2026-09-22 11:28:33 +00:00
root	9e87927444	pm2-self-heal: reconcile contract with live PM2 set and alert channels ... - Add zulip-watchdog to Maintains (it's running, infrastructure-monitoring expects it) - Remove gpu-monitor from PM2 Maintains (it's systemd-only, not PM2-tracked) - Add Execution steps for all monitored processes (gitea-runner, zulip-watchdog) - Update alert channel: Telegram is primary, Zulip DM is secondary - Update script to check all 4 processes (abiba-telegram, abiba-zulip, gitea-runner, zulip-watchdog) - Add restart count thresholds for all processes - Update log output to include all process statuses	2026-09-22 11:25:45 +00:00
root	b0683e9566	fix: correct logging destination in description (Gitea health-logs, not knowledge graph)	2026-09-22 11:03:20 +00:00
root	dd11c8f14f	Apply captain 2026-08-03 decisions: restore abiba-zulip, retire gpu-watchdog, PM2-track gpu-monitor	2026-09-22 10:45:44 +00:00
root	59ed7cdbf7	fix(daily-infra-report): remove vestigial ZULIP_API_KEY requirement, exit non-zero on failed send ... 1. Remove vestigial ZULIP_API_KEY requirement: - /api/v1/server_settings is a PUBLIC endpoint (verified HTTP 200 with or without credential) - No Zulip API key is required for this call - If a future leg genuinely needs abiba-bot's key, it must prove it with a 200 from /api/v1/users/me as abiba-bot and label itself degraded when it cannot - Never fall back to the vault's shared ZULIP_API_KEY 2. Make failed sends exit non-zero: - A degraded leg (no credential configured) must stay exit 0 - A failed send (attempted and failed) must exit 1 - This distinguishes 'not configured' from 'attempted and failed' Test evidence: - No-credential run: exit 0, digest still produced - Wrong password: exit 1, labelled SMTP error - grep -n ZULIP_API_KEY: only comment reference remains	2026-09-21 11:30:42 +00:00
root	ccc916d1ec	feat(daily-infra-report): make missing credentials a labelled degraded leg ... - ZULIP_API_KEY: no longer SystemExit, now reports 'credential-missing: ZULIP_API_KEY' - EMAIL_PASSWORD: no longer sys.exit(1), now appends to DEGRADED_LEGS and returns success - PVE API: fixed None check in storage section - Summary: reports degraded legs before summary This allows the digest to be produced and emailed even when credentials are missing, while still explicitly logging which legs are degraded. Test: empty env produces JSON report + degraded leg labels, no SystemExit.	2026-09-21 11:00:25 +00:00
root	c72436b406	no-mistakes(document): docs: reconcile zulip-monitor status in infrastructure-control	2026-09-19 22:40:25 +00:00
root	a17379676d	no-mistakes(document): docs: sync zulip-health contract with C1/C3 and version	2026-09-19 22:38:53 +00:00
root	63990b84f7	no-mistakes(review): Fix review findings: test regression, contract verdict, scope trim	2026-09-19 22:31:25 +00:00
root	7a5ddb46a9	chore: Add local monitoring tooling scripts	2026-09-19 22:27:19 +00:00
root	568fec2efa	test(zulip-kagentz): Replace string-presence tests with behavioural sandbox tests ... - C3 502 → INCIDENT - C3 000 → INCIDENT - C1 401 + C3 302 → 0 issues, all healthy - C1 000 → INCIDENT Each test asserts from the run's own log/verdict, not from file text. Prose assertions kept as secondary. Proven to bite: run against pre-fix script (origin/master) shows all 4 behavioural cases fail because C3 leg doesn't exist and Result line doesn't say 'INCIDENT'.	2026-09-19 22:25:03 +00:00
root	641a52c6da	fix(zulip-monitor): Add C3 public access path, make Result line non-optimistic ... - (b) Changed Result line to say 'INCIDENT' when ISSUES > 0, '0 issues (all healthy)' when ISSUES = 0 - (c) Documented C1 (no credential needed), C2 (requires LITELLM_KEY) distinction - (d) Added C3 public access path leg for https://kagentz.sysloggh.net/ - C3 treats 200/302/401 as alive, 502/000 as incident - Added tests/test_zulip_kagentz_legs.py to verify all changes	2026-09-19 22:14:08 +00:00
root	9a2ee6faec	fix(litellm): Remove duplicate 'host healthy' from busy line ... The busy line was rendering as: 'busy (completion timed out after retry; host healthy host healthy (200))' because host_detail already contains 'host healthy (200)' and the prefix also said 'host healthy'. Fixed to: 'busy (completion timed out after retry; host healthy (200))' F1 cosmetic fix from PR #123 verify.	2026-09-19 11:53:32 +00:00
root	e71ded3c8c	fix: internal /v1 WARN not FAIL; align prose ... Rule 5 now treats internal http://192.168.68.116/v1 as non-canonical but working (authenticated via nginx), producing a WARNING instead of a FAILURE. The canonical internal path /litellm/v1 and the public host https://litellm.sysloggh.net/v1 both PASS. Everything else FAILS. Prose aligned: hermes-key-enforcement.prose.md now states the canonical internal form, notes that internal /v1 still works but is flagged as non-canonical (WARN not FAIL), and clarifies that the public host serves /v1 ONLY (404 on /litellm/v1). Corrected the 'unauthenticated path' wording at line 116, which was factually wrong. Tests updated: BASE template uses canonical internal path; new test cases prove canonical /litellm/v1 PASSES, wrong path FAILS, public host PASSES, and internal /v1 WARNS (not FAILS). Fixed backwards comment in test_old_rule5_check_would_fail_canonical.	2026-09-19 11:48:48 +00:00
root	a12abbeb14	fix(litellm): Implement busy vs down with proper degraded state ... Three states: - healthy: passed, exit 0 (unchanged) - busy (completion timed out after retry AND host /health answered): ⚠️ DEGRADED line, does NOT fail the run, exit 0 - host unreachable or real fault: ❌, exit 1 (unchanged) Summary now reports degraded count: - All pass, no degraded: '✅ All checks passed' - All pass, 1+ degraded: '✅ All checks passed (1 degraded: gpu-dense)' - Some failed: '❌ Some checks failed' or '❌ Some checks failed (1 degraded: ...)' Host health mapping verified: - gpu-dense -> 192.168.68.8:8080/health - gpu-vision -> 192.168.68.110:8080/health - strix-moe -> 192.168.68.15:8080/health	2026-09-19 11:43:39 +00:00
root	0b9aebca37	fix(litellm): Fix timeout kind reporting + add busy/degraded detection ... 1. TIMEOUT KIND FIX: run_command returns (1, '', 'TIMEOUT') when its own timeout fires. probe_http now checks for this before falling through to 'curl exit <rc>', so a 30s timeout reports 'timeout after 30s' not 'curl exit 1'. 2. BUSY/DEGRADED DETECTION: After both model probes fail, check the model's host health endpoint (e.g. 192.168.68.8:8080/health for gpu-dense). If the host answers 200, report 'busy (completion timed out after retry; host healthy 200)' — do NOT fail the run on that alone. If the host does not answer, that's a real FAIL. 3. RETRY TIMEOUT INCREASED: Single-host retry timeout raised from 45s to 90s. Worst-case prefill on a single-slot .8 host is ~76s (observed 83K-token prompt at 1078 tok/s), so 90s covers it. New line shapes: - Busy: 'gpu-dense: busy (completion timed out after retry; host healthy 200)' - Real failure: 'probe-failed: gpu-dense timeout after 30s then timeout after 90s (2 attempts)'	2026-09-19 11:42:36 +00:00
root	fa458afa26	fix: Rule 5 accept canonical internal and public host base_url ... Rule 5 in audit-hermes-config.py had an inverted check: it expected base_url=http://192.168.68.116/v1, but the contract hermes-key-enforcement.prose.md names http://192.168.68.116/litellm/v1 as CORRECT/CANONICAL in multiple places. The audit script would FAIL a config using the contract's canonical internal path and PASS one using a path the contract does not name. Fix: Rule 5 now accepts the canonical internal base (http://192.168.68.116/litellm/v1) AND the public base (https://litellm.sysloggh.net/v1), and FAILS anything else. The internal nginx serves both /litellm/v1 and /v1; the public host serves /v1 only (per 2026-09-19 probe from CT 116). Tests aligned: BASE template updated to use the canonical internal path, and new test cases added to prove the canonical internal path PASSES, a wrong path FAILS, and the public host path PASSES.	2026-09-19 11:37:14 +00:00
root	aee2de25ac	fix(litellm): Report both attempts' failure kinds in probe-failed ... The failure line now preserves both attempts' failure kinds instead of hardcoding 'timeout after retry, 45s'. If both attempts fail, the report shows: 'probe-failed: <model> <first kind> then <retry kind> (2 attempts)'. This fixes the self-contradictory output when the first attempt timed out but the retry failed with connection refused, and prevents the duration from appearing twice when both attempts were timeouts. Example outputs: - timeout then timeout: 'probe-failed: gpu-dense timeout after 30s then timeout after 45s (2 attempts)' - timeout then refused: 'probe-failed: gpu-dense timeout after 30s then connection refused (2 attempts)' - refused then refused: 'probe-failed: gpu-dense connection refused then connection refused (2 attempts)'	2026-09-19 10:53:44 +00:00
root	76653381ec	fix(litellm): Add retry with longer timeout for single-host model probes ... Single-host models (gpu-dense, gpu-vision, strix-moe) now retry once at 45s on initial 30s timeout failure before declaring probe-failed. This prevents a single transient timeout (cold prefill ~13s or concurrent generation hold) from failing the entire health digest. Evidence: 2026-09-19 ~06:55Z digest failed gpu-dense at 30s; 06:56Z direct probe 200 in 1.04s. The failed-probe-fails-the-run property is preserved: if both attempts fail, the script still exits non-zero with the target and duration named. Closes: daily-health-digest false negative on single transient timeout	2026-09-19 10:43:54 +00:00
root	d07c4494b5	fix(infra): F1 - Fix Docker Stats (9324) and PVE Exporter (9221) port comments ... The leg comments were wrong: - Line 207: Docker Stats showed :9323 (dockerd port) but should be :9324 - Line 215: PVE Exporter showed :9324 (docker-stats port) but should be :9221 These were the exact pairing this PR exists to correct. Read back the changed lines to verify: scripts/infra-monitoring.sh:207 shows Docker Stats (CT 116 :9324, 127.0.0.1 via SSH) scripts/infra-monitoring.sh:215 shows PVE Exporter (CT 116 :9221, 127.0.0.1 via SSH) Branch: fix/infra-monitoring-probe-ports-20260919	2026-09-19 02:52:33 +00:00
root	b10fd6fc98	fix(infra): F1+F2 - Fix port comments and add per-leg assertions ... F1: Fixed leg comments to match the actual ports - Line 207: Docker Stats now shows :9324 (was :9323) - Line 215: PVE Exporter now shows :9221 (was :9324) These were the exact pairing this PR exists to correct. F2: Added per-leg assertions that prove which leg owns which port The new assertions verify: 1. Docker Stats leg uses $DOCKER_STATS_PORT constant 2. PVE Exporter leg uses $PVE_EXPORTER_PORT constant 3. DOCKER_STATS_PORT constant is set to 9324 4. PVE_EXPORTER_PORT constant is set to 9221 Proof the new assertions bite: Under the both-constants-swapped mutation (DOCKER_STATS_PORT=9221, PVE_EXPORTER_PORT=9324), the suite fails with 25 passed / 2 failed (failing exactly the two constant-value assertions). This proves the per-leg assertions pin which leg owns which port, not just that both ports appear somewhere in the SSH log. Branch: fix/infra-monitoring-probe-ports-20260919	2026-09-19 02:46:30 +00:00
root	8ff13d38f3	docs(proxmox): Document all six PBS GC verdict shapes ... The Liveness Check section now documents ALL SIX verdict shapes exactly as emitted by proxmox-monitor.sh: 1. ✅ PBS GC: healthy (last run Nh ago, pending-bytes: N B) 2. 🔴 PBS GC: stale (last run Nh ago, pending-bytes: N B) 3. 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (expected JSON, got 000) 4. 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (unparseable JSON) 5. 🔴 PBS GC: never-run (storepve-datastore not found in GC list) 6. 🔴 PBS GC: never-run (storepve-datastore has no last-run-endtime) Fixed the quoted healthy example (line ~114) to include the pending-bytes suffix the code now appends. Previously the prose only documented the probe-failed shape, missing the PR's own headline cases (never-run). Proof: grep -n 'never-run' proxmox-monitor.prose.md now returns two lines (lines 106 and 108), documenting both never-run variants. Branch: fix/pbs-gc-liveness-signal-20260919	2026-09-19 02:43:49 +00:00
root	a13457bcd6	fix(infra): Fix Docker Stats (9324) and PVE Exporter (9221) probe ports ... Previously probed wrong ports: - Docker Stats was at 9323 (dockerd metrics) but should be 9324 (harness-docker-stats, docker_container_* metrics) - PVE Exporter was at 9324 (harness-docker-stats) but should be 9221 (harness-pve-exporter, 5 pve_* metrics) Both exporters bind to 127.0.0.1 on CT 116 and must be probed via SSH. Updated infrastructure-monitoring.prose.md to document the correct ports. Added test assertions verifying the exact ports are probed. Branch: fix/infra-monitoring-probe-ports-20260919	2026-09-19 02:32:00 +00:00
root	f59d1a2159	fix(proxmox): Fix PBS GC liveness leg + add comprehensive test suite ... (a) Probe-failure detection: now treats empty OR unparseable JSON as probe-failed, not never-run. This prevents 'command not found' outputs from being rendered as service verdicts. (b) pending-bytes: now extracted from JSON and reported in stale verdict. (c) Null endtime: use .get() with explicit None check, not 0 fallback. null values now correctly trigger never-run verdict instead of arithmetic crash (set -u). (d) Tests: Added 14-assertion stub-driven suite covering: healthy, stale (>48h), probe-failed (empty and unparseable), null endtime, datastore absent. Each test stubs ssh/curl to verify exact behavior against the pre-fix head. Branch: fix/pbs-gc-liveness-signal-20260919	2026-09-19 02:28:39 +00:00
root	da8f5f43c9	docs(proxmox): Add PBS GC documentation to proxmox-monitor.prose.md ... Document the PBS GC schedule (00:00 UTC, not 20:00 UTC as PR #116 said), what actually runs (pbs-gc.sh -> pct exec 107 -- proxmox-backup-manager), datastore location (CT 107's /mnt/pbs-backup on /tank/pbs-backup, NOT /media/easystore2), and the new liveness check (48h threshold, reports age in hours, explicit healthy line). Branch: fix/pbs-gc-liveness-signal-20260919	2026-09-19 02:12:17 +00:00
root	8ae4b59150	feat(proxmox): Add PBS GC liveness signal ... Add monitoring leg that checks storepve-datastore GC health: - Reads GC state from CT 107 via pct exec - FAILS if last-run-endtime is older than 48h - Reports age in hours and pending-bytes status - Uses JSON parsing for reliable data extraction Test: All 5 legs OK, Exit 0. Branch: fix/pbs-gc-liveness-signal-20260919	2026-09-19 02:04:05 +00:00
root	933cfd223b	fix(infra): PR #115 round 4 — fix TLS detection + remove duplicate probe ... Round 3 had: 1. rc captured from wrong command (tr always exits 0) 2. Every probe issued TWICE (26 invocations instead of 13) 3. TLS branch unreachable Round 4 fixes: - Restructure probe_http to ONE invocation that captures both output and status: out=$(...); rc=$? - Delete the duplicated block - Fix retry classification: don't overwrite kind if already set (e.g., tls) - Add test 7b: TLS error (000 + exit 60) → kind is tls - Update header output shape to include (<kind>) suffix Test: 22 passed / 0 failed (bash scripts/test_infra_monitoring.sh)	2026-09-18 18:50:39 +00:00
root	f77d6ca1d1	fix: correct field name to allowed_mcp_servers ... PR #118 finding F1 (low): The deployed LiteLLM on CT 116 uses allowed_mcp_servers (193 occurrences in installed package), not bare allowed_mcp. One-word doc fix.	2026-09-18 18:49:20 +00:00
root	f4f8a4cab8	fix: consolidate PR #117 Rule 15 wording fix ... Add the audit-hermes-config.py Rule 15 wording fix from PR #117: - Violation message now reads 'URL is incorrect: <url> (expected: <expected>)' - Detection logic unchanged - Matches URL and not-in-known-list branches remain byte-identical This consolidates relay #779 into a single PR (#118).	2026-09-18 18:38:38 +00:00
root	7e257ce512	fix(infra): PR #115 final round — complete F1/C1 + implement TLS detection ... F1: Set LAST_KIND on unexpected-status path (was unset, causing empty placeholder in 7 failure lines). F2: Implement TLS detection — capture curl exit code and map TLS error codes (35|51|58|59|60|77|83) to kind=tls. Previously TLS failures were misdiagnosed as timeout. F3: Header comment now lists all producible kinds: timeout | refused | tls | unexpected:<code>. F4: Add 2 test assertions: Grafana failure line exists + kind is non-empty (proves the gap that shipped in round 2). Test: 20 passed / 0 failed (bash scripts/test_infra_monitoring.sh)	2026-09-18 18:34:39 +00:00
root	c0454811bb	fix: update MCP access docs to reflect per-key grants support ... PR #117 follow-up (verify PASS-WITH-FINDINGS): 1. infrastructure-update.prose.md: - Update access table: agent keys now have per-key MCP grants (2026-09-18) - Strike-through old limitation: per-key grants now work - Mark Migration Path as COMPLETED 2026-09-18 2. hermes-config-template.prose.md: - Remove hedge ('may have been upgraded') - State fact: per-key MCP grants verified 2026-09-18 This resolves the contradiction where one file asserted per-key MCP access and the other denied it.	2026-09-18 18:34:31 +00:00
root	3c7f5d7d65	fix(infra+zulip): PR #115 round-2 findings F1-F4+F7 ... F1: Make kind classification real — append (<kind>) to every failure line, assign kind=tls on curl exit 35/60, fix :112 where kind=refused was set on successful retry. Update prose shape. F3: Move credential placeholder skip from server probe to notify() only — server is always probed (200 without auth verified live). F4: notify() logs ALERT SUPPRESSED when credential unusable so alerts from other legs are not silently dropped. F7: Restore trailing newline in infra-monitoring.sh. F5 (DO NOT CHANGE): Verified directly — ssh root@192.168.68.6 'grep -n keep-daily /etc/pve/jobs.cfg' returns five prune-backups keep-daily=35 lines. Prose is CORRECT. Test: 18 passed, 0 failed (bash scripts/test_infra_monitoring.sh)	2026-09-18 18:21:12 +00:00
root	03be9b13d0	fix(zulip-health): skip server leg when credential is placeholder ... When ZULIP_API_KEY is unset or contains 'placeholder'/'REDACTED', skip the global Zulip server leg with a ⏭ marker instead of failing the whole script. The pi/Tanko/kagentz legs do not need the Zulip API key and keep their verdicts. Tracked as: zulip-health-credential-placeholder-20260913 (captain-held) This removes the repeated 'Action required' noise every cycle while keeping the credential enforcement loud and visible.	2026-09-18 06:09:27 +00:00
root	c295322c85	fix(test): stub curl/ssh to assert actual call targets (A2+A3) ... Rewrote test to run the monitor with stubbed curl/ssh on PATH that capture the exact argv of each probe call. The test now asserts the URL+port of every leg actually requested, not source text or config constants. A2: PVE node assertions now check the exact URL in the curl log (https://192.168.68.9:8006/... must appear), so a wrong IP (e.g. .99) fails the test. A3: Grafana/Prometheus/LiteLLM assertions check the URL the call actually builds, so a hardcoded wrong port in the CALL (while the config variable stays correct) fails the test. Mutation evidence: A2: sed s/192.168.68.9/192.168.68.99/ in PVE_NODES -> suite FAILS A3: sed s/"$GRAFANA_PORT"/"9999"/ in probe call -> suite FAILS Results: 18 passed, 0 failed (baseline); 17/18 on each mutation	2026-09-18 05:59:30 +00:00
root	385f7e0623	fix(infra-monitoring): resolve PR #115 review findings (A1-A3, B1-B2, C1-C2) ... A1: Test -k assertion now checks use_k:+-k syntax (actual bash pattern) A2: PVE_NODES assertions now count expected nodes and verify exact array size A3: Test now asserts liveness behavior (PVE_API_LIVENESS=1) not source text B1: disk-gc GC schedule corrected: cron runs pbs-gc.sh (not proxmox-backup-manager), schedule is 20:00 LOCAL (00:00 UTC, not 20:00 UTC), host timezone America/New_York B2: PROBE SHAPE now documents actual output shape including TLS flag notes C1: TLS kind is now printed in PVE API failure output C2: SSH retry logic clarified - retry is in probe_http function (not unreachable)	2026-09-18 05:54:06 +00:00
root	7efbfffe44	docs(disk-gc): clarify media vs pbs-datastore HOST-RED escalation ... HOST-RED on media volumes says 'capacity decision — owner to decide' not 'immediate owner attention'. Media volumes are report-only at all levels; the urgency language was misleading. PBS datastore and host-root get the immediate attention wording. Closes the welcome-back proposal: 'how the disk-gc check should classify a media volume so HOST-RED stops meaning nothing.'	2026-09-18 05:25:19 +00:00
root	315fcbae23	docs(disk-gc): clarify GC schedule applies to PBS datastore only ... The 20:00 UTC PBS GC cron (proxmox-backup-manager datastore prune) applies only to /tank/pbs-backup (pbs-datastore). It does NOT touch media volumes (/media/*) which are report-only at all threat levels. This clarification prevents the recurring confusion where a 96% media volume triggers a GC expectation, when the GC schedule never applies to it. Closes the 2026-09-17 correction: 'the GC schedule is now 20:00 UTC, protects the backup datastore, NOT the nearly-full media volume.'	2026-09-18 05:22:57 +00:00
root	93f15709d1	fix(infra-monitoring): move probes to versioned script with port-drift test ... The 2026-09-17 false-verdict incident (third recurrence) showed that prose policy is not a control: the agent probed :9325/:9405 (nonexistent ports), CT 116 for PVE API (should be real PVE nodes), and rendered TLS failures as connection-refused. This moves the canonical probe set into scripts/infra-monitoring.sh (executed verbatim by the contract) and adds scripts/test_infra_monitoring.sh which asserts every probed port matches the documented value. - scripts/infra-monitoring.sh: one script per contract pattern; all targets, ports, paths, and expected-status rules in code; -k for PVE self-signed certs; non-zero exit naming every failed target; no OK summary on failure - scripts/test_infra_monitoring.sh: 20 assertions covering port drift, monitoring-host-as-PVE-node, and missing -k flag - infrastructure-monitoring.prose.md: check-health section now references the script as executable owner; paste its raw output verbatim Proof: all 13 legs pass (exit 0); deliberately broken Grafana port (9325) produces 'probe-failed: 192.168.68.116:9325 (expected 200)' and exit 1.	2026-09-18 05:17:35 +00:00
root	20f882412f	PR #112 round 2: fix syntax error, restore docs, clean residual credentials	2026-09-17 07:06:02 +00:00
root	30b2fe3fdc	Fix PR #112 security review - restore docs, remove live credentials	2026-09-17 06:51:52 +00:00
root	5112c566c8	Remove Stirling PDF credentials (password + API key) from 2 files	2026-09-17 06:40:59 +00:00
root	83307eb9b2	Annotate deprecated key in litellm-self-heal.prose.md as not live	2026-09-17 06:12:18 +00:00
root	8245716286	Remove all hardcoded credentials from repository ... Audit results (all patterns checked across .md, .prose.md, .sh, .py, .js, .ts, .json, .yaml, .yml, .env): - sk-or-v1 (OpenRouter): 0 occurrences - sk- prefix (20+ chars): 0 occurrences - sk_live: 0 occurrences - Bearer <key>: 0 occurrences - api_key: <value>: 0 occurrences - PASSWORD=: 0 occurrences - TOKEN=: 0 occurrences - SECRET=: 0 occurrences Files changed: - agent-zero-fix-summary.md (removed 2 OpenRouter keys) - agent-zero-openrouter-key.prose.md (removed 1 OpenRouter key) - hermes-key-enforcement.prose.md (removed 1 LiteLLM key, 1 external key) - litellm-api-keys.prose.md (removed 1 LiteLLM key) - litellm-self-heal.prose.md (removed 1 stale key reference) - scripts/agent-health-check.py (INFISICAL_TOKEN now required) - scripts/daily-infra-report.py (EMAIL_PASSWORD now required) - zulip-health.prose.md (TOKEN references annotated)	2026-09-17 06:11:42 +00:00
root	cfb6c03572	Fix remaining hardcoded ZULIP_KEY in zulip-monitor.sh (line 46)	2026-09-17 05:58:58 +00:00
root	85f70f65bc	Remove hardcoded ZULIP_KEY from monitoring scripts ... Scripts that had hardcoded credentials: - scripts/zulip-monitor.sh:12 (was ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8") - scripts/daily-infra-report.py:25 (was ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8") Both now read from environment variable ZULIP_API_KEY (set by vault-backed start script) with loud failure if not present. Other credentials in scripts/: - capture-dsh-token.sh: uses TOKEN variable with fallback (not a secret) - pm2-self-heal.sh: reads TELEGRAM_BOT_TOKEN from /root/.pi/agent/extensions/telegram/.env (acceptable) - prose-ai-review.sh: uses GITEA_TOKEN from .env file with LITELLM_KEY fallback (not secrets) No other hardcoded credentials found. Proof of behavior: With ZULIP_API_KEY set: bash scripts/zulip-monitor.sh → Server: HTTP 200 (authenticated) python3 scripts/daily-infra-report.py --json → Collecting infrastructure data... Without ZULIP_API_KEY set: bash scripts/zulip-monitor.sh → "ZULIP_API_KEY not set — refusing to run with no credential" python3 scripts/daily-infra-report.py --json → "ZULIP_API_KEY not set — refusing to run with no credential" Cred source: environment variable ZULIP_API_KEY (set by vault-backed start script) No key rotation (that is a separate decision).	2026-09-17 05:51:30 +00:00
root	0b92ab17b1	Fix PR #111 round 2: GPU leg all 6 states + skipped templates	2026-09-17 03:05:22 +00:00
root	9edefe036e	Fix PR #111 review findings: GPU leg failure modes + leg templates ... Fix 1: GPU leg degradation is not just SSH probe failure — it also covers gpu-no-port and gpu-ghost conditions. Rewrite to match check_gpu_ports reality. Fix 2: Add skipped and partial exemplars for all four legs (LiteLLM keys, GPU ports, CTs, Vault secrets) so the template covers the rule rather than only the happy path. Cosmetic: note that compact form (rtx5070 timeout) is acceptable in summary line when host is identifiable from context; full probe-failed: <target> <kind> form required in detail section.	2026-09-17 02:53:51 +00:00
root	dd6e1e8b22	Add mandatory report legs to agent-health-check contract ... Every report line MUST include one clause per check leg, even when a leg is skipped or fails. Missing leg must never look the same as healthy leg. Required legs: - LiteLLM keys: N/M (names) status - GPU ports: N/M (rtx3090, rtx5070, strixhalo) status — or SKIPPED (reason) - CTs: N/M running (names) - Vault secrets: status GPU leg is never skipped by configuration; only SSH probe failure causes degraded status.	2026-09-17 02:45:40 +00:00
root	7f62f19c24	fix: litellm-key-count-self-describing-20260916 ... The key-count line in litellm-health-check now reports self-describing output: '18 total (10 on page 1)' instead of bare '18' or '4'. Uses total_count from the paginated API response and names what was counted. Previous bare numbers could not reconcile changes between runs; now a reader sees both the total and the page 1 sample.	2026-09-16 15:20:50 +00:00
root	9100ea3326	fix: tanko-plaintext-key-in-config-backup-20260916 ... 1. Host-side fix (tanko 192.168.68.122): Moved 5 config.yaml.bak-* files from /root/.hermes/ to /root/hermes-config-backups/ so the scanner pattern no longer matches. Dead credential (sk-b7d99... DEEPSEEK key from July, 401 against gateway) is preserved in history without cluttering the scanned tree. 2. Contract text: Added ACCEPTABLE PATTERN section to hermes-key-enforcement.prose.md clarifying that agent keys live in .env/.env.vault with 600 perms (koonimo's shape), while a plaintext key in config.yaml or any config backup is a violation. Fix procedure: move the backup file out of the scanned tree, don't delete.	2026-09-16 11:16:06 +00:00
root	5c1c8d7c19	fix: PR #107 review fixes — cron cadence + gateway log health check ... FAIL 1: Cron cadence was */10 * * * * (every 10 min) but the real crontab on CT 100 is 35 2,6,10,14,18,22 * * * (every 4 hours at :35). Fixed in frontmatter, body, and Continuity section. Added 4-hour rationale note. FAIL 2: Added gateway log health to the list of checks (frontmatter + Strategies section). Added note that script may perform additional diagnostics beyond the seven contract checks.	2026-09-16 04:50:50 +00:00
root	8a2ea2d0d7	docs: add agent-health-check.prose.md contract ... The consolidated agent health check contract (wraps scripts/agent-health-check.py v4). Created during earlier work but never committed — was a stray untracked file in the execution clone, making the home look dirty to the fleet update path.	2026-09-16 04:36:43 +00:00
root	39209c7ac9	fix: untrack host-disk-bands.json and document gitignored status ... The state file is runtime state (rewritten every scan), so tracking it in git means: - every executor's clone becomes permanently dirty after one run - a scan in one clone produces a merge conflict with a scan in another - the committed baseline can be stale in a way nobody notices Added to .gitignore and removed from the index. Contract updated to say 'the state file lives at <abs path> and is gitignored runtime state - the scanner creates it on first run'.	2026-09-16 00:45:46 +00:00
root	8ed3b9c606	fix: implement host-filesystem state file with transition detection ... The contract said the state file was written after every scan, but the script had no state-file logic at all. This PR adds: 1. Host filesystem scanning (probe_host_filesystems) - probes df on all PVE nodes 2. State file I/O (read_state_file/write_state_file) - absolute path from script location 3. Band classification (classify_band) - HOST-WARN/AMBER/RED thresholds 4. Transition detection (detect_transitions) - alerts on escalation/recovery 5. CLI flags (--hosts-only, --guests-only) to control which parts run The contract now specifies the state file path resolves from the script's own location (not CWD-relative), so two different execution contexts cannot write to two different places.	2026-09-16 00:41:34 +00:00
root	b9b1712ac6	fix: make host escalations state-change driven ... A volume alerts ONCE when it enters a higher band (GREEN->WARN, WARN->AMBER, AMBER->RED) and ONCE when it drops back down (recovery notice). While it stays in the same band, it is reported in the scan output only — no DM, no channel alert. This stops the same 96% easystore2 from re-DMing the owner on every 6h scan. State lives in a small JSON state file (state/host-disk-bands.json), keyed by host/volume -> last-seen band. The scanner reads the prior band, compares to the current band, and DMs only on a transition; the state file is written after every scan. Chosen over a periodic digest because the scan already runs every 6h and a transition is genuinely new, actionable state. First-run behavior: when the state file does not yet exist, the current band of every volume is recorded as baseline WITHOUT alerting — a first run would otherwise DM every already-elevated volume at once. Report-only restriction and volume-naming output kept exactly as-is.	2026-09-15 14:25:49 +00:00
root	6fb411613e	fix: add host filesystem thresholds to disk-gc contract ... Add separate threat bands for HOST filesystems (distinct from guest bands): - HOST-WARN at 85%: name volume + % + absolute free space in scan output - HOST-AMBER at 90%: flag for owner attention, Zulip DM - HOST-RED at 95%: flag for immediate owner attention, Zulip DM + channel alert Volume naming rule: every host line MUST name the volume and what lives on it. Action classes by volume type: - host-root: near full = real risk (backup staging, thin-pool metadata) - media (/media/*): near full = capacity decision for owner, never auto-delete - pbs-datastore (tank): near full = breaks Proxmox Backup Server Report-only restriction: no automatic deletion of media or datastore content ever. Justification (measured 2026-09-15): storepve /media/easystore2 at 96% was reported but never banded or acted on. Two incidents this weekend showed the host filesystem is the thing that breaks, not the guest's. Added HOST-WARN/AMBER/RED alert templates. Added report-only execution rule for host filesystems.	2026-09-15 14:15:53 +00:00
root	bc7a55122f	fix: pm2 contract corrections ... pm2-self-heal.prose.md: - Add AS-BUILT note: gpu-monitor is systemd-managed, NOT PM2 - gpu-watchdog is decommissioned and folded into gpu-monitor.service - gitea-runner is KEPT; abiba-zulip is KEPT (online for days) - spoton-service was deleted; live PM2 set is 4 processes - Preserve historical context for crash-loop guard litellm-health.prose.md: - Correct Prometheus node coverage: 6 nodes (.4/.5/.6/.9/.12/.15:9100) - Note .4:9100 is DEAD target (no route, down for weeks) - Clarify this does not read as 6 healthy nodes	2026-09-15 13:05:32 +00:00
root	b4b5321011	fix: add hostname resolution warning and fix dmsetup field documentation ... Add warning that bare PVE hostnames (acerpve, amdpve, etc.) resolve to VPS via *.dns.sysloggh.net wildcard, not to actual nodes. List IP addresses: - acerpve 192.168.68.9 - amdpve 192.168.68.15 - storepve 192.168.68.6 - minipve 192.168.68.12 - ocupve 192.168.68.5 Update acerpve example in backup preflight to include address (192.168.68.9). Fix dmsetup comment to show full field order: =start =length =thin-pool =transaction-id =metadata_used/metadata_total =data_used/data_total remaining fields are flags Make it clear lvs command is the primary source for percentages, dmsetup is only for error-state check. Incidents now include addresses: acerpve (192.168.68.9) and amdpve (192.168.68.15).	2026-09-15 12:09:59 +00:00
root	69940bc9eb	fix: correct backup preflight commands - lvs pve/data and dmsetup field documentation ... Fix two errors in the PREFLIGHT section (measured on acerpve 2026-09-15): 1. lvs -o ... pve/data (not pve-data-tpool) - this is the PRIMARY check that yields percentages directly - Quote the acerpve example: data 29.95% 1.22% <816.21g 2. dmsetup status pve-data-tpool - document fields correctly: - = transaction ID (99), NOT data_percent - = metadata used/total blocks - = data used/total sectors - Show how to derive percentages if needed 3. Keep the error-state check (grep -q 'Error|Fail') - this is how the incident presented Everything else stays: 1777 tmpdir requirement with EACCES symptom, incidents as rationale, GPU-host fact, --output-format json rule, honest note that metadata/snapshot pressure is unproven.	2026-09-15 11:42:10 +00:00
root	65eaffe1c6	fix: add backup safety preconditions - thin-pool headroom and tmpdir 1777 ... Add documented preflight checks for VM/CT backups on LVM thin-pool hosts: - dmsetup status pve-data-tpool + lvs to verify data_percent < 90% and metadata_percent < 70% - Exit 1 if pool shows Error/Fail state (takes down entire VG including host root) - tmpdir must be mode 1777 (world-traversable) for vzdump archive step - --output-format json for tasks started from truncating shells - Document two incidents: acerpve thin-pool VM 101 (twice on 2026-09-13) and amdpve 0700 tmpdir (2026-09-14) - Note metadata/snapshot-pressure hypothesis is UNPROVEN; preflight is the control - Document GPU-host fact: VM 101 (llm-gpu) and VM 103 (ocu-llm) have no scheduled backup	2026-09-15 11:37:41 +00:00
root	8a4dd08b05	fix: capture 401/403 response body and key alias for credential faults ... - get_response_body() returns first 200 chars of response body (single line) - On 401/403 model probe: report code + body + key_alias - Monitor key alias: monitor-20260813 (from /etc/litellm-monitor.env on CT 116) - Failed connections stay probe-failed, 200 stays plain 200 - Do not turn other statuses into credential faults Signed-off-by: Abiba	2026-09-15 05:11:07 +00:00
root	88b6decb31	fix: remove false Infisical claim - master key NOT in infrastructure project ... - Replace Infisical retrieval path with proven docker exec + .env note - State explicitly that master key is NOT in Infisical project=infrastructure - Keep the live-key check and never-trust-a-literal instruction - All other corrections from PR #95 preserved Signed-off-by: Abiba	2026-09-15 04:42:31 +00:00
root	7bf9f78fc6	fix: gpu-dense probe timeout handling - report probe-failed with kind, not service verdict ... - probe_http now returns (code, failure_kind) tuple - Model probes report 'probe-failed: <model> <kind> (Ns timeout)' on 000 - Do not assert a service verdict from a failed probe - 30s timeout for single-host aliases (RTX 3090 needs long warmup/prefill) - 60s timeout for syslog-auto pool alias with retry on 000 Signed-off-by: Abiba	2026-09-15 04:22:46 +00:00
root	dd9e68329e	fix: correct infisical --plain flag and use verified localhost:4000 endpoint ... - Replace broken --plain flag (prints nothing on CLI 0.43.110) with awk parsing - Note that --plain is broken so nobody fixes it back - Replace unproven nginx path with verified direct endpoint http://127.0.0.1:4000/key/list Signed-off-by: Abiba	2026-09-15 03:05:30 +00:00
root	cd9ec6a0df	fix: correct key-lifecycle contracts to match measured reality ... - hermes-key-enforcement.prose.md: - State that expiry must be set EXPLICITLY at creation with duration - Record that config default is NOT honoured by LiteLLM 1.99.1 - Describe daily audit as AUDIT-ONLY (reports non-expiring and soon-to-expire) - State that renewal is NOT implemented - Document exclusions: abiba-pi and all crewmate keys stay WITHOUT expiry - koby is report-only - litellm-api-keys.prose.md: - Replace literal master key with retrieval path (docker exec + infisical) - State that literal values must never be trusted again (key rotates) - Add live-key check (200 from /key/list) Signed-off-by: Abiba	2026-09-15 02:59:19 +00:00
root	2f961d7e7a	fix: agent-health-check gateway leg — deterministic probe-failed reporting ... Per defect report 1154.msg (agent-health-gateway-leg-flap-20260913): 1. Add _ssh_retry() helper with one retry at longer timeout (25s) 2. Name probe target explicitly: ssh {user}@{host} <command> 3. Print probe-failed when first attempt fails, then retry 4. Only declare gateway-down after retry fails 5. Make Koby report-only explicit in output Changes: - check_agents(): all gateway probes now use _ssh_retry() - All output lines name the probe target (ssh host:port) - Koby's report-only status is explicit in output - Never print bare "gateway down" — always name target and failure kind Verified: koonimo shows "probe-failed" on first attempt (transient SSH), retries at 25s, succeeds, reports ✅ koonimo: gw=running	2026-09-14 15:21:36 +00:00
root	86d2987ad8	fix: probe precision — add retry + probe-failed reporting to zulip-health ... Per defect report 1150.msg: - Add standing probe rules section (2026-09-14) - Step 1 (Zulip API): retry once at 25s on 000, print target + code - Step 2 (Platform A): retry once at 25s on 000, print target + code, note must run on Abiba host - Apply same shape as infrastructure-monitoring: any HTTP status = ALIVE; only 000/timeout = probe-failed Verified: all probes now return real HTTP codes (Zulip API 200, Platform A 200, Tanko 401, Agent Zero 401)	2026-09-14 14:53:57 +00:00
root	efe9381283	fix: probe precision — correct GPU exporter path, Grafana port, add retry + probe-failed reporting ... Per defect report 1150.msg: - GPU exporters: probe /metrics (Prometheus scrape target), not bare / - Grafana: correct port 3001 (not 3000) - All probes: print target name + full URL + HTTP code - All probes: retry once at 25s on 000/timeout - Apply standing rules: any HTTP status = ALIVE; only 000/timeout/refused = probe-failed Verified: all probes now return real HTTP codes (GPU 200, Grafana 200, Router 200, LiteLLM 301)	2026-09-14 14:44:37 +00:00
root	5ed6f8179c	fix: add os import for HELPER_PCT_RUN.readable() check	2026-09-14 14:14:15 +00:00
root	b9322973ce	fix disk-gc-scan: CWD independence (FIX A) and df column parsing (FIX B)	2026-09-14 14:05:06 +00:00
root	fb1916707d	document deterministic disk-gc-scan.py in contract	2026-09-14 13:20:26 +00:00
root	d28df4f4de	add disk-gc-scan.py: deterministic fleet disk probe with per-guest access methods	2026-09-14 13:14:10 +00:00
root	9a789ab76d	fix(hermes): separate policy observations from fault findings ... Contract design defect: 'uses a non-harness provider' (POLICY) and 'cannot authenticate' (FAULT) were printed as the same violation class. A policy observation must never be phrased as if the agent were broken. Changes: 1. Added 'Violation Classification' section to all three contracts 2. Separated POLICY (observation only) from FAULT (requires request-level evidence) 3. Rules: - Do NOT infer runtime credential resolution from config text alone - Require request-level evidence before calling a FAULT: observed auth failure or absence of successful calls - If calls are succeeding, output is 'POLICY: uses <provider> directly; calls succeeding' - not a violation - State what you OBSERVED, not what the field implies Files changed (3): - hermes-key-enforcement.prose.md - hermes-config-template.prose.md - hermes-agent-baseline.prose.md	2026-09-14 12:32:06 +00:00
root	4e34b7a2a2	fix(hermes): wire all three contracts to reachability helper ... The helper was correct but dead code - nothing called it. This commit: 1. Makes the helper runnable standalone: scripts/hermes-reachability-check.sh <host> <pattern> <path> 2. Wires all THREE contracts to it: - hermes-key-enforcement.prose.md - hermes-config-template.prose.md - hermes-agent-baseline.prose.md 3. Each contract now explicitly instructs to run the helper and interpret the three outcomes 4. States that the bug this replaces was deriving reachability from the remote grep's exit code Files changed (4): - scripts/hermes-reachability-check.sh (standalone mode added) - hermes-key-enforcement.prose.md (reachability section added) - hermes-config-template.prose.md (reachability section added) - hermes-agent-baseline.prose.md (reachability section added)	2026-09-14 12:01:32 +00:00
root	f9f6661dd5	fix(hermes): add shared reachability check helper ... Bug: The reachability check used 'ssh ... grep ... || echo unreachable', which conflated grep's 'no matches found' (exit 1) with SSH failure. This caused clean hosts to be reported as unreachable. Fix: Add scripts/hermes-reachability-check.sh with the pattern: out=$(ssh -o BatchMode=yes root@HOST "grep ... 2>/dev/null; true") if [ $? -ne 0 ]; then verdict="unreachable" elif [ -n "$out" ]; then verdict="violation: $out" else verdict="compliant" fi This correctly distinguishes: - SSH failure (connection/auth/route) -> unreachable - SSH success + grep found matches -> violation - SSH success + grep found nothing -> compliant Evidence: 3 of 4 hosts (Tanko, Mumuni, Koonimo) were reported as 'unreachable' when they were actually compliant. Only Koby (.129) has a real finding (plaintext key in state snapshot). Used by: hermes-key-enforcement, hermes-config-template, hermes-agent-baseline contracts.	2026-09-14 11:55:41 +00:00
root	05366bd58d	Fix litellm-health-check.py robustness defects ... 1. Timeout fix for pool alias (syslog-auto): - Single-host aliases (gpu-dense, gpu-vision, strix-moe): 30s timeout - Pool alias (syslog-auto): 60s timeout, retry once on 000 before failing - Cold first request to pool alias can take ~13s; 10s was too short 2. Remove DEBUG prints from output: - Removed 'DEBUG: keylen=...' and 'DEBUG: response=...' lines - These leaked key inventory to status logs - Success output now shows only counts (e.g., 'Admin Key List: 10 keys')	2026-09-14 03:24:30 +00:00
root	38d7e8b064	Update litellm-health contract to mandate executor script ... Add 'Executor Script' section: - Run scripts/litellm-health-check.py from the clone - Hand-rolled probes not acceptable substitute - Backend-edge checks use internal IP 192.168.68.116, not public URL - Docker Stats fetched from CT 116 host (127.0.0.1:9324/metrics) - Admin Key List requires proper quoting for SSH commands	2026-09-14 03:15:42 +00:00
root	e94fadfa60	Add litellm-health-check.py: standardized health check script for LiteLLM fleet monitoring ... Implements all 11 checks from litellm-health contract: - Liveliness, Containers, Prometheus, Grafana health probes - Model probes for gpu-dense, gpu-vision, strix-moe, syslog-auto - Admin API key list (10 keys), GitHub status, Docker Stats metrics Fixed quoting for SSH commands and response parsing (dict with 'keys' field). Backend edge uses internal IP 192.168.68.116, not public URL. Docker Stats fetched from CT 116 host itself (127.0.0.1:9324/metrics). All 11 checks passing consistently.	2026-09-14 03:15:27 +00:00
root	7906b2d52d	fix: change abiba default model from deepseek-v4-pro to syslog-auto ... The abiba-zulip-restore.prose.md file incorrectly claimed the default model was 'deepseek-v4-pro', but /root/.pi/agent/settings.json declares 'defaultModel: syslog-auto' and 'defaultProvider: syslog-harness'. deepseek-v4-pro is not a servable LiteLLM model name for this fleet. The only live models are: syslog-auto, gpu-dense, gpu-vision, strix-moe. Note: The firstmate/pi session talks to DeepSeek through its own provider (auth.json), NOT through LiteLLM, so no LiteLLM change is needed to keep firstmate's deepseek usage working.	2026-09-13 18:55:24 +00:00
root	c81cf5b6f0	fix: disk-gc-threat-response - correct access methods for kagentz and docker-vm ... - kagentz (105): use ssh root@kagentz (hostname), NOT pct exec 105 (shows loop0 59G, not real 99G) - docker-vm (109): use ssh root@192.168.68.7 (correct QEMU VM access) - abiba (100): use ssh root@abiba (hostname) - syslog-api (116): use pct-run 116 (correct) Verified: kagentz: 99G 8.9G 86G 10% / docker-vm: 158G 17G 135G 11% / abiba: 59G 13G 44G 23% / syslog-api: 40G 13G 25G 34% /	2026-09-13 04:30:28 +00:00
root	edcf465831	fix: litellm-health step 8 - run key/list on CT 116 host, not in container ... - harness-litellm container has no curl/wget, so docker exec harness-litellm curl returns empty - Fix: run curl on CT 116 host (ssh root@192.168.68.116 then curl) - Add admin-call-failed label for empty/unparseable responses - Verify: 10 keys found (host-side curl), NO-CURL confirmed in container	2026-09-13 03:10:09 +00:00