560 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
abiba-bot	4320369bb9	Merge pull request 'fix(alignment): resolve /root/ hardcoding and stale tanko-DSH references' (#147) from fix/agent-health-root-hardcoding-20260928 into master	2026-09-28 23:05:30 +00:00
abiba-bot	3b74ca28d1	Merge branch 'master' into fix/agent-health-root-hardcoding-20260928	2026-09-28 23:03:53 +00:00
root	af9397d672	fix(alignment): accept multiple wrapper shapes in hermes-real check ... The fleet's wrappers do not all use a hermes-real indirection. Some (tanko, mumuni) exec the venv module directly. This check now: 1. Tries hermes-real at {home}/.local/bin (koonimo's shape) 2. Tries the venv under {home}/.hermes/hermes-agent/venv (tanko/mumuni shape) 3. Tries /usr/local/lib/hermes-agent/venv (legacy system-wide shape) Each match is reported with which shape it matched, so a genuinely broken wrapper is still a failure while a different-but-valid shape is not.	2026-09-28 22:23:08 +00:00
root	97dc2d772f	fix(alignment): repair f-string quoting in config check, add home to scope ... - Fixed line 537: f-string now uses single quotes inside double-quoted shell command to avoid nested quote collision - Added home = get_user_home(user) to check_config_integrity loop scope so the config check can resolve the correct home directory	2026-09-28 21:20:42 +00:00
root	2238777a2f	fix(alignment): resolve /root/ hardcoding and stale tanko-DSH references ... F1: agent-health-check.py now resolves the home directory from the agent's user field via a shared helper (get_user_home) instead of hardcoding /root/. This fixes the false-positive wrapper-missing:tanko report — tanko has a working wrapper at /home/jerome/.local/bin/hermes, but the check was looking in /root/.local/bin/. F2: Updated stale references that described tanko as DSH-only: - hermes-zulip-restore.prose.md: tanko excluded — hybrid (DSH + Hermes) - hermes-zulip-plugin.prose.md: tanko excluded — hybrid (DSH + Hermes) - infrastructure-control.prose.md: tanko is hybrid (DSH + Hermes) agent - docs/probe-drift-round2-evidence.md: marked as historical record with dated note explaining that the DSH-only observations reflected the /root/ hardcoding bug, not the underlying truth Refs: fix/agent-health-root-hardcoding-20260928	2026-09-28 20:48:49 +00:00
abiba-bot	6b2ba1bba5	Merge pull request 'fix(alignment): clarify tanko live LiteLLM proxy status in health-check description' (#145) from fix/contract-alignment-f2-hermes-baseline-20260928 into master	2026-09-28 12:59:50 +00:00
root	a48b947242	fix(alignment): correct tanko status - hybrid (DSH + Hermes), not DSH-only ... PR #145 originally claimed 'the other 3 CTs are DSH-only' but all 4 agents run a Hermes gateway. Only tanko additionally runs DSH, making it the hybrid one. Fixed line 300 to say what the contract actually checks for each agent, and fixed line 76 to stop asserting tanko's Hermes config is gone.	2026-09-28 12:57:00 +00:00
abiba-bot	ba9d29b4b9	Merge pull request 'fix(alignment): recognize tanko as hybrid (DSH + Hermes) in agent-health-check' (#146) from fix/contract-alignment-f3-tanko-hybrid-20260928 into master	2026-09-28 12:54:42 +00:00
abiba-bot	d6376e5142	Merge pull request 'fix(alignment): use llmuser with sudo for swap-gpu-dense-model.sh' (#144) from fix/contract-alignment-f1-swap-gpu-user-20260928 into master	2026-09-28 12:53:53 +00:00
root	2ea6b4fc17	Merge PR #146: fix(alignment): recognize tanko as hybrid (DSH + Hermes) in agent-health-check	2026-09-28 12:50:23 +00:00
root	c6fd8eece3	Merge PR #145: fix(alignment): clarify tanko live LiteLLM proxy status in health-check description	2026-09-28 12:50:23 +00:00
root	4c715526ef	Merge PR #144: fix(alignment): use llmuser with sudo for swap-gpu-dense-model.sh	2026-09-28 12:50:23 +00:00
root	308265e7ce	fix(alignment): recognize tanko as hybrid (DSH + Hermes) in agent-health-check ... Tanko runs both DSH (pnpm dsh web) and Hermes (hermes gateway) concurrently. The false premise was that tanko is DSH-only with no Hermes gateway, which caused the gateway liveness, config.yaml, and wrapper integrity checks to be skipped entirely. Now tanko gets the full Hermes-era checks like koonimo and koby, while dsh/pi-only agents still skip those legs correctly.	2026-09-28 12:44:25 +00:00
root	88e9243ce4	fix(alignment): clarify tanko's live LiteLLM proxy status in health-check description	2026-09-28 12:38:42 +00:00
root	13ac189365	fix(alignment): use llmuser with sudo for swap-gpu-dense-model.sh (root SSH to .8 denied)	2026-09-28 12:34:32 +00:00
root	2851a0cfc8	Merge PR #143: fix(agent-health): use llmuser for .8 GPU health probe instead of root	2026-09-28 07:27:50 +00:00
root	e0c9852de8	no-mistakes(document): Document llmuser SSH user for .8 GPU health probe	2026-09-28 07:22:13 +00:00
root	bf3a1ba523	fix(agent-health): use llmuser for .8 GPU health probe instead of root ... The rebuilt VM 101 (192.168.68.8) kept only one SSH key in root's authorized_keys, so the health check's root probe returns Permission denied and misreports the healthy host as UNREACHABLE. The llama-server runs as llmuser, so that user can see the :8080 pid via ss -tlnp. Add per-host user to GPU_HOSTS (default root, llmuser for .8) and pass it through check_gpu_ports() into all ssh() calls. Closes the gpu-unreachable:192.168.68.8 leg while leaving the root SSH security decision for the captain.	2026-09-28 07:12:57 +00:00
root	f230812e3a	Merge PR #142: sync tanko CT 112 PVE mapping to minipve across consumers	2026-09-28 07:01:19 +00:00
root	96769a103f	no-mistakes(review): sync tanko CT 112 mapping to minipve across consumers	2026-09-28 06:47:46 +00:00
root	d697baa7b6	fix(agent-health): update tanko PVE mapping from amdpve to minipve (CT 112 migrated 2026-09-27)	2026-09-28 06:35:34 +00:00
abiba-bot	fb7f351a2b	Merge pull request 'fix(audit-hermes): handle fallback_providers as list or dict' (#141) from fix/audit-hermes-fallback-list-20260927 into master	2026-09-27 11:35:07 +00:00
root	e42b970dec	fix(audit-hermes): handle fallback_providers as list or dict ... The audit assumed fallback_providers was always a dict (single provider). Two live agents (koby, koonimo) carry it as a LIST of dicts (one entry per fallback), so the script crashed with: File "audit-hermes-config.py", line 211, in audit fb.get("provider") == "deepseek", AttributeError: 'list' object has no attribute 'get' Both are REAL agent configs, so this is not a malformed-input case — the script simply could not audit two of the four agents it exists to audit. Fix: - Normalize fallback_providers to a list of entries (dict → [dict], list → list) - Apply the existing checks to each entry - A malformed entry (not a mapping) produces a reported VIOLATION naming the offending entry, NOT an uncaught exception Adds regression test using the real failing shape (list of dicts) and proves it bites against the pre-fix revision. Real audit results after fix: - mumuni: FAIL — 7 violations - tanko: FAIL — 21 violations - koby: FAIL — 16 violations (previously crashed) - koonimo: FAIL — 10 violations (previously crashed) No agent configs were changed. No existing rules were relaxed.	2026-09-27 11:17:18 +00:00
abiba-bot	eadb927ec1	Merge pull request 'fix(hermes): clarify auxiliary model policy to match audit' (#140) from fix/hermes-aux-model-policy-20260927 into master	2026-09-27 10:05:44 +00:00
root	d4e238047d	fix(hermes): clarify auxiliary model policy to match audit ... The 'Auxiliary Tasks (CONSISTENCY RULE)' header claimed ALL auxiliary services must use an identical model (gpu-vision) and that syslog-auto must never be used for auxiliary tasks. Both are false for compression, which the script (and Rule 7/8) require to be syslog-auto. An agent following the template produced a config the audit then failed. - Split auxiliary into two classes: light (vision, web_extract/browsing) -> gpu-vision (RTX 5070); context-heavy (compression) -> syslog-auto, citing the existing 2026-07-23 OPERATIONAL DECISION in Rule 7. - Remove the absolute 'Do NOT use syslog-auto for auxiliary tasks' line. - State gpu-dense + strix-moe are the reasoning hosts; do not pin aux to them. - Note the change in the frontmatter UPDATED log. audit-hermes-config.py unchanged (it is the enforcement contract); prose now matches it line-for-line on vision/web_extract/compression. Verified: prose-lint.sh PASS, secret-scan.sh clean, 14/14 tests in tests/test_audit_hermes_config_alias.py pass, audit PASSES on the template's stated policy.	2026-09-27 09:46:37 +00:00
mumuni-bot	73d5097555	Merge pull request 'feat(memory-fixer): add duplicate-detection phase; correct stale canonical pointer + reporting model (v2.2.0)' (#139) from fix/memory-fixer-duplicate-detection-20260926 into master ... Merge PR #139 — memory-fixer duplicate-detection phase (v2.2.0). All CI green; authorized by AGENTS.md step 4 (all green -> merge) and the authorization table (memory-fixer = normal sensitivity, any registered agent).	2026-09-27 02:16:43 +00:00
Mumuni	c460ef905c	feat(memory-fixer): add duplicate-detection phase; correct stale canonical pointer + reporting model (v2.2.0) ... Level 1 gains phase 5, duplicate-node detection: runs memory_dup_detect.py (read-only) and reports clusters by verdict. - WRITER-DEFECT (run_family): one task creating a node per run -> writer fix, never merge (per-run nodes are the audit trail) - SAFE-MERGE: identical bodies, still needs an explicit Kwame decision - HUMAN-DECISION: same subject, bodies differ -> connect, never merge Replaces the naive '>70% title overlap' duplicate rule, which false-positives on distinct work: four client workflows (#357-#361) and two machines' migrations (#1792/#1793) score high on titles with bodies 0.1-0.3 apart. Also corrected in the same pass: - the 'canonical copy' pointer named /root/.hermes/contracts/memory-fixer-v3.md, which does not exist on kagentz (no /root access); the job's instruction set is inline in ~/.hermes/cron/jobs.json - 'reports to Kwame via this Zulip DM' described the pre-2026-09-21 model; the report is DROPped to the gate (comms_drop.py) and exit 0 means QUEUED, not sent - added a Checks line so a skipped phase 5 is visible in the report	2026-09-26 19:32:36 +00:00
abiba-bot	d99b552448	Merge pull request 'feat(search): agent-consumption layer — dedupe, filter, rerank, extract content' (#138) from feat/search-agent-consumption-20260926 into master	2026-09-26 16:06:47 +00:00
abiba-bot	fc0cd7a032	Merge pull request 'feat(daily-digest): deliver via Zulip DM as an HTML attachment; drop mail entirely' (#137) from fix/daily-digest-zulip-delivery-20260926 into master	2026-09-26 16:06:35 +00:00
root	ba38efcd75	test(search): quality guard so the ranking layer cannot silently rot ... Extends search-stack-visibility with a ranking assertion: for the fixed query set, no config demote_domains host may appear in the top 3, and the known non-answers (bestbuy.com, merriam-webster.com) must not be returned at all. Without this the layer could rot back to raw engine ordering unnoticed - the same way the endpoint colours silently rotted before 2026-09-26. It reads the demote list from the SAME config the layer uses, so the guard cannot drift from the policy it is guarding. Live: 'ok: no demoted host in the top 3; no banned non-answer returned'; visibility contract still PASSES end to end.	2026-09-26 15:44:10 +00:00
root	0d30091f62	feat(search): agent-consumption layer - dedupe, filter, rerank, extract ... Raw multi-engine aggregation had no dedupe, no filtering and no reranking. Measured 2026-09-26: 'best practices agent context management' returned bestbuy.com and merriam-webster.com, plus 4 content farms, with medium.com twice; 'proxmox thin pool metadata exhaustion recovery' put four SEO blogs ABOVE the real Proxmox forum threads. Identical queries also ranked DIFFERENTLY between runs, which is why the fix is deterministic rather than trusting the engines. scripts/search-agent-consume.py: 1. dedupe by normalised URL (tracking params and fragments stripped) 2. drop non-answers - shopping/dictionary hosts, navigational host roots, search/shopping/cart/login paths and query keys 3. demote content farms and promote primary sources 4. STABLE sort (score desc, then original position) so runs are reproducible 5. extract page text for the top N via Firecrawl POST /v1/scrape under an explicit character budget, so an agent gets usable material in ONE call 6. emit stable JSON with engine provenance and source_type Policy is config, not code: config/search-ranking.yaml holds demote_domains, prefer_domains, non_answer rules and the extraction budget, so it is reviewable and changeable without touching the module. Content farms are DEMOTED rather than dropped so a useful hit is not lost, it just cannot outrank a primary. A '/products/' path rule was REMOVED after the before/after run caught it dropping docs.digitalocean.com/products/inference/... - a legitimate docs page. Shopping is caught by the host list instead, which has no such false positive. Measured: 'best practices...' top 8 becomes anthropic, langchain, jetbrains, blog.jetbrains, docs.langchain, reddit, cursor, reddit - no content farm. 'proxmox thin pool...' moves the forum threads from positions 5-9 to 1-4. Extraction: 5 items, 12000 chars of 12000 budget, 0 failures, 5.28s; whole run 6.4s wall. Contract: search-agent-consumption.prose.md, including the honest reachability gap - the pi MCP search server's shape is not ours to change, so this layer is NOT wired into it.	2026-09-26 15:44:09 +00:00
root	0a41a2d584	fix(daily-digest): probe Firecrawl on its real liveness path and classify endpoints per fleet policy ... Folds into the same branch as the Zulip delivery change, as instructed. FINDING 1 - the Firecrawl probe path was wrong; the service is fine. scripts/daily-infra-report.py probed http://192.168.68.7:3002/health, which Firecrawl does not serve - it 404s. The root answers 200 with {"message":"Firecrawl API",...}. Live 2026-09-26: Firecrawl(/) -> 200 Firecrawl(/health) -> 404 <- what the report was showing The probe is now the root, which is its liveness endpoint. FINDING 2 - the Network Endpoints classification was wrong twice over. It read: color = green if code in (200,302,401) else (yellow if code >= 400 else red). Two defects: (a) it ignored the fleet's own probe policy, codified 2026-09-14 in the monitoring contracts: ANY HTTP status proves the service answered, so the service is ALIVE, and only a failed CONNECTION is a failed probe. A 404 from a wrong path is not a service fault. (b) was a STRING comparison. Reproduced: 301 -> red (a live redirect rendered as a failure), 404 -> yellow, 500 -> yellow (a real server error softened to a warning). Replaced with classify_endpoint(), which returns: any 2xx/3xx/4xx -> green 'alive' (code still shown) 5xx -> yellow 'server error' (kept distinct from 4xx, as asked) 000/no answer -> red 'no connection' Verified against the live endpoints after the change: Gitea 200, Authentik 302, Zulip 302, Pulse 200, Proxmox 200, SearXNG 200, Firecrawl 200 - all green/alive; the only red state is a genuine no-connection. ALSO CHECKED, as asked: scripts/search-stack-check.py does NOT depend on the wrong route. It POSTs to {FIRECRAWL_URL}/v1/scrape with formats=[markdown], and that path really works - live POST returned HTTP 200 and 180 chars of markdown for https://example.com. It was never using /health. prose-lint: PASSED.	2026-09-26 15:36:38 +00:00
root	de32f54337	feat(daily-digest): deliver via Zulip DM as an HTML attachment; drop mail entirely ... Captain's decision 2026-09-26, clarified the same day: the digest is delivered to his Zulip DM (user id 9) from abiba-bot as an HTML FILE - an attachment, not HTML rendered in the message body and not a Markdown translation of it. Closes daily-digest-mail-transport-20260921; the Google dependency is gone (no SMTP, no EMAIL_PASSWORD, no app password, nothing to rotate). WHAT CHANGES * scripts/daily-infra-report.py: send_email() is replaced by send_zulip(), which writes the styled dashboard to /var/log/daily-infra-report/infra-report-<ts>.html, uploads it via POST /api/v1/user_uploads, then posts a SHORT Markdown pointer to user 9. The message body carries subject, top-line status and the attachment link; it does not reproduce the report. * the 10,000-character cap is irrelevant here - it bounds message TEXT only, and the report travels as a file, so nothing is shrunk to fit. * the key is abiba-bot's, already on the execution host at /root/.pi/agent/extensions/zulip/.env (mode 600). No vault entry was added: under the auth-keys charter that is a captain decision. * daily-health-digest.prose.md -> v2.0.0 and contract-registry.yaml updated: transport, healthy/degraded definitions, and exit codes now match observed behaviour. There is NO degraded delivery leg any more - delivery is the only output path, so a missing or rejected key is a real failure (exit 1). * queued defect folded in: a failed delivery used to print only the transport error while the report body never surfaced. Now the HTML is printed to stdout AND persisted on every failure, and the message names which step failed. EVIDENCE (all against the live stack) * real send: message id 86221 to user 9, attachment 16208 bytes at /user_uploads/2/45/m1cQesBFV78BGeNY2lN8xkN5/infra-report-20260926-153406.html * the message is type=private, sender abiba-bot@chat.sysloggh.net, recipients [9, 21], body carries the top-line status and the attachment link, and does NOT contain a <table> - i.e. it does not reproduce the report * the attachment fetches HTTP 200, 16208 bytes, content-type text/html, starts with <!DOCTYPE html>, and contains <style>, <table> and 16 class="card" blocks - it opens as a standalone styled document * failure path: a bad key gives 'Delivery FAILED at upload: Malformed API key', EXIT=1, the HTML is printed to stdout and persisted to disk * scheduled path: the run's own output is pasted in the PR prose-lint: PASSED (19 warnings); secret scan clean.	2026-09-26 15:35:36 +00:00
abiba-bot	042fd3ccc6	Merge pull request 'fix: reconcile the health-logs self-heal contracts, and add a dead-man's-switch' (#136) from fix/health-log-freshness-watchdog-20260926 into master	2026-09-26 15:16:36 +00:00
root	226f2ad55e	fix: reconcile the health-logs self-heal contracts, and add a dead-man's-switch ... relay-785 / self-heal-contract-stale-script-pointers-20260924. gpu/ had been silent since 2026-09-14T18:02:03Z (12 days) and pm2/ had never posted a run. FINDING - the gpu-self-heal executor was never lost, only its schedule was. The brief concluded the mechanism was gone. It is not: on CT 116 /opt/inference-harness/scripts/gpu-self-heal.py exists (mtime 2026-09-21 11:43), is self-posting via gitea-logger.sh, and runs cleanly - I ran it and it posted health-logs/gpu/gpu-self-heal-20260926-144618.json. What was missing was the cron entry, dropped during a CT 116 /etc/cron.d rework on 2026-09-21. So this is a schedule restoration, not a resurrection. DECISIONS * gpu-self-heal -> RESTORE. Schedule re-added as /etc/cron.d/gpu-self-heal on CT 116, matching the observed historical cadence (:02 past 0/6/12/18). Proven with a REAL unattended cron fire (temporary */1 entry, removed after): gpu-self-heal-20260926-144802.json committed 14:48:03Z. The contract now names the executor, schedule, log and posting, which it previously did not. * pm2 health-logs posting -> RETIRE the claim. pm2-self-heal.prose.md required appending to health-logs/pm2/ as a 'hard rule', but scripts/pm2-self-heal.sh contains no Gitea or push code and never did; the directory has held only its init commit since 2026-07-28. Corrected to point at the contract-runner's durable per-run logs and failure note instead of adding a second, redundant posting path. DEAD-MAN'S-SWITCH - scripts/health-log-freshness.py + contract. Absence of logs must raise an alarm, and the producer cannot raise it, so this runs on CT 100, a different host from the producers, and fails when the newest health-logs/gpu entry is older than 12h (litellm 18h). Verified it would have caught the real gap: evaluated at 2026-09-20 the newest gpu/ entry was 126h old against a 12h limit -> STALE. Bugs found and fixed while testing, each caught by a test that bit: * the documented HEALTH_LOG_MAX_AGE_* override was never implemented; * a Gitea password held under GITEA_TOKEN was sent as an API token -> HTTP 401; now every candidate auth is tried and the first that works is used; * the ~/.git-credentials fallback filtered on GITEA_URL's host, which missed whenever GITEA_URL pointed at the internal IP -> zero candidates. Exit codes: 0 fresh, 1 stale-or-unreadable, 2 cannot run. A directory that cannot be read is a failure, never a skip. Evidence: live PASS; stale override names the directory and producer; no credential -> exit 2; whole thing runs green through contract-run.sh. prose-lint: PASSED.	2026-09-26 14:50:52 +00:00
abiba-bot	552c776c0c	Merge pull request 'feat: land the revision-preflight guard, fixed and wired into contract execution' (#134) from fix/land-revision-preflight-guard-20260925 into master	2026-09-25 11:46:29 +00:00
root	778424acd4	Merge remote-tracking branch 'origin/master' into fix/land-revision-preflight-guard-20260925	2026-09-25 11:29:30 +00:00
root	c5a6dcd42a	fix(revision-preflight): default to warn, name the refusal class, bound the fetch ... Bounded correction round on PR #134 after a PASS-WITH-FINDINGS review whose Finding 4 is High. The guard's purpose and its fail-closed fix stand; the problem was that with 'enforce' as the default it gates EVERY scheduled contract, and three legitimate states produce a refusal - a clone legitimately ahead of origin/master mid-review, a detached HEAD, and an offline or failed fetch - so any of them would turn the fleet's monitoring into withheld verdicts. That risk outweighs the staleness the guard catches. 1. DEFAULT IS NOW 'warn'. 'enforce' remains available and documented. The criteria for flipping the default later are written into the doc as a decision with evidence - a sustained window (30 days / 200+ runs) with zero mismatch:* and zero cannot-verify:* refusals, no fetch blips, and a pinned clone demonstrably kept current - explicitly as its own change, not a silent flip. 2. 'COULD NOT CHECK' IS NOW DISTINGUISHABLE FROM 'THIS COPY IS WRONG'. Every non-zero exit prints a machine-readable REASON=<class> line: cannot-verify:fetch-failed | cannot-verify:ref-unresolvable (exit 2) mismatch:path-absent | mismatch:content mismatch:detached-head | mismatch:clone-ahead (exit 1) detached-head and clone-ahead are named separately because they are legitimate states, far less alarming than a hand-edited file. clone-ahead requires HEAD to be STRICTLY ahead; an uncommitted edit on a commit that IS the ref is a plain content mismatch (my own first cut got this wrong and the new test 7d caught it). 3. THE DEFAULT FETCH IS BOUNDED: --fetch-timeout, default 20s, 0 = unbounded, and a missing 'timeout' binary is itself a cannot-verify rather than an unbounded fetch inside a scheduled contract. 4. TEST COVERAGE ADDED for every new class: fetch failure, fetch timeout (asserted to return promptly under a 1s bound), unresolvable ref, detached HEAD, clone-ahead, genuine content mismatch, and the contract-run.sh default. The pre-fix draft fixture comparisons are kept: 31 passed, 0 failed. 5. MERGE-TIME SEQUENCE documented: fast-forward /opt/contract-runner, confirm clean, prove a contract runs and reports. Baseline recorded as of today - firstmate has already fast-forwarded it to 9faffe4 - with the note that an untracked file blocks a fast-forward even when byte-identical. Live behaviour re-verified on the real runner path: default: REASON=mismatch:clone-ahead -> 'continuing because ...=warn' -> VERDICT: PASS, exit 0 enforce: REASON=mismatch:clone-ahead -> 'VERDICT WITHHELD: mismatch:clone-ahead', exit 2 MANDATORY CHECKS (master went red once from a credential-SHAPED string, so these are now run on every shippable branch): bash scripts/prose-lint.sh -> LINT PASSED (18 warning(s)) secret scan -> secret scan clean (tree; 34 allowlisted, 24 inert value(s) ignored); No committed credentials shellcheck revision-preflight.sh -> clean shellcheck test_revision_preflight.sh -> clean shellcheck contract-run.sh -> SC2034 x1, SC2086 x2 - byte-identical on master, i.e. pre-existing, none introduced tests/test_probe_drift.py::test_prose_lint_accepts_report_format_with_provenance fails both before and after this branch (it runs prose-lint from a temp CWD and cannot find its sibling secret-scan.sh). Pre-existing, unrelated, not fixed here.	2026-09-25 11:29:15 +00:00
abiba-bot	9faffe4f6b	Merge pull request 'feat: add the missing daily-health-digest contract + fix the red lint from #133' (#135) from fix/daily-health-digest-contract-20260925 into master	2026-09-25 11:12:56 +00:00
root	cd00cd0475	feat: add the missing daily-health-digest contract and register it ... Backlog row daily-health-digest-contract-missing-20260925. The digest has been dispatched on a schedule with NO contract file at all - no *daily*.prose.md, absent from contract-registry.yaml, the only reference anywhere being the CT100 cron line. That absence is why the choice of execution copy was silently the operator's, and how a stale clone could run the check unnoticed. New daily-health-digest.prose.md states: * the PINNED execution path /root/abiba-workspace/projects/prose-contracts/ scripts/daily-infra-report.py and the pinned clone - the cron's FM_HOME clone, the only stable non-ephemeral copy; the treehouse clone is a per-agent working copy and must NOT be pinned; * the output shape (all 18 top-level --json keys) and what a healthy run is; * the exit-code semantics AS THEY ACTUALLY BEHAVE, verified case by case: missing PVE_TOKEN or an unreachable probe exits 1 and raises an alert, while a missing EMAIL credential is a deliberate DEGRADED leg that still exits 0 and still produces the report. PROBE_FAILURES and DEGRADED_LEGS are separate lists on purpose and must not be merged; * the email-delivery dependency, that EMAIL_PASSWORD must be a Google app password, that it is failing with 534 5.7.9 as of 2026-09-25, and that a delivery failure is a credential dependency rather than a code defect; * what counts as a failure versus degraded. Registered in contract-registry.yaml (contracts entry plus index.by_category .monitoring and index.by_domain.infrastructure). Verified: YAML parses, 31 contracts, exactly one daily-health-digest entry.	2026-09-25 11:08:28 +00:00
root	819cd53bce	fix(lint): restore the repo secret scan, which PR #133 broke ... Master's lint is RED right now, and it is my doing. at 483a66b (before #133): prose-lint -> LINT PASSED at 9d64b0b (after #133): prose-lint -> LINT FAILED, 4 credential-shaped strings The regression came from #133's new pve_auth() work. The secret scanner flags the literal header shape PVEAPIToken=<value> (rule proxmox-token), and three occurrences landed in the tree: scripts/daily-infra-report.py the f-string building the auth header tests/test_daily_infra_report.py a docstring quoting the old placeholder tests/test_daily_infra_report.py a synthetic token in an assertion Fixed without allowlisting anything, because none of these is a credential: * the header prefix becomes PVE_AUTH_HEADER = "PVEAPIToken=", a constant ending at '=' so the scanner's pattern (which needs a character after '=') cannot match, and the f-string no longer contains the literal; * the test docstring no longer reproduces the old placeholder verbatim; * the test builds its expected value from the constant plus a local sample variable instead of embedding a credential-shaped literal. Behaviour is unchanged and re-verified: with PVE_TOKEN unset the script still exits 1 with the probe failure, and with the vault token it still reports pve_probe_status ok / node_count 5 / nodes_online 5. before: LINT FAILED — 4 credential-shaped strings after: LINT PASSED (18 warnings)	2026-09-25 11:08:28 +00:00
root	574cb99d76	feat: land the revision-preflight guard, fixed and wired into contract execution ... A contract verdict is only meaningful if it came from the merged copy. The fleet has been bitten three times on 2026-09-25 (a clone parked on a merged feature branch while executing from another clone; a script copied into the runner clone by hand; a stale local origin/master making an ancestry check report unlanded work). The control for this existed as an untracked draft and protected nobody, because it was entirely fail-open. Defect in the draft, preserved verbatim as tests/fixtures/revision-preflight.prefix.sh: git -C "$CLONE" show "origin/master:$(basename "$SCRIPT")" basename drops the scripts/ prefix, so for any script under scripts/ it queried the repo root, failed, took the "warn but don't block" branch and exited 0 - passing a script that exists in no revision at all. Reproduced: pre-fix + scripts/demo.sh under scripts/ -> 'could not resolve', EXIT=0 pre-fix + a script in no revision -> EXIT=0 Fixed guard (scripts/revision-preflight.sh): * resolves the repo-relative path inside the clone, so scripts/ paths resolve; * FAILS CLOSED - a path absent from the ref, an unresolvable ref, or a failed fetch is a failure, never a warning; * fetches the remote by default, because a stale local ref would otherwise pass a stale script as current; --no-fetch states the assumption instead of hiding it. Wiring (scripts/contract-run.sh): before executing, the wrapper runs the guard against the clone it lives in. Default CONTRACT_REVISION_PREFLIGHT=enforce withholds the verdict, alerts and exits 2 on mismatch; =warn logs and continues; =off skips. Verified live: match -> contract proceeds and PASSes; mismatch -> 'VERDICT WITHHELD', exit 2; =warn -> continues. Pinning (docs/contract-execution-pinning.md): every contract pins the clone contract-run.sh lives in - the deployed runner being /opt/contract-runner on CT 100. Documented that daily-health-digest has no contract file at all, which is why its execution copy was silently operator-chosen. Tests: tests/test_revision_preflight.sh, 15 assertions over a throwaway clone with a real bare remote. It runs the pre-fix draft against the same cases and shows it passing a ghost script, so the tests provably bite. shellcheck: scripts/revision-preflight.sh and the new test are clean. The three findings remaining in contract-run.sh (SC2086 x2, SC2034) are pre-existing and byte-identical on master.	2026-09-25 11:04:29 +00:00
abiba-bot	9d64b0bd66	Merge pull request 'fix(daily-infra-report): resolve PVE token from env; make a dead probe non-silent' (#133) from fix/daily-health-digest-pve-token-20260925 into master	2026-09-25 10:51:41 +00:00
root	cdc7ad2c79	fix(daily-infra-report): resolve PVE token from env; make a dead probe non-silent ... The Proxmox leg of the daily digest has been reporting NOTHING while exiting 0. Root cause: the auth header was a literal placeholder string, AUTH = "Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»" which was sent verbatim. The API rejected it, pve_get() returned None, and the report rendered node_count=0 / nodes_online=0 with pve_probe_status='unreachable' while still exiting 0. A monitoring gap that looks like a healthy run. Also: the vault key is PVE_TOKEN, not PVE_API_TOKEN, so even reading os.environ by the old name would not have found it. Fixes: * pve_auth() resolves the token at call time from PVE_TOKEN (injected by 'infisical run --env=prod'). Nothing is hardcoded; a missing token raises. * pve_get() builds the command inside its try block, so a missing token degrades to None instead of escaping as an unhandled exception. * PROBE_FAILURES records an unreachable node/resources probe. Probe failures are deliberately separate from DEGRADED_LEGS: a missing credential stays exit 0 (existing intent), but a probe with no data now exits 1 in both the report and --json paths, so it cannot pass unnoticed. Measured effect on the live host: pve_probe_status unreachable -> ok, node_count 0 -> 5, nodes_online 0 -> 5, total_vms 0 -> 22, running_vms 0 -> 22. Tests: 4 new regression tests; all 4 fail against the pre-fix script and pass after, and the 3 pre-existing tests still pass (7/7). Not fixed here (needs the captain): the email leg fails with '534 5.7.9 Application-specific password required' - EMAIL_PASSWORD in the vault is not a valid Gmail app password for jtabiri@gmail.com. That is a credential action, not a code change.	2026-09-25 10:34:07 +00:00
abiba-bot	5409dfd73a	Merge pull request 'feat: multi-engine search stack + visibility check' (#132) from fix/search-stack-multi-engine-20260925 into master	2026-09-25 01:31:55 +00:00
root	8b2eba4f7a	docs+check: correct DDG status, explain silent-zero semantics, credit google cse ... Follow-up corrections after review: 1. DuckDuckGo is NOT fixed. The VPS fallback egress has since been flagged by DuckDuckGo too (HTTP 202 + challenge markers), so it reports CAPTCHA on both paths. The contract and script docstring now say so instead of claiming a fix that had already expired. It stays enabled as best-effort coverage so a recovery shows up as a contribution. 2. The relationship between silent zeros and the verdict is now explicit in both the script output and the contract: an enabled expected engine that contributes zero with no error is REPORTED, not fatal. Only the <SEARCH_CHECK_MIN_ENGINES> floor and the extraction leg fail the run. This is deliberate - de-duplication and query-shape make a zero non-probative. 3. Recorded that 'google cse' uses a THIRD PARTY's public search-engine id hardcoded in the SearXNG build, not a key we own; its quota and availability are outside our control, and our own free key would need a wrapper (not built). VPS forward proxy is now a real service: /opt/fwd-proxy docker compose with restart: unless-stopped, a healthy healthcheck, and Docker enabled at boot.	2026-09-25 01:19:27 +00:00
root	040fecef3e	feat: multi-engine search stack + visibility check ... Search stack (192.168.68.7) was effectively Bing-only: google served a JS shell, duckduckgo CAPTCHA'd from the house egress, and every other shipped engine returned a silent zero. Upgraded SearXNG to 2026.9.23 (same pinned digest as the image already pulled by other hosts) which uses browser impersonation, and routed DuckDuckGo through a VPS forward proxy over the existing WireGuard tunnel via a per-engine 'network'. Live result: bing, google cse, brave and yandex contribute on every query; duckduckgo is best-effort via the datacenter egress. Adds the visibility leg so a future regression cannot be silent: scripts/search-stack-check.py * two fixed queries; FAILS when fewer than two engines contribute, printing contributing engines and every unresponsive_engines entry * FAILS when Firecrawl extraction returns empty markdown or errors * reports silent-zero engines explicitly scripts/contract-run.sh * maps search-stack-visibility -> search-stack-check.py search-stack-visibility.prose.md * contract text, execution model, pass/fail shapes, residual risk Scheduled hourly at :15 on CT 100 via /etc/cron.d/contract-runner.	2026-09-25 01:14:03 +00:00
abiba-bot	483a66b7b4	Merge pull request 'Fix PBS GC monitor false positive and add contract-run.sh wrapper' (#131) from fix/contract-run-pbs-gc-20260924 into master	2026-09-24 06:07:08 +00:00
root	c0d04a2c02	fix: three wrapper holes in contract-run.sh ... 1. Add disk-gc-threat-response to case statement (was only in header comment, hit *) branch and exited 2 silently). Mapped to scripts/disk-gc-scan.py per contract's Execution section. 2. Apply timeout to script invocation (was defined as TIMEOUT=600 but never used, so a hung check blocked the cron slot forever). Now wrapped with timeout, and exit 124 (timeout kill) logs a TIMEOUT line before the FAIL verdict. 3. Send alert on exit-2 paths (unknown contract and missing script). Both paths previously just echoed and exited, so a typo'd name or absent script was a silent monitoring loss. Now they send the same Zulip DM as a failed check. Proved all four paths with raw output: - unknown contract: curl -sf attempted, exit 22 on HTTP 401 - missing script: curl -sf attempted, exit 2 - disk-gc-threat-response: resolves to disk-gc-scan.py, runs, PASS - stub sleep > timeout: TIMEOUT line logged, exit 1, alert failure recorded	2026-09-24 06:01:01 +00:00
root	fda6c844ff	fix: add -f to curl to fail on HTTP >= 400 ... Without -f, a rejected credential (HTTP 401) returns curl exit 0, making a failed alert indistinguishable from a successful one. With -f, curl exits non-zero on HTTP >= 400, so DM_EXIT and STREAM_EXIT correctly capture the transmission failure and the run log records it.	2026-09-24 05:27:28 +00:00