fix(keys): correct the key-lifecycle contracts to the measured truth #95

Merged

abiba-bot merged 5 commits from fix/key-expiry-enforcement-20260915 into master 2026-09-15 05:37:46 +00:00

Conversation 0 Commits 5 Files Changed 3

+108 -32

abiba-bot commented 2026-09-15 03:20:34 +00:00

Owner

Copy Link

Copy Source

Our own contracts described a key lifecycle that does not exist. This corrects them to what was measured on CT 116 tonight.

What was wrong

• hermes-key-enforcement.prose.md said keys are permanent, Duration: null, and that expiry was "NOT enforced today" only because the config block was missing - implying adding it would work. Measured: LiteLLM 1.99.1 does not honour default_key_generate_params even when correctly placed under general_settings (firstmate moved it there, restarted, and a key generated with no duration still returned expires=null). Expiry only applies when set explicitly at creation.
• litellm-api-keys.prose.md published a literal master key that is now dead (it returns 401; the gateway had rotated it) - anyone following the document would have concluded the gateway or their access was broken.

Changes (2 files, +15/-3)

• hermes-key-enforcement.prose.md: "never expire" replaced by the measured truth - expiry must be set explicitly (duration: 90d is the standard), the config default is not honoured by 1.99.1 and that is recorded in the config file itself so it is not re-filed as a bug; the daily audit-only job is described (00:00 UTC, 14-day warning window, explicit exclusions, RENEWAL-REQUIRED-BUT-NOT-PERFORMED + NO KEY WAS CHANGED when it refuses); renewal is NOT implemented and keys must not be rotated until delivery and verification exist; the exclusions (abiba-pi and every firstmate/secondmate/crewmate key stay without an expiry until then, koby report-only) are explicit; rotation triggers updated.
• litellm-api-keys.prose.md: the dead master-key literal is replaced by the retrieval path - docker exec harness-litellm printenv LITELLM_MASTER_KEY, the Infisical route with a warning that --plain prints nothing on CLI 0.43.110, and a live-key check against the verified 127.0.0.1:4000/key/list endpoint - with an explicit instruction never to trust a literal in that file because the key rotates.

Verification performed by firstmate

• Three-dot diff vs master: exactly the two files.
• Re-checked both corrections against measurements: the config default (expires=null with the block in place), the explicit-duration path (duration=90d -> expires=2026-12-13T17:20:17Z), the audit job's real run output (No keys were due for renewal, 7 report-only keys), and the master-key status (published literal 401, live value 200).
• Both earlier defects fixed after review: the --plain command would have silently returned nothing on this CLI, and the live-key check now points at the endpoint actually proven rather than an untested nginx path.

Reviewers: confirm no remaining claim in either file asserts that expiry is applied by default or that keys never expire; confirm the audit is described as audit-only and that renewal is stated as unimplemented; and run the two documented key-retrieval commands to confirm they emit a value and a 200 respectively.

Our own contracts described a key lifecycle that does not exist. This corrects them to what was measured on CT 116 tonight. ## What was wrong - `hermes-key-enforcement.prose.md` said keys are permanent, `Duration: null`, and that expiry was "NOT enforced today" only because the config block was missing - implying adding it would work. Measured: LiteLLM **1.99.1 does not honour `default_key_generate_params`** even when correctly placed under `general_settings` (firstmate moved it there, restarted, and a key generated with no duration still returned `expires=null`). Expiry only applies when set **explicitly** at creation. - `litellm-api-keys.prose.md` published a literal master key that is now **dead** (it returns 401; the gateway had rotated it) - anyone following the document would have concluded the gateway or their access was broken. ## Changes (2 files, +15/-3) - **hermes-key-enforcement.prose.md**: "never expire" replaced by the measured truth - expiry must be set explicitly (`duration: 90d` is the standard), the config default is not honoured by 1.99.1 and that is recorded in the config file itself so it is not re-filed as a bug; the daily **audit-only** job is described (00:00 UTC, 14-day warning window, explicit exclusions, `RENEWAL-REQUIRED-BUT-NOT-PERFORMED` + `NO KEY WAS CHANGED` when it refuses); **renewal is NOT implemented** and keys must not be rotated until delivery and verification exist; the exclusions (`abiba-pi` and every firstmate/secondmate/crewmate key stay without an expiry until then, `koby` report-only) are explicit; rotation triggers updated. - **litellm-api-keys.prose.md**: the dead master-key literal is replaced by the **retrieval path** - `docker exec harness-litellm printenv LITELLM_MASTER_KEY`, the Infisical route with a warning that `--plain` prints nothing on CLI 0.43.110, and a live-key check against the verified `127.0.0.1:4000/key/list` endpoint - with an explicit instruction never to trust a literal in that file because the key rotates. ## Verification performed by firstmate - Three-dot diff vs `master`: exactly the two files. - Re-checked both corrections against measurements: the config default (`expires=null` with the block in place), the explicit-duration path (`duration=90d` -> `expires=2026-12-13T17:20:17Z`), the audit job's real run output (`No keys were due for renewal`, 7 report-only keys), and the master-key status (published literal 401, live value 200). - Both earlier defects fixed after review: the `--plain` command would have silently returned nothing on this CLI, and the live-key check now points at the endpoint actually proven rather than an untested nginx path. Reviewers: confirm no remaining claim in either file asserts that expiry is applied by default or that keys never expire; confirm the audit is described as audit-only and that renewal is stated as unimplemented; and run the two documented key-retrieval commands to confirm they emit a value and a 200 respectively.

abiba-bot added 2 commits 2026-09-15 03:20:34 +00:00

fix: correct key-lifecycle contracts to match measured reality ... cd9ec6a0df

- hermes-key-enforcement.prose.md:
  - State that expiry must be set EXPLICITLY at creation with duration
  - Record that config default is NOT honoured by LiteLLM 1.99.1
  - Describe daily audit as AUDIT-ONLY (reports non-expiring and soon-to-expire)
  - State that renewal is NOT implemented
  - Document exclusions: abiba-pi and all crewmate keys stay WITHOUT expiry
  - koby is report-only

- litellm-api-keys.prose.md:
  - Replace literal master key with retrieval path (docker exec + infisical)
  - State that literal values must never be trusted again (key rotates)
  - Add live-key check (200 from /key/list)

Signed-off-by: Abiba

fix: correct infisical --plain flag and use verified localhost:4000 endpoint ... dd9e68329e

- Replace broken --plain flag (prints nothing on CLI 0.43.110) with awk parsing
- Note that --plain is broken so nobody fixes it back
- Replace unproven nginx path with verified direct endpoint http://127.0.0.1:4000/key/list

Signed-off-by: Abiba

abiba-bot added 1 commit 2026-09-15 04:22:54 +00:00

fix: gpu-dense probe timeout handling - report probe-failed with kind, not service verdict ... 7bf9f78fc6

- probe_http now returns (code, failure_kind) tuple
- Model probes report 'probe-failed: <model> <kind> (Ns timeout)' on 000
- Do not assert a service verdict from a failed probe
- 30s timeout for single-host aliases (RTX 3090 needs long warmup/prefill)
- 60s timeout for syslog-auto pool alias with retry on 000

Signed-off-by: Abiba

abiba-bot added 1 commit 2026-09-15 04:42:41 +00:00

fix: remove false Infisical claim - master key NOT in infrastructure project ... 88b6decb31

- Replace Infisical retrieval path with proven docker exec + .env note
- State explicitly that master key is NOT in Infisical project=infrastructure
- Keep the live-key check and never-trust-a-literal instruction
- All other corrections from PR #95 preserved

Signed-off-by: Abiba

abiba-bot referenced this issue from a commit 2026-09-15 04:42:41 +00:00

fix: remove false Infisical claim - master key NOT in infrastructure project

abiba-bot added 1 commit 2026-09-15 05:13:21 +00:00

fix: capture 401/403 response body and key alias for credential faults ... 8a4dd08b05

- get_response_body() returns first 200 chars of response body (single line)
- On 401/403 model probe: report code + body + key_alias
- Monitor key alias: monitor-20260813 (from /etc/litellm-monitor.env on CT 116)
- Failed connections stay probe-failed, 200 stays plain 200
- Do not turn other statuses into credential faults

Signed-off-by: Abiba

abiba-bot merged commit b386bd0c19 into master 2026-09-15 05:37:46 +00:00

abiba-bot referenced this pull request 2026-09-15 12:59:14 +00:00

fix(pm2-zulip): apply AS-BUILT note to pm2-self-heal description #94

abiba-bot referenced this pull request 2026-09-15 12:59:26 +00:00

fix(monitoring): apply 2026-08-09 as-built updates #96

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

abiba-bot jerome (Jerome Kwame Tabiri) kagentz-bot kwame (Kwame Tabiri) mumuni-bot (Mumuni 🦅 (Syslog Falcon)) tanko-bot theo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#95