fix(monitoring): make credential sourcing explicit and fail loudly when missing #83

2026-09-12T23:19:09Z

abiba-bot commented

2026-09-12 23:19:09 +00:00

Follow-up to PR #82 (merged). It was pushed to that same branch after #82 merged, so no new PR existed until now.

Second and third instance of the credential-sourcing defect (task infra-monitor-envfile-sourcing-20260912). Tonight's litellm-health run at 23:09Z produced TWO false negatives:

"Model inference: 401 (monitor key sk-litellm-monitor not in DB)" - the executor looked for /etc/litellm-monitor.env on CT 100, where it does not exist, then fell back to a placeholder literal.
"Agent keys: 0 found" - the admin call did not use the container's master key.

Live truth verified from CT 116 at 23:12Z: /health/liveliness 200; /key/list with the real master key 200 with TEN keys; the monitor key from /etc/litellm-monitor.env returns 200 for gpu-dense, gpu-vision, strix-moe and syslog-auto.

Changes

litellm-health.prose.md - states the exact retrieval commands that work from the executor's host (ssh to CT 116 for the monitor key and for the container master key); requires a missing/unreadable credential to be reported as credential-missing rather than a bare 401 or "0 keys"; key-scope note covers the syslog-auto pool.
infrastructure-monitoring.prose.md - same fix for the Zulip API probe, which sourced the same CT-116-only file on the wrong host and reported a Zulip 401 on every run.

Verification

Live re-run: model probes return 200 for all four aliases (they returned 401 only because of the missing credential). Reviewers should re-run and paste the leg-by-leg output.

Follow-up to PR #82 (merged). It was pushed to that same branch after #82 merged, so no new PR existed until now. Second and third instance of the credential-sourcing defect (task `infra-monitor-envfile-sourcing-20260912`). Tonight's litellm-health run at 23:09Z produced TWO false negatives: - "Model inference: 401 (monitor key sk-litellm-monitor not in DB)" - the executor looked for `/etc/litellm-monitor.env` on CT 100, where it does not exist, then fell back to a placeholder literal. - "Agent keys: 0 found" - the admin call did not use the container's master key. Live truth verified from CT 116 at 23:12Z: `/health/liveliness` 200; `/key/list` with the real master key 200 with TEN keys; the monitor key from `/etc/litellm-monitor.env` returns 200 for `gpu-dense`, `gpu-vision`, `strix-moe` and `syslog-auto`. ## Changes - **litellm-health.prose.md** - states the exact retrieval commands that work from the executor's host (ssh to CT 116 for the monitor key and for the container master key); requires a missing/unreadable credential to be reported as `credential-missing` rather than a bare 401 or "0 keys"; key-scope note covers the `syslog-auto` pool. - **infrastructure-monitoring.prose.md** - same fix for the Zulip API probe, which sourced the same CT-116-only file on the wrong host and reported a Zulip 401 on every run. ## Verification Live re-run: model probes return 200 for all four aliases (they returned 401 only because of the missing credential). Reviewers should re-run and paste the leg-by-leg output.

abiba-bot added 2 commits 2026-09-12 23:19:09 +00:00

fix: credential-sourcing fix for litellm-health and infrastructure-monitoring d9368467ff

- litellm-health.prose.md: Make monitor key retrieval explicit (ssh from CT 100 to CT 116)
  with executable commands; add syslog-auto alias; document credential-missing failure
  condition (not bare 401 or 0 keys)

- infrastructure-monitoring.prose.md: Fix Zulip POST probe to retrieve keys from CT 116
  via ssh instead of sourcing local env file that doesn't exist on executor host

- Verify model inference probes return 200 for gpu-dense, gpu-vision, strix-moe,
  syslog-auto with corrected credential retrieval

fix: address credential-sourcing review findings

PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s

Details

PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s

Details

PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s

Details

PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s

Details

PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s

Details

6f40a3be60

- infrastructure-monitoring: Correct Zulip key path to /etc/litellm-monitor.env
  (not /etc/zulip-bot.env which doesn't exist); add credential-missing check;
  remove unused monitor_key variable

- litellm-health: Clarify that syslog-auto is a fallback pool alias, not a
  step 7 probe; monitor key must be scoped for all four aliases

abiba-bot merged commit 89651cf37c into master

2026-09-12 23:22:44 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#83