7 Commits

Include renames

Author	SHA1	Message	Date
root	aee2de25ac	fix(litellm): Report both attempts' failure kinds in probe-failed ... The failure line now preserves both attempts' failure kinds instead of hardcoding 'timeout after retry, 45s'. If both attempts fail, the report shows: 'probe-failed: <model> <first kind> then <retry kind> (2 attempts)'. This fixes the self-contradictory output when the first attempt timed out but the retry failed with connection refused, and prevents the duration from appearing twice when both attempts were timeouts. Example outputs: - timeout then timeout: 'probe-failed: gpu-dense timeout after 30s then timeout after 45s (2 attempts)' - timeout then refused: 'probe-failed: gpu-dense timeout after 30s then connection refused (2 attempts)' - refused then refused: 'probe-failed: gpu-dense connection refused then connection refused (2 attempts)'	2026-09-19 10:53:44 +00:00
root	76653381ec	fix(litellm): Add retry with longer timeout for single-host model probes ... Single-host models (gpu-dense, gpu-vision, strix-moe) now retry once at 45s on initial 30s timeout failure before declaring probe-failed. This prevents a single transient timeout (cold prefill ~13s or concurrent generation hold) from failing the entire health digest. Evidence: 2026-09-19 ~06:55Z digest failed gpu-dense at 30s; 06:56Z direct probe 200 in 1.04s. The failed-probe-fails-the-run property is preserved: if both attempts fail, the script still exits non-zero with the target and duration named. Closes: daily-health-digest false negative on single transient timeout	2026-09-19 10:43:54 +00:00
root	7f62f19c24	fix: litellm-key-count-self-describing-20260916 ... The key-count line in litellm-health-check now reports self-describing output: '18 total (10 on page 1)' instead of bare '18' or '4'. Uses total_count from the paginated API response and names what was counted. Previous bare numbers could not reconcile changes between runs; now a reader sees both the total and the page 1 sample.	2026-09-16 15:20:50 +00:00
root	8a4dd08b05	fix: capture 401/403 response body and key alias for credential faults ... - get_response_body() returns first 200 chars of response body (single line) - On 401/403 model probe: report code + body + key_alias - Monitor key alias: monitor-20260813 (from /etc/litellm-monitor.env on CT 116) - Failed connections stay probe-failed, 200 stays plain 200 - Do not turn other statuses into credential faults Signed-off-by: Abiba	2026-09-15 05:11:07 +00:00
root	7bf9f78fc6	fix: gpu-dense probe timeout handling - report probe-failed with kind, not service verdict ... - probe_http now returns (code, failure_kind) tuple - Model probes report 'probe-failed: <model> <kind> (Ns timeout)' on 000 - Do not assert a service verdict from a failed probe - 30s timeout for single-host aliases (RTX 3090 needs long warmup/prefill) - 60s timeout for syslog-auto pool alias with retry on 000 Signed-off-by: Abiba	2026-09-15 04:22:46 +00:00
root	05366bd58d	Fix litellm-health-check.py robustness defects ... 1. Timeout fix for pool alias (syslog-auto): - Single-host aliases (gpu-dense, gpu-vision, strix-moe): 30s timeout - Pool alias (syslog-auto): 60s timeout, retry once on 000 before failing - Cold first request to pool alias can take ~13s; 10s was too short 2. Remove DEBUG prints from output: - Removed 'DEBUG: keylen=...' and 'DEBUG: response=...' lines - These leaked key inventory to status logs - Success output now shows only counts (e.g., 'Admin Key List: 10 keys')	2026-09-14 03:24:30 +00:00
root	e94fadfa60	Add litellm-health-check.py: standardized health check script for LiteLLM fleet monitoring ... Implements all 11 checks from litellm-health contract: - Liveliness, Containers, Prometheus, Grafana health probes - Model probes for gpu-dense, gpu-vision, strix-moe, syslog-auto - Admin API key list (10 keys), GitHub status, Docker Stats metrics Fixed quoting for SSH commands and response parsing (dict with 'keys' field). Backend edge uses internal IP 192.168.68.116, not public URL. Docker Stats fetched from CT 116 host itself (127.0.0.1:9324/metrics). All 11 checks passing consistently.	2026-09-14 03:15:27 +00:00