diff --git a/LITELLM-MIGRATION-PLAN.md b/LITELLM-MIGRATION-PLAN.md index fcf560d..5e72a49 100644 --- a/LITELLM-MIGRATION-PLAN.md +++ b/LITELLM-MIGRATION-PLAN.md @@ -1,5 +1,9 @@ # LiteLLM Integration Migration Plan -## Syslog Solution LLC — June 13, 2026 +## Syslog Solution LLC — June 14, 2026 + +**Deployment Target:** CT 116 `syslog-api` (192.168.68.116) on minipve — all services co-located. +**DNS Strategy:** Option A — /etc/hosts + Docker extra_hosts for internal resolution of `auth.sysloggh.net` → 192.168.68.11. +**GitOps:** This plan lives in `SyslogSolution/syslog-harness` on Gitea. All changes tracked via git with conventional commits. --- @@ -56,7 +60,9 @@ ## 1. Current State Baseline -### 1.1 Router (`router-fixed.py` — port 9000, deployed on CT 116 / docker-vm) +### 1.1 Router (`router-fixed.py` — port 9000, deployed on CT 116 / syslog-api) + +**Deployment Host:** CT 116 `syslog-api` on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host. | Feature | Implementation | |---------|---------------| @@ -79,10 +85,47 @@ | qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K | | gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K | -### 1.3 Existing LiteLLM Attempt -- `/root/litellm-fix.sh` — previous setup script for docker-vm +### 1.3 Existing LiteLLM POC on CT 116 + +CT 116 already has a LiteLLM container running (POC, 6 days uptime): + +``` +harness-litellm | ghcr.io/berriai/litellm:main-stable | 127.0.0.1:8081→4000 +harness-redis | redis:7-alpine | 127.0.0.1:6379 +harness-router | inference-harness-router | 127.0.0.1:9000 +harness-nginx | nginx:alpine | 0.0.0.0:80 +harness-dashboard | inference-harness-dashboard | 127.0.0.1:3000 +``` + +- `/opt/litellm/` — previous setup directory on CT 116 - Configured with Postgres, host networking, master key -- **Never productionized** — still in exploratory phase +- Currently bypassed — router routes directly to GPUs +- **Goal: Productionize with two-layer architecture on this same host** + +### 1.4 DNS Routing (Split-Horizon) + +For OIDC SSO with Authentik, CT 116 must resolve `auth.sysloggh.net` internally: + +**Problem:** `auth.sysloggh.net` CNAMEs to `netbird.sysloggh.net` → 72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily. + +**Solution — Option A: /etc/hosts on CT 116 host:** +```bash +# On CT 116 (syslog-api) +echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts +``` + +**Docker containers** also need this resolution — add to docker-compose.yml: +```yaml +services: + nginx: + extra_hosts: + - "auth.sysloggh.net:192.168.68.11" + litellm: + extra_hosts: + - "auth.sysloggh.net:192.168.68.11" +``` + +**DNS Servers:** CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS. --- @@ -211,6 +254,36 @@ model_list: api_base: http://router:9000/v1 api_key: os.environ/ROUTER_API_KEY +# Guardrails: Pre-call and post-call content moderation +guardrails: + - guardrail_name: "input-moderation" + litellm_params: + guardrail: openai_moderation + mode: "pre_call" + + - guardrail_name: "output-moderation" + litellm_params: + guardrail: openai_moderation + mode: "post_call" + + - guardrail_name: "harmful-content-filter" + litellm_params: + guardrail: litellm_content_filter + mode: "pre_call" + categories: + - category: "harmful_self_harm" + enabled: true + action: "BLOCK" + severity_threshold: "medium" + - category: "harmful_violence" + enabled: true + action: "BLOCK" + severity_threshold: "medium" + - category: "harmful_illegal_weapons" + enabled: true + action: "BLOCK" + severity_threshold: "medium" + litellm_settings: num_retries: 0 # Disabled — our router handles retry request_timeout: 600 # Match our 10-min llama-server timeout @@ -248,90 +321,148 @@ resp.headers["X-Usage-Tokens"] = json.dumps({ --- -## 5. Deployment Plan (3 Phases) +## 5. Deployment Plan (4 Phases) — Zero-Downtime Strategy -### Phase 1: Shadow Mode (Week 1) — Zero Risk +### Phase 0: Infrastructure Prep (Current Week) — Zero Downtime -**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode. +**Goal:** Prepare CT 116 infrastructure without affecting running agents. + +**Tasks:** +1. **Set up DNS split-horizon on CT 116** + ```bash + # On CT 116 host (syslog-api) + echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts + ``` + Add to docker-compose.yml (see §7): + ```yaml + extra_hosts: + - "auth.sysloggh.net:192.168.68.11" + ``` + +2. **Deploy Postgres container** alongside existing services: + ```bash + cd /opt/litellm + # Add postgres to docker-compose.yml + docker compose up -d postgres + ``` + +3. **Replace LiteLLM config** with production config.yaml (see §4.3) + - All 4 models → `http://router:9000/v1` + - Add guardrails (pre-call, post-call, content filter) + - Set `num_retries: 0` (router handles retry) + - Set `request_timeout: 600` + +4. **Deploy custom_sso.py** for Authentik OIDC integration + - Mount to LiteLLM container volume + - Reference in config.yaml: `custom_ui_sso_sign_in_handler: custom_sso.custom_ui_sso_sign_in_handler` + +5. **Restart LiteLLM container** with new config + ```bash + docker compose restart litellm + ``` + +6. **Verify internal routing** — LiteLLM → Router pass-through works: + ```bash + curl -X POST http://127.0.0.1:4000/v1/chat/completions \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}' + ``` + +**Verification Checklist:** +- [ ] DNS resolution: `getent hosts auth.sysloggh.net` → 192.168.68.11 +- [ ] Postgres container healthy +- [ ] LiteLLM `/health` returns 200 +- [ ] LiteLLM → Router pass-through returns valid chat completion +- [ ] GPU health metrics unaffected (watch `/metrics`) + +### Phase 1: Shadow Mode (Week 1) — Zero Risk, Zero Downtime + +**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode. **Agents continue using :9000 directly.** ``` Agent → LiteLLM (:4000) → Router (:9000) → GPU (new, testing) (existing, unchanged) -Agent can also directly hit :9000 as fallback +Agent can also directly hit :9000 as fallback (unchanged) ``` **Tasks:** -1. **Deploy Postgres + LiteLLM on docker-vm** +1. **Create virtual keys for test agents** via LiteLLM UI or API: ```bash - cd /opt/litellm - # Apply litellm-fix.sh (already prepared) - docker compose up -d + curl -X POST http://127.0.0.1:4000/key/generate \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ + -H "Content-Type: application/json" \ + -d '{"models":["syslog-auto"],"metadata":{"agent":"test"}}' ``` - -2. **Create config.yaml** with router as upstream (see §4.3) - -3. **Create virtual keys for test agents** via LiteLLM UI - Mirror existing API_KEYS in LiteLLM's key store - Set per-key budgets (test with $100 cap) -4. **Verify pass-through works** +2. **Verify pass-through works** ```bash - curl -X POST http://docker-vm:4000/v1/chat/completions \ + curl -X POST http://127.0.0.1:4000/v1/chat/completions \ -H "Authorization: Bearer sk-litellm-test-key" \ -H "Content-Type: application/json" \ -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}' ``` -5. **Run 24-hour shadow**: Both :4000 and :9000 active, agents use :9000 +3. **Run 24-hour shadow**: Both :4000 and :9000 active, agents use :9000 - Monitor LiteLLM spend logs vs router metrics — confirm parity - Verify GPU health metrics unaffected + - Check guardrails not generating false positives -### Phase 2: Cutover (Week 2) — Gradual Migration +**Zero-Downtime Guarantee:** Router :9000 remains the primary agent endpoint. LiteLLM :4000 is tested in parallel with no impact on production traffic. -**Goal:** Move agents one-by-one to LiteLLM endpoint. +### Phase 2: Cutover (Week 2) — Gradual Agent Migration, Zero Cumulative Downtime + +**Goal:** Move agents one-by-one to LiteLLM endpoint. Each agent is migrated individually — other agents unaffected. **Tasks:** 1. **Migrate API keys to LiteLLM virtual keys:** - Create virtual key per agent in LiteLLM UI - Set model access: `syslog-auto` (default), plus individual GPU models - Set per-agent budget limits - - Create teams: "Core Agents" (Abiba, Mumuni, Tanko), "Dev Agents" (Kagenz0, Koby, Koonimo) + - Create teams: + - "Core Agents" (Abiba, Mumuni, Tanko) — enterprise tier + - "Dev Agents" (Kagenz0, Koby, Koonimo) — professional tier -2. **Update agent configs:** - - Change `OPENAI_API_BASE` from `http://docker-vm:9000/v1` → `http://docker-vm:4000/v1` +2. **Update agent configs one at a time:** + - Change `OPENAI_API_BASE` from `http://192.168.68.116:9000/v1` → `http://192.168.68.116:4000/v1` - Replace agent API keys with LiteLLM virtual keys - - Test each agent one at a time + - Test each agent individually — verify routing works + - **Per-agent downtime: <2 minutes** 3. **Migrate admin functions:** - Key creation/revocation → LiteLLM UI - Rate limit management → LiteLLM per-key RPM + router hardware RPM (dual enforcement) - Deprecated key tracking → LiteLLM UI key list -4. **Enable SSO** (optional, Phase 2+): - ```yaml - general_settings: - litellm_dashboard_sso: true - sso_provider: "google" # or github, microsoft, keycloak - sso_client_id: os.environ/SSO_CLIENT_ID - sso_client_secret: os.environ/SSO_CLIENT_SECRET - ``` +4. **Enable SSO via Authentik + custom_sso.py:** + - Authentik OAuth2 provider configured + - NGINX auth_request for /ui/* paths + - Custom UI SSO sign-in handler reads x-authentik-* headers + - Users authenticate with existing Authentik credentials 5. **Keep router :9000 accessible** as emergency fallback for 48 hours + - NGINX configured with router_fallback for 502 errors + - Agents can revert by changing `OPENAI_API_BASE` back to :9000 -### Phase 3: Production Hardening (Week 3+) — Optimize +**Zero-Downtime Guarantee:** Each agent has <2 minutes downtime during config update. Router :9000 stays online throughout. Fallback path available for instant rollback. -**Goal:** Lock down, optimize, monitor. +### Phase 3: Production Hardening (Week 3+) — Optimize & Scale + +**Goal:** Lock down, optimize, monitor. Prepare for client-facing services. **Tasks:** -1. **Remove deprecated router endpoints:** + +1. **Remove deprecated router endpoints** (only after all agents migrated): - Drop `/admin/keys/*` — fully migrated to LiteLLM UI - Drop Phase 0 dual-key logic (LiteLLM handles key rotation) - Simplify `API_KEYS` to single `ROUTER_API_KEY` 2. **Add LiteLLM observability:** - Prometheus metrics export - - Slack/email budget alerts + - Slack/email budget alerts via webhook → Hermes - Daily spend report webhook 3. **Enable LiteLLM caching** (Redis, shared with router): @@ -343,7 +474,7 @@ Agent can also directly hit :9000 as fallback cache_ttl: 3600 ``` -4. **Optional: External model fallbacks** +4. **Add external model fallbacks** for client-facing services: - Add Anthropic Claude as fallback for code-heavy requests - Add OpenAI GPT-4o as fallback for reasoning overflow - LiteLLM's native fallback chains handle this cleanly @@ -352,9 +483,18 @@ Agent can also directly hit :9000 as fallback - Keep: routing, slots, health checks, performance recording - Remove: key management, dual-key logic, admin endpoints +6. **Multi-tenancy setup** for client-facing inference services: + - Organization → Team → User hierarchy per client + - Per-client budget limits and guardrail policies + - Self-service onboarding via Authentik SSO → LiteLLM UI + +**Zero-Downtime Guarantee:** All changes are additive — new features added while existing routing continues uninterrupted. Router remains the GPU intelligence layer throughout. + --- -## 6. Nginx Configuration +--- + +## 6. Nginx Configuration (with Authentik OIDC Forward Auth) The existing nginx config routes `/admin/` → router :9000. This MUST change: @@ -364,44 +504,77 @@ The existing nginx config routes `/admin/` → router :9000. This MUST change: # proxy_pass http://127.0.0.1:9000/admin/; # } -# NEW +# === Authentik auth subrequest endpoint === +location /authentik/auth { + internal; + # Proxy to Authentik's outpost on acerpve (192.168.68.11) + # Uses internal /etc/hosts resolution: auth.sysloggh.net → 192.168.68.11 + proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx; + proxy_pass_request_body off; + proxy_set_header Content-Length ""; + proxy_set_header X-Original-URL $scheme://$http_host$request_uri; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +} + +# === LiteLLM Admin UI — Authentik-protected === location /ui/ { + auth_request /authentik/auth; + auth_request_set $auth_user $upstream_http_x_authentik_username; + auth_request_set $auth_email $upstream_http_x_authentik_email; + + proxy_set_header X-Authentik-Username $auth_user; + proxy_set_header X-Authentik-Email $auth_email; + proxy_pass http://127.0.0.1:4000/ui/; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } +# === LiteLLM SSO callback (for OIDC redirect flow) === +location /sso/callback { + proxy_pass http://127.0.0.1:4000/sso/callback; + proxy_set_header Host $host; +} + +# === API endpoint — Bearer token auth (no Authentik) === location /v1/ { # Primary: LiteLLM gateway proxy_pass http://127.0.0.1:4000/v1/; proxy_set_header Host $host; proxy_read_timeout 600s; # Fallback: direct router (if LiteLLM down) - # error_page 502 = @router_fallback; + error_page 502 = @router_fallback; } location @router_fallback { proxy_pass http://127.0.0.1:9000/v1/; + proxy_set_header Host $host; +} + +# === Key management API (needs master_key, not Authentik) === +location /key/ { + proxy_pass http://127.0.0.1:4000/key/; + proxy_set_header Host $host; + proxy_set_header Authorization $http_authorization; } # Keep router metrics accessible (not behind LiteLLM) location /router/ { proxy_pass http://127.0.0.1:9000/; - # Rewrite /router/stream → :9000/stream - # Rewrite /router/metrics → :9000/metrics } # Health check — combines both layers location /health { - # Check LiteLLM first, then router proxy_pass http://127.0.0.1:4000/health; } ``` --- -## 7. Docker Compose (`docker-compose.yml` on docker-vm) +## 7. Docker Compose (`docker-compose.yml` on CT 116) + +**Deployment host:** CT 116 `syslog-api` (192.168.68.116) on minipve. All services co-located. ```yaml services: @@ -409,15 +582,19 @@ services: litellm: image: ghcr.io/berriai/litellm:main-stable network_mode: "host" + extra_hosts: + - "auth.sysloggh.net:192.168.68.11" volumes: - ./config.yaml:/app/config.yaml:ro + - ./custom_sso.py:/app/custom_sso.py:ro environment: - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY} - - UI_USERNAME=admin - - UI_PASSWORD=${UI_PASSWORD} - DATABASE_URL=postgresql://litellm:${POSTGRES_PASSWORD}@localhost:5432/litellm - STORE_MODEL_IN_DB=True - ROUTER_API_KEY=${ROUTER_API_KEY} + - OPENAI_API_KEY=${OPENAI_API_KEY} # Optional: external fallback + - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} # Optional: external fallback + - PROXY_BASE_URL=https://litellm.sysloggh.net command: - --config - /app/config.yaml @@ -445,6 +622,17 @@ services: retries: 5 restart: unless-stopped + # Nginx also needs auth resolution + nginx: + image: nginx:alpine + extra_hosts: + - "auth.sysloggh.net:192.168.68.11" + volumes: + - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro + ports: + - "80:80" + restart: unless-stopped + # Layer 2: Custom Router (Intelligence & Hardware) # Already deployed separately — not in this compose file # The router is managed by the existing harness deployment on CT 116 @@ -486,43 +674,151 @@ volumes: --- -## 10. Migration Commands (Quick Reference) +## 10. Migration Commands (Quick Reference) — CT 116 + +**Target:** CT 116 `syslog-api` (192.168.68.116), minipve. All services co-located. ```bash -# On docker-vm (CT 116): +# On CT 116 (SSH via minipve: pct exec 116 bash): -# 1. Deploy LiteLLM stack +# === Phase 0: Infrastructure Prep === + +# 0. DNS split-horizon fix +echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts +getent hosts auth.sysloggh.net # Verify → 192.168.68.11 + +# 1. Navigate to harness deployment directory cd /opt/litellm -docker compose down -v # Clean slate -docker compose up -d # Postgres + LiteLLM -# 2. Verify -curl http://localhost:4000/health -curl http://localhost:4000/ui # Admin dashboard +# 2. Deploy Postgres (LiteLLM DB) +docker compose up -d postgres -# 3. Create first virtual key via UI or CLI -docker compose exec litellm litellm-proxy keys create \ - --key-alias "abiba-test" \ - --models "syslog-auto" \ - --max-budget 10.0 \ - --team-id "core-agents" +# 3. Replace LiteLLM config with production config.yaml (see §4.3) +# - All models → http://router:9000/v1 +# - Guardrails enabled +# - custom_sso.py mounted -# 4. Test end-to-end -curl -X POST http://localhost:4000/v1/chat/completions \ +# 4. Restart LiteLLM with new config +docker compose restart litellm + +# 5. Verify core health +curl http://127.0.0.1:4000/health # LiteLLM alive +curl http://127.0.0.1:9000/health # Router alive +curl http://127.0.0.1:4000/health/liveliness # LiteLLM readiness + +# 6. Test LiteLLM → Router pass-through +curl -X POST http://127.0.0.1:4000/v1/chat/completions \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model":"syslog-auto","messages":[{"role":"user","content":"ping"}]}' +# Expected: valid chat completion with GPU routing metadata + +# === Phase 1: Shadow Mode === + +# 7. Create virtual key for testing +curl -X POST http://127.0.0.1:4000/key/generate \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ + -H "Content-Type: application/json" \ + -d '{"models":["syslog-auto","qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"],"metadata":{"agent":"test"},"max_budget":100}' + +# 8. Test with virtual key +curl -X POST http://127.0.0.1:4000/v1/chat/completions \ -H "Authorization: Bearer " \ - -d '{"model":"syslog-auto","messages":[{"role":"user","content":"Hello"}]}' + -H "Content-Type: application/json" \ + -d '{"model":"syslog-auto","messages":[{"role":"user","content":"Hello from LiteLLM!"}]}' -# 5. Update nginx (see §6) +# 9. Verify NGINX → Authentik routing (after Phase 2) +curl -I http://127.0.0.1/ui/ # Should return 302 → Authentik login +curl -I http://127.0.0.1/v1/chat/completions # Should proxy to LiteLLM :4000 + +# 10. Monitor both layers +curl http://127.0.0.1:4000/global/spend/logs # LiteLLM spend +curl http://127.0.0.1:9000/metrics # Router GPU metrics +curl http://127.0.0.1:9000/stream # Router SSE dashboard + +# 11. NGINX reload after config changes nginx -t && nginx -s reload -# 6. Monitor both layers -curl http://localhost:4000/global/spend/logs # LiteLLM spend -curl http://localhost:9000/metrics # Router GPU metrics -curl http://localhost:9000/stream # Router SSE dashboard +# 12. View logs +docker compose logs -f litellm # LiteLLM gateway logs +docker compose logs -f nginx # Reverse proxy logs ``` --- +## 11. GitOps Workflow + +All harness changes tracked in `SyslogSolution/syslog-harness` on Gitea (http://192.168.68.17:3000). Multiple agents (Abiba, Mumuni, Kagenz0, Agent Zero) collaborate on this repo. + +**Branching Strategy:** +- `main` — Production-ready, deployed on CT 116 +- `dev/` — Feature branches for each migration phase +- PR required for main merges (allows cross-agent review) + +**Conventional Commits:** +``` +feat(router): add X-Usage-Tokens header for LiteLLM cost tracking +fix(docker): add extra_hosts for Authentik internal DNS +chore(plan): update LITELLM-MIGRATION-PLAN.md with Phase 0-3 zero-downtime +refactor(router): remove deprecated /admin/keys/* endpoints (Phase 3) +infra(dns): configure /etc/hosts split-horizon for CT 116 +``` + +**Commit Signatures (for audit trail):** +- Agent commits include source: `Co-authored-by: Abiba ` +- Human commits: `Signed-off-by: Jerome Tabiri ` + +**Deployment Flow:** +1. PR opened on Gitea (feature → main) +2. Review by at least one other agent or Jerome +3. Merge to main +4. `git pull` on CT 116 (`/opt/litellm/`) +5. `docker compose up -d` (rolling update) + +**Post-Deployment Verification:** +- Check LiteLLM `/health` — 200 OK +- Check Router `/metrics` — GPU health unchanged +- Check Dashboard SSE — streaming live +- Test agent inference — valid chat completion + +--- + +## 12. Zero-Downtime Migration Strategy + +**Principle:** Agents must experience **zero cumulative downtime** throughout the migration. Router `:9000` is the safety net — never taken offline until all agents verified on LiteLLM. + +**Per-Agent Migration (2 min downtime each):** +1. Create LiteLLM virtual key for agent (no impact on running agent) +2. Update agent config: `OPENAI_API_BASE` → `:4000`, `OPENAI_API_KEY` → virtual key +3. Restart agent (2 min) +4. Verify inference works via LiteLLM → Router → GPU +5. If issue: revert `OPENAI_API_BASE` → `:9000`, restart (30 sec rollback) + +**Dual-Path Architecture (throughout migration):** +``` +Path A: Agent → LiteLLM :4000 → Router :9000 → GPU (new, for migrated agents) +Path B: Agent → Router :9000 → GPU (legacy, for unmigrated agents) +``` +Both paths work simultaneously. Router unchanged in both paths. + +**Rollback Procedure (per agent, 30 seconds):** +1. Set `OPENAI_API_BASE=http://192.168.68.116:9000/v1` +2. Set `OPENAI_API_KEY=` +3. Restart agent + +**Emergency Global Rollback (if LiteLLM catastrophic failure):** +1. NGINX auto-fallback: `error_page 502 = @router_fallback` → traffic routes to :9000 +2. Manual: Update all agent configs back to :9000 (scripted) +3. Restart all agents (5 min) + +**Health Checks (continuously monitored):** +- `/health` — LiteLLM + Router combined status +- `/router/metrics` — GPU health, circuit breaker, active slots +- `/stream` — SSE dashboard (real-time) +- Redis: `circuit:*` keys for tripped breakers + +--- + ## Appendix A: Router Slim-Down (Phase 3) After full migration, `router-fixed.py` can be simplified by removing: @@ -574,4 +870,5 @@ litellm_settings: --- *Plan drafted: 2026-06-13 by Abiba 🦊⚡* -*Status: Ready for Kwame review* +*Updated: 2026-06-14 by Agent Zero (kagentz-bot) — CT 116 deployment target, DNS split-horizon, Authentik OIDC, Zero-Downtime strategy, GitOps workflow* +*Status: Ready for Phase 0 deployment*