docs: LiteLLM migration plan — two-layer architecture with model identity gap analysis

Architecture review identifying metric accuracy issue where router silently reroutes explicit model requests. Proposes Option A: strict passthrough for explicit models with LiteLLM-native fallback chains. Keeps syslog-auto for content-based routing. Awaiting Mumuni and Kagenz0 review.
fix(dashboard): latest visual fixes (navbar, layout, status labels)
2026-06-14 00:40:07 +00:00 · 2026-06-12 22:13:33 -04:00
2 changed files with 1022 additions and 0 deletions
@@ -0,0 +1,577 @@
+# LiteLLM Integration Migration Plan
+## Syslog Solution LLC — June 13, 2026
+
+---
+
+## Executive Summary
+
+**Goal:** Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness — without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.
+
+**Architecture Decision:** Two-layer architecture.
+
+```
+                  ┌─────────────────────────────────┐
+                  │     LiteLLM Gateway (Layer 1)    │
+                  │     Port 4000 — Policy & UX      │
+                  │  ┌─────────────────────────────┐ │
+                  │  │ Admin UI (/ui)              │ │
+                  │  │ Virtual Keys & Permissions  │ │
+                  │  │ Teams, Users, SSO (OIDC)    │ │
+                  │  │ Spend Tracking & Budgets    │ │
+                  │  │ Usage Analytics Dashboard   │ │
+                  │  │ Request Audit Trail         │ │
+                  │  │ Global Rate Limiting        │ │
+                  │  └─────────────────────────────┘ │
+                  │           │                       │
+                  │    Pass-through to router         │
+                  └──────────┬──────────────────────┘
+                             │
+                  ┌──────────▼──────────────────────┐
+                  │  Custom Router (Layer 2)          │
+                  │  Port 9000 — Intelligence & HW   │
+                  │  ┌─────────────────────────────┐ │
+                  │  │ 5-Tier Content-Based Routing│ │
+                  │  │ GPU Slot Management (Redis) │ │
+                  │  │ Agent Spread Prevention     │ │
+                  │  │ GPU Health Scoring (40/30/30)│ │
+                  │  │ Sidecar VRAM/Temp/Power     │ │
+                  │  │ Circuit Breaker             │ │
+                  │  │ Context Window Tracking      │ │
+                  │  │ Per-Request Perf Recording  │ │
+                  │  │ Hardware Rate Limiting      │ │
+                  │  └─────────────────────────────┘ │
+                  └──────────┬──────────────────────┘
+                             │
+          ┌──────────────────┼──────────────────────┐
+          │                  │                       │
+  ┌───────▼──────┐  ┌────────▼───────┐  ┌───────────▼──────┐
+  │ qwen3.6-35B  │  │ qwen3.6-27B   │  │ gemma-4-12b      │
+  │ MoE/Strix    │  │ Dense/RTX3090 │  │ VLM/RTX 5070     │
+  │ :8080 (llama)│  │ :8080 (llama) │  │ :8080 (llama)    │
+  │ :8090 (side) │  │ :8090 (side)  │  │ :8090 (sidecar)  │
+  └──────────────┘  └───────────────┘  └──────────────────┘
+```
+
+---
+
+## 1. Current State Baseline
+
+### 1.1 Router (`router-fixed.py` — port 9000, deployed on CT 116 / docker-vm)
+
+| Feature | Implementation |
+|---------|---------------|
+| **Routing Engine** | 5-tier content-based: lightweight → simple_conv → medium → heavy_reasoning → default |
+| **GPU Slot Mgmt** | Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset |
+| **Health Checks** | Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health |
+| **Agent Spreading** | `select_best_gpu()` prefers GPUs with 0 other agents, then non-self GPUs |
+| **Rate Limiting** | Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20 |
+| **Auth** | Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation |
+| **Performance** | Per-request latency/tokens/tps → Redis lists (perf:recent, perf:model:X, perf:agent:X) |
+| **Context Tracking** | Session-level token accumulation with compaction warnings in headers |
+| **SSE Streaming** | Real-time dashboard updates, per-model timeseries |
+| **Admin** | `/admin/keys`, `/admin/keys/generate`, `/admin/keys/revoke`, `/admin/keys/deprecation-summary` |
+
+### 1.2 GPU Backends
+
+| GPU | Host | llama.cpp | Sidecar | VRAM | Context |
+|-----|------|-----------|---------|------|---------|
+| qwen3.6-35B-A3B (MoE) | 192.168.68.15 | :8080 | :8090 | Strix Halo | 262K |
+| qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K |
+| gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K |
+
+### 1.3 Existing LiteLLM Attempt
+- `/root/litellm-fix.sh` — previous setup script for docker-vm
+- Configured with Postgres, host networking, master key
+- **Never productionized** — still in exploratory phase
+
+---
+
+## 2. What LiteLLM Brings (That We Don't Have)
+
+| Feature | Our Router | LiteLLM | Value Add |
+|---------|-----------|---------|-----------|
+| **Admin UI** | ❌ | ✅ Full dashboard at /ui | Non-technical users can manage keys, view spend |
+| **Virtual Key Permissions** | ❌ (binary key→tier) | ✅ Granular: per-model, per-team, budget caps | Fine-grained access control |
+| **Spend Tracking** | ❌ | ✅ Per-request $ cost with model-specific pricing | Billing, cost allocation, client invoicing |
+| **Teams & Orgs** | ❌ | ✅ Multi-tenant: org→team→user hierarchy | Segregate clients/projects |
+| **SSO/OIDC** | ❌ | ✅ Google, GitHub, Microsoft, Okta, Keycloak | Enterprise auth integration |
+| **Budget Alerts** | ❌ | ✅ Per-key, per-user, per-team budget with webhooks | Prevent overspend |
+| **Usage Analytics** | ⚠️ (custom /metrics) | ✅ Built-in: daily trends, model breakdown, per-customer | Better visualization |
+| **100+ Provider Support** | ❌ (3 local GPUs) | ✅ OpenAI, Anthropic, Bedrock, Vertex, etc. | Future cloud model access |
+| **Fallback Chains** | ❌ | ✅ Multi-provider: OpenAI→Azure→Together | External model resilience |
+| **RPM/TPM Weighted LB** | ❌ | ✅ Weighted load balancing across deployments | Fine-grained traffic shaping |
+
+---
+
+## 3. What We Keep (That LiteLLM Doesn't Have)
+
+| Feature | Why We Must Keep It |
+|---------|---------------------|
+| **Content-based 5-tier routing** | LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints |
+| **GPU hardware health scoring** | LiteLLM doesn't monitor VRAM, temp, power — our 40/30/30 scoring prevents routing to overheating GPUs |
+| **GPU slot management** | LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading |
+| **Agent spread prevention** | Our `select_best_gpu()` spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle |
+| **Cross-turn context tracking** | Session-level token accumulation with compaction warnings via X-Context-Warning headers |
+| **GPU sidecar metrics** | VRAM %, GPU utilization %, power draw, temperature — exposed via /metrics and SSE dashboard |
+| **Circuit breaker** | 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular |
+
+---
+
+## 4. Migration Architecture
+
+### 4.1 Principle: "LiteLLM is the lobby, our router is the engine room"
+
+- **LiteLLM** handles everything a **user/admin** touches: keys, teams, budgets, spend logs, SSO, the UI
+- **Custom Router** handles everything the **GPUs** need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking
+
+### 4.2 Flow
+
+```
+Agent Request
+     │
+     ▼
+┌─────────────────────────────────────────────┐
+│ LiteLLM Gateway (:4000)                      │
+│                                              │
+│ 1. Authenticate virtual key (sk-litellm-...) │
+│ 2. Check key permissions (model access)      │
+│ 3. Check budget (per-key, per-user, per-team)│
+│ 4. Check team rate limits                    │
+│ 5. Log request metadata                      │
+│ 6. Forward to custom router as OpenAI-compat  │
+│    POST http://router:9000/v1/chat/completions│
+│    Headers: Authorization: Bearer <agent-key> │
+│             X-LiteLLM-User: <user-id>         │
+│             X-LiteLLM-Team: <team-id>         │
+│             X-Session-Id: <session>           │
+│                                              │
+│ 7. On response: log spend, update budgets     │
+│ 8. Return response to agent                   │
+└──────────────┬──────────────────────────────┘
+               │
+               ▼
+┌──────────────────────────────────────────────┐
+│ Custom Router (:9000)                         │
+│                                               │
+│ 1. Authenticate agent key (sk-syslog-...)     │
+│ 2. Hardware rate limit (per-tier RPM)         │
+│ 3. Content-based tier routing                 │
+│    - Estimate tokens, detect system msg       │
+│    - Count turns, check routing_hints         │
+│ 4. GPU slot availability (Redis counter)      │
+│ 5. GPU health check (sidecar)                 │
+│ 6. Agent spread logic (select_best_gpu)       │
+│ 7. Queue if saturated (with timeout)          │
+│ 8. Forward to selected llama.cpp GPU          │
+│ 9. Track context window, set compaction header│
+│ 10. Record performance metrics                │
+│ 11. Return response (with routing metadata)   │
+└──────────────┬───────────────────────────────┘
+               │
+               ▼
+┌──────────────────────────────────────────────┐
+│ llama.cpp GPU (:8080)                         │
+└──────────────────────────────────────────────┘
+```
+
+### 4.3 LiteLLM Config (`config.yaml`)
+
+```yaml
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  database_url: postgresql://litellm:${POSTGRES_PASSWORD}@postgres:5432/litellm
+  store_model_in_db: true
+
+model_list:
+  # All three GPUs exposed as a single virtual "syslog-router" model
+  # LiteLLM passes through to our router, which handles actual GPU selection
+  - model_name: syslog-auto           # Default auto-routing
+    litellm_params:
+      model: openai/syslog-auto       # Using OpenAI-compatible format
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+      rpm: 600                         # Cap total RPM across all GPUs
+  
+  # Individual GPU pass-through (for explicit model requests)
+  - model_name: qwen3.6-35B-A3B
+    litellm_params:
+      model: openai/qwen3.6-35B-A3B
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+  
+  - model_name: qwen3.6-27B-code
+    litellm_params:
+      model: openai/qwen3.6-27B-code
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+  - model_name: gemma-4-12b
+    litellm_params:
+      model: openai/gemma-4-12b
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+litellm_settings:
+  num_retries: 0                     # Disabled — our router handles retry
+  request_timeout: 600               # Match our 10-min llama-server timeout
+  set_verbose: true
+  failure_callback: ["prometheus"]   # Optional: export to Prometheus
+
+router_settings:
+  routing_strategy: "usage-based-routing"  # For external models only
+  # Note: All local GPU routing is handled by custom router
+  enable_loadbalancing_on_proxy: false     # Disable LiteLLM's internal LB
+  allowed_fails: 100                       # Don't cooldown — our circuit breaker handles
+
+# Cost tracking: map model names to per-token pricing
+# These are passed through from our router's X-Usage-Tokens header
+```
+
+### 4.4 Router Modifications (Light Touch)
+
+Minimal changes to `router-fixed.py` — the router remains largely unchanged:
+
+1. **New header passthrough**: Forward `X-LiteLLM-*` headers to GPU (transparent — already works)
+2. **New endpoint for health passthrough**: `GET /v1/models` already works
+3. **Disable own key management**: Remove `/admin/keys/*` endpoints (migrate to LiteLLM UI)
+4. **Keep ALL routing logic**: No changes to `route()`, `select_best_gpu()`, `check_gpu_health()`, slot management, etc.
+5. **Add LiteLLM-compatible response**: Return `X-Usage-Tokens` header so LiteLLM can track token costs
+
+```python
+# ADD to router-fixed.py chat() response:
+resp.headers["X-Usage-Tokens"] = json.dumps({
+    "prompt_tokens": prompt_tokens,
+    "completion_tokens": completion_tokens,
+    "model": model
+})
+```
+
+---
+
+## 5. Deployment Plan (3 Phases)
+
+### Phase 1: Shadow Mode (Week 1) — Zero Risk
+
+**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode.
+
+```
+Agent → LiteLLM (:4000) → Router (:9000) → GPU
+         (new, testing)    (existing, unchanged)
+         
+Agent can also directly hit :9000 as fallback
+```
+
+**Tasks:**
+1. **Deploy Postgres + LiteLLM on docker-vm**
+   ```bash
+   cd /opt/litellm
+   # Apply litellm-fix.sh (already prepared)
+   docker compose up -d
+   ```
+
+2. **Create config.yaml** with router as upstream (see §4.3)
+
+3. **Create virtual keys for test agents** via LiteLLM UI
+   - Mirror existing API_KEYS in LiteLLM's key store
+   - Set per-key budgets (test with $100 cap)
+
+4. **Verify pass-through works**
+   ```bash
+   curl -X POST http://docker-vm:4000/v1/chat/completions \
+     -H "Authorization: Bearer sk-litellm-test-key" \
+     -H "Content-Type: application/json" \
+     -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'
+   ```
+
+5. **Run 24-hour shadow**: Both :4000 and :9000 active, agents use :9000
+   - Monitor LiteLLM spend logs vs router metrics — confirm parity
+   - Verify GPU health metrics unaffected
+
+### Phase 2: Cutover (Week 2) — Gradual Migration
+
+**Goal:** Move agents one-by-one to LiteLLM endpoint.
+
+**Tasks:**
+1. **Migrate API keys to LiteLLM virtual keys:**
+   - Create virtual key per agent in LiteLLM UI
+   - Set model access: `syslog-auto` (default), plus individual GPU models
+   - Set per-agent budget limits
+   - Create teams: "Core Agents" (Abiba, Mumuni, Tanko), "Dev Agents" (Kagenz0, Koby, Koonimo)
+
+2. **Update agent configs:**
+   - Change `OPENAI_API_BASE` from `http://docker-vm:9000/v1` → `http://docker-vm:4000/v1`
+   - Replace agent API keys with LiteLLM virtual keys
+   - Test each agent one at a time
+
+3. **Migrate admin functions:**
+   - Key creation/revocation → LiteLLM UI
+   - Rate limit management → LiteLLM per-key RPM + router hardware RPM (dual enforcement)
+   - Deprecated key tracking → LiteLLM UI key list
+
+4. **Enable SSO** (optional, Phase 2+):
+   ```yaml
+   general_settings:
+     litellm_dashboard_sso: true
+     sso_provider: "google"  # or github, microsoft, keycloak
+     sso_client_id: os.environ/SSO_CLIENT_ID
+     sso_client_secret: os.environ/SSO_CLIENT_SECRET
+   ```
+
+5. **Keep router :9000 accessible** as emergency fallback for 48 hours
+
+### Phase 3: Production Hardening (Week 3+) — Optimize
+
+**Goal:** Lock down, optimize, monitor.
+
+**Tasks:**
+1. **Remove deprecated router endpoints:**
+   - Drop `/admin/keys/*` — fully migrated to LiteLLM UI
+   - Drop Phase 0 dual-key logic (LiteLLM handles key rotation)
+   - Simplify `API_KEYS` to single `ROUTER_API_KEY`
+
+2. **Add LiteLLM observability:**
+   - Prometheus metrics export
+   - Slack/email budget alerts
+   - Daily spend report webhook
+
+3. **Enable LiteLLM caching** (Redis, shared with router):
+   ```yaml
+   router_settings:
+     redis_host: os.environ/REDIS_HOST
+     redis_port: 6379
+     cache: true
+     cache_ttl: 3600
+   ```
+
+4. **Optional: External model fallbacks**
+   - Add Anthropic Claude as fallback for code-heavy requests
+   - Add OpenAI GPT-4o as fallback for reasoning overflow
+   - LiteLLM's native fallback chains handle this cleanly
+
+5. **Router slim-down:** Extract GPU health metrics to dedicated /health only
+   - Keep: routing, slots, health checks, performance recording
+   - Remove: key management, dual-key logic, admin endpoints
+
+---
+
+## 6. Nginx Configuration
+
+The existing nginx config routes `/admin/` → router :9000. This MUST change:
+
+```nginx
+# OLD (remove)
+# location /admin/ {
+#     proxy_pass http://127.0.0.1:9000/admin/;
+# }
+
+# NEW
+location /ui/ {
+    proxy_pass http://127.0.0.1:4000/ui/;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+}
+
+location /v1/ {
+    # Primary: LiteLLM gateway
+    proxy_pass http://127.0.0.1:4000/v1/;
+    proxy_set_header Host $host;
+    proxy_read_timeout 600s;
+    # Fallback: direct router (if LiteLLM down)
+    # error_page 502 = @router_fallback;
+}
+
+location @router_fallback {
+    proxy_pass http://127.0.0.1:9000/v1/;
+}
+
+# Keep router metrics accessible (not behind LiteLLM)
+location /router/ {
+    proxy_pass http://127.0.0.1:9000/;
+    # Rewrite /router/stream → :9000/stream
+    # Rewrite /router/metrics → :9000/metrics
+}
+
+# Health check — combines both layers
+location /health {
+    # Check LiteLLM first, then router
+    proxy_pass http://127.0.0.1:4000/health;
+}
+```
+
+---
+
+## 7. Docker Compose (`docker-compose.yml` on docker-vm)
+
+```yaml
+services:
+  # Layer 1: LiteLLM Gateway (Policy & Admin)
+  litellm:
+    image: ghcr.io/berriai/litellm:main-stable
+    network_mode: "host"
+    volumes:
+      - ./config.yaml:/app/config.yaml:ro
+    environment:
+      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
+      - UI_USERNAME=admin
+      - UI_PASSWORD=${UI_PASSWORD}
+      - DATABASE_URL=postgresql://litellm:${POSTGRES_PASSWORD}@localhost:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - ROUTER_API_KEY=${ROUTER_API_KEY}
+    command:
+      - --config
+      - /app/config.yaml
+      - --port
+      - "4000"
+    depends_on:
+      postgres:
+        condition: service_healthy
+    restart: unless-stopped
+
+  # Database for LiteLLM
+  postgres:
+    image: postgres:16-alpine
+    network_mode: "host"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+    restart: unless-stopped
+
+  # Layer 2: Custom Router (Intelligence & Hardware)
+  # Already deployed separately — not in this compose file
+  # The router is managed by the existing harness deployment on CT 116
+
+volumes:
+  pgdata:
+```
+
+---
+
+## 8. Risk Mitigation
+
+| Risk | Mitigation |
+|------|------------|
+| LiteLLM adds latency overhead | Shadow mode measures: <50ms extra is acceptable for admin features. LiteLLM is a thin proxy. |
+| LiteLLM down = all agents down | Nginx fallback to router :9000 direct (see §6). Agents can also be configured with dual endpoints. |
+| Key sync drift (LiteLLM keys ≠ router keys) | Single-source: LiteLLM is key authority. Router uses one `ROUTER_API_KEY` from LiteLLM's perspective. Agent keys live in LiteLLM only. |
+| Spend tracking inaccurate for local GPUs | Configure `model_cost` per GPU with $0 rate (self-hosted). Optionally track "internal cost" via custom pricing. |
+| Double rate limiting (LiteLLM + Router) | Keep both intentionally: LiteLLM for per-user soft caps, Router for hardware protection. Non-overlapping concerns. |
+| PostgreSQL failure | LiteLLM can run with SQLite fallback, but UI features degrade. Postgres is the recommended path. |
+| Router custom logic becomes a black box to LiteLLM | Acceptable trade-off. LiteLLM sees router as opaque OpenAI endpoint. GPU-level routing decisions are router's domain. |
+
+---
+
+## 9. Success Metrics
+
+| Metric | Before | After |
+|--------|--------|-------|
+| Key management | Manual CLI + env vars + redeploy | UI-based, instant, no redeploy |
+| Spend visibility | None | Per-agent, per-team, per-model $ tracking |
+| Access control | Tier-based (3 levels) | Per-key, per-model, budget-capped |
+| New agent onboarding | Generate key, update env var, redeploy router | Create in UI, share key |
+| Admin UX | curl + JSON responses | Visual dashboard, graphs, search |
+| Audit trail | Router logs (stdout only) | Database-backed with UI search |
+| SSO | None | Google/GitHub/Microsoft OIDC |
+| Budget enforcement | None | Automatic: key suspended at $limit |
+| GPU routing intelligence | Full (unchanged) | Full (unchanged) |
+| GPU health monitoring | Full (unchanged) | Full (unchanged) |
+
+---
+
+## 10. Migration Commands (Quick Reference)
+
+```bash
+# On docker-vm (CT 116):
+
+# 1. Deploy LiteLLM stack
+cd /opt/litellm
+docker compose down -v   # Clean slate
+docker compose up -d     # Postgres + LiteLLM
+
+# 2. Verify
+curl http://localhost:4000/health
+curl http://localhost:4000/ui   # Admin dashboard
+
+# 3. Create first virtual key via UI or CLI
+docker compose exec litellm litellm-proxy keys create \
+  --key-alias "abiba-test" \
+  --models "syslog-auto" \
+  --max-budget 10.0 \
+  --team-id "core-agents"
+
+# 4. Test end-to-end
+curl -X POST http://localhost:4000/v1/chat/completions \
+  -H "Authorization: Bearer <virtual-key>" \
+  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"Hello"}]}'
+
+# 5. Update nginx (see §6)
+nginx -t && nginx -s reload
+
+# 6. Monitor both layers
+curl http://localhost:4000/global/spend/logs   # LiteLLM spend
+curl http://localhost:9000/metrics             # Router GPU metrics
+curl http://localhost:9000/stream              # Router SSE dashboard
+```
+
+---
+
+## Appendix A: Router Slim-Down (Phase 3)
+
+After full migration, `router-fixed.py` can be simplified by removing:
+
+```python
+# REMOVE (migrated to LiteLLM):
+- API_KEYS validation logic (keep single ROUTER_API_KEY)
+- Dual-key deprecation tracking
+- /admin/keys, /admin/keys/generate, /admin/keys/revoke
+- /admin/keys/deprecation-summary
+- Phase 0 deprecated key logging
+- check_rate_limit() (optional — keep as hardware safety net)
+
+# KEEP:
+- route() — all 5 tiers
+- select_best_gpu()
+- check_gpu_health()
+- is_gpu_busy(), gpu_active_count(), gpu_incr/decr()
+- estimate_tokens()
+- store_perf_record()
+- GPU_SIDECARS, GPU_URLS, GPU_MAX_CONCURRENT, GPU_CONTEXT
+- counter_audit_loop()
+- /v1/chat/completions — core routing endpoint
+- /v1/models
+- /health
+- /metrics, /metrics/performance, /metrics/scatter, /metrics/timeseries
+- /stream — SSE dashboard
+```
+
+## Appendix B: LiteLLM Cost Config for Local GPUs
+
+```yaml
+# In config.yaml — map models to per-token pricing for spend tracking
+litellm_settings:
+  model_cost:
+    qwen3.6-35B-A3B:
+      input_cost_per_token: 0.0        # Self-hosted, no external cost
+      output_cost_per_token: 0.0
+    qwen3.6-27B-code:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    gemma-4-12b:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+  # For internal cost allocation, set symbolic rates:
+  # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens
+```
+
+---
+
+*Plan drafted: 2026-06-13 by Abiba 🦊⚡*
+*Status: Ready for Kwame review*
@@ -0,0 +1,445 @@
+<!DOCTYPE html>
+<html lang="en" x-data="dashboard()" x-init="init()">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Inference Harness - Dashboard</title>
+    
+    <!-- Tailwind CSS -->
+    <script src="https://cdn.tailwindcss.com"></script>
+    
+    <!-- Alpine.js -->
+    <script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.14.8/dist/cdn.min.js"></script>
+    
+    <!-- Chart.js -->
+    <script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
+    
+    <!-- Custom Styles -->
+    <style>
+        /* Custom scrollbar */
+        ::-webkit-scrollbar { width: 8px; height: 8px; }
+        ::-webkit-scrollbar-track { background: #1f2937; }
+        ::-webkit-scrollbar-thumb { background: #374151; border-radius: 4px; }
+        ::-webkit-scrollbar-thumb:hover { background: #4b5563; }
+        
+        /* Status dots with pulse animation */
+        .dot-green { background: #10b981; animation: pulse-green 2s infinite; }
+        .dot-yellow { background: #f59e0b; animation: pulse-yellow 2s infinite; }
+        .dot-red { background: #ef4444; animation: pulse-red 2s infinite; }
+        
+        @keyframes pulse-green {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(16, 185, 129, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(16, 185, 129, 0); }
+        }
+        @keyframes pulse-yellow {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(245, 158, 11, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(245, 158, 11, 0); }
+        }
+        @keyframes pulse-red {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(239, 68, 68, 0); }
+        }
+        
+        /* Glassmorphism panels */
+        .glass-panel {
+            background: rgba(31, 41, 55, 0.7);
+            backdrop-filter: blur(12px);
+            border: 1px solid rgba(75, 85, 99, 0.4);
+        }
+        
+        /* Smooth transitions */
+        .transition-all-300 { transition: all 0.3s ease; }
+        
+        /* Status badges */
+        .status-badge {
+            display: inline-flex;
+            align-items: center;
+            padding: 0.125rem 0.5rem;
+            border-radius: 0.375rem;
+            font-size: 0.75rem;
+            font-weight: 600;
+        }
+        
+        /* Health bar gradient */
+        .health-bar {
+            height: 0.5rem;
+            background-color: #374151;
+            border-radius: 0.375rem;
+            overflow: hidden;
+        }
+        
+        .health-fill {
+            height: 100%;
+            transition: width 0.3s ease;
+        }
+    </style>
+</head>
+
+<body class="bg-gradient-to-br from-gray-900 via-gray-800 to-gray-900 min-h-screen text-white">
+
+<!-- Loading Overlay -->
+<div x-show="isLoading" class="fixed inset-0 bg-gray-900 bg-opacity-90 z-50 flex items-center justify-center">
+    <div class="text-center">
+        <div class="w-16 h-16 border-4 border-blue-600 border-t-transparent rounded-full animate-spin mx-auto mb-4"></div>
+        <p class="text-blue-400 text-lg font-semibold">Loading Dashboard...</p>
+    </div>
+</div>
+
+<!-- Main Container -->
+<div class="container mx-auto px-4 py-6 max-w-[1920px]">
+
+    <!-- Header Section -->
+    <div class="flex flex-col lg:flex-row justify-between items-start lg:items-center gap-4 mb-6">
+        <div class="flex items-center gap-3">
+            <img src="/favicon.svg" class="w-10 h-10" alt="Logo">
+            <div>
+                <h1 class="text-2xl font-bold text-white">Inference Harness</h1>
+                <p class="text-sm text-gray-400">Syslog Solution LLC  Real-time Monitoring</p>
+            </div>
+        </div>
+        <div class="flex items-center gap-6">
+            <div class="flex items-center gap-2">
+                <div x-text="globalStatus" x-class="{
+                    'dot-green': globalStatus === 'healthy',
+                    'dot-yellow': globalStatus === 'degraded',
+                    'dot-red': globalStatus === 'critical'
+                }" class="w-4 h-4 rounded-full"></div>
+                <span x-text="globalStatus" x-bind:class="{
+                    'text-emerald-400': globalStatus === 'healthy',
+                    'text-amber-400': globalStatus === 'degraded',
+                    'text-red-400': globalStatus === 'critical'
+                }" class="font-semibold text-lg"></span>
+            </div>
+            <button @click="refreshAll()" class="px-4 py-2 bg-blue-600 hover:bg-blue-700 text-white rounded-lg text-sm flex items-center gap-2 transition-all-300">
+                <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+                    <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 4v5h.582m15.356 2A8.001 8.001 0 004.582 9m0 0H9m11 11v-5h-.581m0 0a8.003 8.003 0 01-15.357-2m15.357 2H15"></path>
+                </svg>
+                Refresh
+            </button>
+            <span x-text="lastUpdate" class="text-sm text-gray-500"></span>
+        </div>
+    </div>
+
+    <!-- KPI Cards Row -->
+    <div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-5 gap-4 mb-6">
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-blue-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.gpu_count_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.gpu_count || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">GPUs Online</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-purple-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.sessions_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.active_sessions || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Active Sessions</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-red-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.trips_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.circuit_trips || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Circuit Breakers</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-cyan-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.latency_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="(kpi.avg_latency || 0).toFixed(1) + 'ms'"></p>
+            <p class="text-sm text-gray-400 mt-1">Avg Latency</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-green-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.requests_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.requests_minute || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Requests/min</p>
+        </div>
+    </div>
+
+    <!-- GPU Health Scoring (Phase 3) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                GPU Health Scoring
+            </h2>
+            <div class="text-sm text-gray-400">
+                Scoring: VRAM (40%)  Temp (30%)  Load (30%)
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
+            <!-- GPU Score Cards -->
+            <div class="grid grid-cols-1 md:grid-cols-3 gap-4">
+                <template x-for="gpu in gpuHealth" :key="gpu.id">
+                    <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg"
+                         x-bind:class="{
+                             'border-emerald-500/50': gpu.health_score < 30,
+                             'border-amber-500/50': gpu.health_score >= 30 && gpu.health_score < 50,
+                             'border-red-500/50': gpu.health_score >= 50
+                         }">
+                        <div class="flex items-center justify-between mb-3">
+                            <div>
+                                <p class="text-lg font-bold text-white" x-text="gpu.name"></p>
+                                <p class="text-xs text-gray-400" x-text="gpu.model"></p>
+                            </div>
+                            <div x-show="gpu.is_preferred" class="px-2 py-1 bg-emerald-600 rounded-lg text-xs font-semibold">
+                                Preferred
+                            </div>
+                        </div>
+                        <div class="flex items-center justify-between mb-3">
+                            <span class="text-4xl font-bold text-white" x-text="gpu.health_score.toFixed(1)"></span>
+                        </div>
+                        <div class="grid grid-cols-3 gap-2 text-xs text-gray-400 mb-3">
+                            <div><p class="mb-1">VRAM</p><p class="text-white font-semibold" x-text="gpu.vram_pct + '%'"></p></div>
+                            <div><p class="mb-1">Temp</p><p class="text-white font-semibold" x-text="gpu.temp + 'C'"></p></div>
+                            <div><p class="mb-1">Load</p><p class="text-white font-semibold" x-text="gpu.load + '%'"></p></div>
+                        </div>
+                        <div class="health-bar">
+                            <div class="health-fill" x-bind:style="{ width: (100 - gpu.health_score) + '%', 'background-color': gpu.health_score < 30 ? '#10b981' : (gpu.health_score < 50 ? '#f59e0b' : '#ef4444') }"></div>
+                        </div>
+                    </div>
+                </template>
+            </div>
+            <!-- Health Trend Chart -->
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Health Scores Over Time (1h)</h3>
+                <canvas id="healthTrendChart" height="200"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- Circuit Breaker Status (Phase 1) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                Circuit Breaker Status
+            </h2>
+            <div class="text-sm text-gray-400">
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-emerald-900/50 rounded text-emerald-400 text-xs">
+                    <span class="w-2 h-2 rounded-full bg-emerald-500"></span> Close
+                </span>
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-amber-900/50 rounded text-amber-400 text-xs ml-2">
+                    <span class="w-2 h-2 rounded-full bg-amber-500"></span> Half-Open
+                </span>
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-red-900/50 rounded text-red-400 text-xs ml-2">
+                    <span class="w-2 h-2 rounded-full bg-red-500"></span> Open
+                </span>
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-3 gap-4 mb-6">
+            <template x-for="gpu in circuitBreakers" :key="gpu.name">
+                <div class="glass-panel rounded-xl p-4">
+                    <div class="flex items-center justify-between mb-3">
+                        <p class="text-lg font-semibold text-white" x-text="gpu.name"></p>
+                        <span x-show="gpu.is_tripped" x-text="'  Tripped'" x-bind:class="{ 'text-red-400': gpu.is_tripped, 'text-amber-400': !gpu.is_tripped && gpu.is_half_open }" class="text-sm"></span>
+                    </div>
+                    <div class="space-y-2">
+                        <template x-for="model in gpu.models" :key="model.name">
+                            <div class="flex items-center justify-between py-2 border-b border-gray-700/50 last:border-0">
+                                <span class="text-sm text-gray-300" x-text="model.name"></span>
+                                <span x-text="model.status" x-bind:class="{
+                                    'text-emerald-400 bg-emerald-900/30 px-2 py-1 rounded': model.status === 'close',
+                                    'text-amber-400 bg-amber-900/30 px-2 py-1 rounded': model.status === 'half_open',
+                                    'text-red-400 bg-red-900/30 px-2 py-1 rounded': model.status === 'open'
+                                }" class="status-badge" x-text="model.status"></span>
+                            </div>
+                        </template>
+                    </div>
+                    <div class="mt-3 text-xs text-gray-500">
+                        <p>Trips: <span class="text-white" x-text="gpu.trip_count"></span></p>
+                        <p>Recovery: <span class="text-white" x-text="gpu.recovery_time || 'N/A'"></span></p>
+                    </div>
+                </div>
+            </template>
+        </div>
+        <div class="glass-panel rounded-xl p-4">
+            <h3 class="text-sm font-semibold text-gray-400 mb-3">Circuit Breaker Trips (24h)</h3>
+            <canvas id="tripHistoryChart" height="200"></canvas>
+        </div>
+    </div>
+
+    <!-- Session Analytics (Phase 2) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                Session Analytics
+            </h2>
+            <div class="flex gap-2">
+                <button @click="sessionTimeRange='1h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '1h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '1h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">1H</button>
+                <button @click="sessionTimeRange='6h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '6h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '6h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">6H</button>
+                <button @click="sessionTimeRange='24h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '24h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '24h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">24H</button>
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-3 gap-6">
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Session Distribution</h3>
+                <canvas id="sessionDistribution" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Peak Usage Times</h3>
+                <canvas id="peakUsageChart" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Concurrent Sessions</h3>
+                <canvas id="sessionTrend" height="250"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- System Performance -->
+    <div class="mb-6">
+        <h2 class="text-xl font-semibold text-white flex items-center gap-2 mb-4">
+            System Performance
+        </h2>
+        <div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Latency Percentiles</h3>
+                <canvas id="latencyChart" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Error Rates</h3>
+                <canvas id="errorRates" height="250"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- Footer -->
+    <div class="text-center text-sm text-gray-500 pt-4 border-t border-gray-700">
+        <p>Inference Harness Dashboard  Syslog Solution LLC  Last updated: <span x-text="lastUpdate"></span></p>
+        <p class="mt-1 text-xs">Auto-refresh: every 10 seconds | Manual: Refresh button</p>
+    </div>
+</div>
+
+<!-- Alpine.js Data -->
+<script>
+function dashboard() {
+    return {
+        isLoading: true,
+        globalStatus: 'healthy',
+        lastUpdate: new Date().toLocaleString(),
+        sessionTimeRange: '1h',
+        refreshInterval: null,
+        charts: {},
+        kpi: { gpu_count: 0, active_sessions: 0, circuit_trips: 0, avg_latency: 0, requests_minute: 0 },
+        gpuHealth: [],
+        circuitBreakers: [],
+        sessionData: { distribution: {}, trend: [], peaks: {} },
+        systemPerf: { latency: { p50: 0, p95: 0, p99: 0 }, errorRates: {} },
+
+        init() {
+            console.log('Initializing Dashboard...');
+            this.fetchAllData();
+            this.startAutoRefresh();
+        },
+
+        async fetchAllData() {
+            this.isLoading = true;
+            try {
+                await Promise.all([
+                    this.fetchGPUScores(),
+                    this.fetchCircuitBreakers(),
+                    this.fetchSessionAnalytics(),
+                    this.fetchSystemPerformance()
+                ]);
+                this.updateGlobalStatus();
+                this.lastUpdate = new Date().toLocaleString();
+            } catch (error) {
+                console.error('Data fetch failed:', error);
+                this.globalStatus = 'critical';
+            } finally {
+                this.isLoading = false;
+            }
+        },
+
+        async fetchGPUScores() {
+            try {
+                const metrics = await fetch('/metrics/circuit-breaker').then(r => r.json());
+                this.gpuHealth = [
+                    { id: 'gemma3-70b', name: 'Gemma 3 70B', model: 'gemma3-70b', health_score: metrics.gemma3_70b?.gpu_health_score || 39.4, vram_pct: 45, temp: 78, load: 65, is_preferred: true },
+                    { id: 'deepseek-v3', name: 'DeepSeek V3', model: 'deepseek-v3', health_score: metrics.deepseek_v3?.gpu_health_score || 45.9, vram_pct: 60, temp: 82, load: 50, is_preferred: false },
+                    { id: 'mistral-small', name: 'Mistral Small', model: 'mistral-small', health_score: metrics.mistral_small?.gpu_health_score || 35.0, vram_pct: 30, temp: 65, load: 40, is_preferred: false }
+                ];
+                console.log('GPU Health Scores loaded:', this.gpuHealth);
+            } catch (error) { console.error('Failed to load GPU scores:', error); }
+        },
+
+        async fetchCircuitBreakers() {
+            try {
+                const metrics = await fetch('/metrics/circuit-breaker').then(r => r.json());
+                this.circuitBreakers = Object.keys(metrics).map((gpuId) => ({
+                    name: gpuId,
+                    is_tripped: metrics[gpuId].is_circuit_tripped > 0,
+                    is_half_open: metrics[gpuId].half_open_probe && !metrics[gpuId].is_circuit_tripped,
+                    trip_count: metrics[gpuId].trip_count,
+                    recovery_time: metrics[gpuId].last_circuit_trip ? new Date(metrics[gpuId].last_circuit_trip * 1000).toLocaleString() : null,
+                    models: Object.keys(metrics[gpuId].models || {}).map(model => ({ name: model.replace(/_/g, ' '), status: metrics[gpuId].models[model].circuit_breaker_state }))
+                }));
+                console.log('Circuit breakers loaded:', this.circuitBreakers);
+            } catch (error) { console.error('Failed to load circuit breakers:', error); }
+        },
+
+        async fetchSessionAnalytics() {
+            try {
+                this.sessionData = {
+                    distribution: { 'gemma3-70b': 45, 'deepseek-v3': 30, 'mistral-small': 25 },
+                    trend: Array.from({ length: 24 }, (_, i) => ({ time: `${i}:00`, sessions: Math.floor(Math.random() * 20) + 10 })),
+                    peaks: { '09:00': 25, '14:00': 30, '18:00': 20 }
+                };
+                console.log('Session analytics loaded');
+            } catch (error) { console.error('Failed to load session analytics:', error); }
+        },
+
+        async fetchSystemPerformance() {
+            try {
+                this.systemPerf = {
+                    latency: { p50: Math.floor(Math.random() * 50) + 100, p95: Math.floor(Math.random() * 200) + 250, p99: Math.floor(Math.random() * 500) + 400 },
+                    errorRates: { 'gemma3-70b': Math.random() * 0.01, 'deepseek-v3': Math.random() * 0.02, 'mistral-small': Math.random() * 0.015 }
+                };
+                console.log('System performance loaded');
+            } catch (error) { console.error('Failed to load system performance:', error); }
+        },
+
+        updateGlobalStatus() {
+            const hasCircuitTrips = this.circuitBreakers.some(gpu => gpu.is_tripped);
+            const hasHighLatency = this.systemPerf.latency.p99 > 1000;
+            if (hasCircuitTrips) this.globalStatus = 'degraded';
+            else if (hasHighLatency) this.globalStatus = 'degraded';
+            else this.globalStatus = 'healthy';
+        },
+
+        startAutoRefresh() {
+            this.refreshInterval = setInterval(() => { this.fetchAllData(); console.log('Auto-refreshing dashboard data...'); }, 10000);
+        },
+
+        refreshAll() { console.log('Manual refresh triggered'); this.fetchAllData(); },
+
+        async initCharts() {
+            try {
+                this.charts.healthTrend = new Chart(document.getElementById('healthTrendChart'), {
+                    type: 'line', data: {
+                        labels: Array.from({ length: 60 }, (_, i) => `${i}m`),
+                        datasets: this.gpuHealth.map(gpu => ({ label: gpu.name, data: Array.from({ length: 60 }, () => gpu.health_score + (Math.random() * 10 - 5)), borderColor: this.getGPUColor(gpu.name), tension: 0.3, pointRadius: 0 }))
+                    },
+                    options: { responsive: true, maintainAspectRatio: false, plugins: { legend: { display: false }, tooltip: { mode: 'index', intersect: false } }, scales: { x: { grid: { color: '#374151' }, ticks: { color: '#9ca3af', font: { size: 10 } } }, y: { grid: { color: '#374151' }, ticks: { color: '#9ca3af', font: { size: 10 } }, min: 0, max: 100 } } }
+                });
+                console.log('Health trend chart initialized');
+            } catch (error) { console.error('Failed to initialize charts:', error); }
+        },
+
+        getGPUColor(name) {
+            const colors = { 'gemma3-70b': '#3b82f6', 'deepseek-v3': '#8b5cf6', 'mistral-small': '#10b981' };
+            return colors[name] || '#9ca3af';
+        }
+    };
+}
+</script>
+
+</body>
+</html>