docs: lessons learned from 2026-07-12 session
- gpu-fleet: Corrected architecture (direct GPU, no router in path). Updated context values (RTX 3090=256K, not 128K). Added api-key standardization requirement. - gpu-self-heal: Added Lessons Learned section with 5 critical findings: L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed) L2: Fallback chain cascading failure loop detection L3: Verify running state, not documentation L4: Infisical fallback requirement (.env must have uncommented key) L5: Zulip event queue can silently die after ~40 reconnects - litellm-self-heal: Updated status manual-only→deployed, cron schedule - litellm-api-keys: Added Infisical token expiry warning + .env fallback - hermes-config-template: Rule 3 updated with .env fallback requirement
This commit is contained in:
root
@@ -11,7 +11,7 @@ triggers:
|
||||
- on "infra update" command
|
||||
- weekly (Sunday 03:00 EDT) via cron
|
||||
- on security advisory relay from Mumuni
|
||||
version: 1.1.0
|
||||
version: 1.2.0
|
||||
---
|
||||
|
||||
## Maintains
|
||||
@@ -27,11 +27,12 @@ Before ANY update wave:
|
||||
2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
|
||||
3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
|
||||
4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
|
||||
5. ✅ LiteLLM health check passing
|
||||
6. ✅ Zulip server reachable
|
||||
6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
|
||||
7. ✅ Disk >20% free on all nodes
|
||||
8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
|
||||
5. ✅ LiteLLM health check passing (port 4000, /mcp-rest/tools/list with master key)
|
||||
6. ✅ LiteLLM MCP gateway serving RA-H OS tools (90 tools)
|
||||
7. ✅ Zulip server reachable
|
||||
8. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
|
||||
9. ✅ Disk >20% free on all nodes
|
||||
10. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
|
||||
|
||||
## Wave 1: Storage & Infra Nodes (lowest impact)
|
||||
|
||||
@@ -65,6 +66,7 @@ Before ANY update wave:
|
||||
**Verify after Wave 2:**
|
||||
- All VMs/CTs running: check via Proxmox API
|
||||
- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
|
||||
- LiteLLM MCP tools: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools
|
||||
- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
|
||||
- Zulip agents connected: check Mumuni/Tanko gateway state
|
||||
- Abiba PM2 processes online: `pm2 status`
|
||||
@@ -83,6 +85,7 @@ Before ANY update wave:
|
||||
**Verify after Wave 3:**
|
||||
- All containers healthy: `docker ps` on each host
|
||||
- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
|
||||
- MCP integration test: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools (23 RA-H OS + 67 GitHub)
|
||||
- Zulip test: send test message to #agent-hub
|
||||
- Dashboard loading: `curl localhost:3001/` (via CT 116)
|
||||
- Firecrawl test: `curl :3002/`
|
||||
@@ -112,8 +115,8 @@ If ANY verification fails:
|
||||
|
||||
Before Wave 1, snapshot these files:
|
||||
```
|
||||
/opt/inference-harness/docker-compose.yml (CT 116 .116)
|
||||
/opt/inference-harness/litellm_config.yaml (CT 116 .116)
|
||||
/opt/inference-harness/docker-compose.yml (CT 116 .116) ⚡ contains MCP_SERVER env vars
|
||||
/opt/inference-harness/litellm_config.yaml (CT 116 .116) ⚡ contains mcp_servers.ra_h_os
|
||||
/opt/monitoring/prometheus.yml (CT 116 .116)
|
||||
/etc/nginx/nginx.conf (harness-nginx on CT 116)
|
||||
/opt/search-stack/firecrawl-source/docker-compose.yaml (VM 109 .7)
|
||||
@@ -123,9 +126,51 @@ Before Wave 1, snapshot these files:
|
||||
/root/.pi/agent/extensions/config.yaml (CT 100 .24)
|
||||
/etc/systemd/system/ornith-server.service (amdpve .15)
|
||||
/etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110)
|
||||
# Hermes agent configs (key enforcement — 2026-07-10)
|
||||
/root/.hermes/config.yaml (Mumuni CT 114, Tanko CT 112, etc.)
|
||||
/root/.config/systemd/user/hermes-gateway.service (Mumuni CT 114 — EnvironmentFile fixed)
|
||||
/etc/environment (Mumuni CT 114 — LITELLM_API_KEY)
|
||||
```
|
||||
|
||||
Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
|
||||
## MCP Gateway (2026-07-10)
|
||||
|
||||
LiteLLM CT 116 now serves as an authenticated MCP gateway for RA-H OS tools.
|
||||
|
||||
### Configuration
|
||||
|
||||
**litellm_config.yaml** (`/opt/inference-harness/litellm_config.yaml`):
|
||||
```yaml
|
||||
mcp_servers:
|
||||
ra_h_os:
|
||||
url: "http://192.168.68.65:3100/mcp"
|
||||
transport: "http"
|
||||
auth_type: "none"
|
||||
```
|
||||
|
||||
**docker-compose.yml** env vars:
|
||||
```yaml
|
||||
- MCP_SERVER_RAHOS_URL=http://192.168.68.65:3100/mcp
|
||||
- MCP_SERVER_RAHOS_TRANSPORT=http
|
||||
```
|
||||
|
||||
### Access
|
||||
|
||||
| Key | MCP Access |
|
||||
|-----|-----------|
|
||||
| Master key | ✅ Full — 90 tools (vault-injected) |
|
||||
| Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.90.0-rc.1 |
|
||||
|
||||
### Known Limitations
|
||||
- Per-key MCP server grants not functional — only master key has access
|
||||
- Responses API (`/v1/responses`) with MCP tools broken on llama.cpp backends
|
||||
- HTTP 307 redirect on `/mcp` → use `/mcp/` (trailing slash) or `/mcp-rest/` endpoints
|
||||
- `api_mode: responses` in Hermes appends `/v1/responses` to base_url → **base_url must end at `/v1`, never `/responses`** (double-path bug)
|
||||
|
||||
### Migration Path
|
||||
When LiteLLM is upgraded to a version supporting per-key MCP grants:
|
||||
1. Grant agent keys `mcp_servers: ["ra_h_os"]`
|
||||
2. Update Hermes `mcp_servers.ra-h-os.url` from `http://192.168.68.65:3100/mcp` → `http://192.168.68.116:4000/mcp/`
|
||||
3. Add `headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"}` to MCP config
|
||||
|
||||
## Security-Specific Updates
|
||||
|
||||
@@ -144,6 +189,7 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
|
||||
- [ ] LiteLLM inference passing (syslog-auto test)
|
||||
- [ ] Zulip server + all 3 agents connected
|
||||
- [ ] GPU fleet at full capacity (3/3)
|
||||
- [ ] LiteLLM MCP gateway healthy (90 tools via master key)
|
||||
- [ ] Zero security CVEs remaining
|
||||
- [ ] <10 min total downtime per service
|
||||
|
||||
|
||||
Reference in New Issue
Block a user