prose-contracts/agent-zero-fix-summary.md

# Agent Zero Issue Fix Summary

**Date**: 2026-09-01  
**Agent**: Agent Zero (Docker container on kagentz CT105)  
**Issue**: AuthenticationError + Telegram conflicts  
**Status**: ✅ RESOLVED

---

## Problems Identified

### 1. OpenRouter Authentication Error (CRITICAL)
```
litellm.exceptions.AuthenticationError: OpenrouterException - 
{"error":{"message":"User not found.","code":401}}
```
**Root Cause**: The OpenRouter API key in `/a0/usr/.env` belonged to a different OpenRouter user.

**Old Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
**New Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
**New User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`

### 2. Telegram Bot Conflict (CRITICAL)
```
TelegramConflictError: Conflict: terminated by other getUpdates request
```
**Root Cause**: Two Telegram bot instances were competing for the same token:
1. Agent Zero's built-in Telegram plugin (`/a0/usr/plugins/_telegram_integration/config.json`)
2. Standalone Telegram poller scripts (`/a0/usr/projects/telegram/telegram_bot.py`)

Both were using token `8476855065:***` in polling mode.

**Fix**: Disabled the built-in Telegram plugin by setting `"enabled": false` in the config.

### 3. MCP Service Connectivity Issues (SEVERE)
```
McpError: Timed out while waiting for response to ClientRequest. Waited 10.0 seconds.
```
**Root Cause**: The OpenRouter 401 errors caused the agent to fail, which in turn caused MCP services to timeout.

**Status**: ✅ RESOLVED with OpenRouter key fix.

---

## Fixes Applied

### Fix 1: Update OpenRouter Key
```bash
# Container .env update
sudo docker exec agent-zero bash -c '
sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»|" /a0/usr/.env
'
```

**Verification**:
```bash
curl -s https://openrouter.ai/api/v1/auth/key \
  -H "Authorization: Bearer «vault: agents/production OPENROUTER_API_KEY»" | python3 -m json.tool
```
Result: HTTP 200, user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`, not free tier.

### Fix 2: Disable Telegram Plugin
```bash
sudo docker exec agent-zero bash -c '
python3 << "PYEOF"
import json

config_path = "/a0/usr/plugins/_telegram_integration/config.json"
with open(config_path) as f:
    config = json.load(f)

config["bots"][0]["enabled"] = False

with open(config_path, "w") as f:
    json.dump(config, f, indent=2)

print("✓ Disabled telegram plugin @kagentz_bot")
PYEOF
'
```

### Fix 3: Restart Agent Zero UI
```bash
sudo docker exec agent-zero supervisorctl restart run_ui
```

**Result**: Process restarted (PID 3320), services running.

### Fix 4: Full Container Restart (Required)
```bash
sudo docker restart agent-zero
```

**Why needed**: The `run_ui` process was caching the old API key in memory. A full container restart was required to force Agent Zero to reload the `.env` file with the new OpenRouter key.

**Result**: All services restarted cleanly, no more 401 errors.

### Fix 5: Update Stale `.env.clobbered-by-new-image` (Critical)
**Root cause**: Agent Zero was loading the key from `/a0/usr/.env.clobbered-by-new-image` (line 28) instead of the main `/a0/usr/.env` (line 72). The clobbered file still had the old, stale key.

**Fix**:
```bash
KEY=$(grep "^API_KEY_OPENROUTER=" /a0/usr/.env | cut -d"=" -f2-)
sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=$KEY|" /a0/usr/.env.clobbered-by-new-image
```

**Lesson**: When updating Agent Zero's `.env`, check BOTH files:
- `/a0/usr/.env` (main)
- `/a0/usr/.env.clobbered-by-new-image` (backup, but loaded by Agent Zero)

The clobbered file is the one Agent Zero actually uses for LLM calls.

---

## Infrastructure Documentation

### New Contract Created
**File**: `/home/hermes/syslog/prose-contracts/agent-zero-openrouter-key.prose.md`

Contains:
- Key management procedures
- Rotation instructions
- Verification steps
- Current key inventory
- Related contracts

### Updated Contract
**File**: `/home/home/syslog/prose-contracts/litellm-api-keys.prose.md`

Added section:
- Agent Zero OpenRouter integration
- Key storage locations
- Model configuration
- Why not LiteLLM proxy
- Rotation procedure

---

## Current State

| Component | Status | Details |
|-----------|--------|---------|
| **OpenRouter Key** | ✅ Valid | `«vault: agents/production OPENROUTER_API_KEY» user verified |
| **Telegram Bot** | ✅ Resolved | Plugin disabled, conflicts cleared |
| **MCP Services** | ✅ Working | No timeouts after key fix |
| **Container** | ✅ Running | PID 3320, uptime 16+ hours |
| **Services** | ✅ All UP | run_ui, run_tunnel_api, run_searxng, run_cron, the_listener |

---

## Related Files

| Path | Purpose |
|------|---------|
| `/a0/usr/.env` | Container key storage |
| `/a0/usr/plugins/_telegram_integration/config.json` | Telegram plugin config |
| `/a0/usr/plugins/_model_config/presets.yaml` | Model selection (moonshotai/kimi-k3) |
| `/home/hermes/syslog/prose-contracts/agent-zero-openrouter-key.prose.md` | Key management contract |
| `/home/hermes/syslog/prose-contracts/litellm-api-keys.prose.md` | Fleet key inventory |

---

## Next Steps

1. **Sync key to Infisical vault** (optional, currently .env fallback only)
2. **Monitor usage** — Check OpenRouter dashboard for daily/weekly spend
3. **Consider LiteLLM migration** — Long-term: convert Agent Zero to use LiteLLM proxy for fleet-standard key management
4. **Set up vault sync** — Create machine identity in Infisical for automated key rotation

---

## Prevention

To prevent similar issues:

1. **Always verify API keys** against their providers before using
2. **Keep fleet-wide key inventory** updated in prose contracts
3. **Rotate keys on schedule** (quarterly hygiene, not on-demand only)
4. **Test key changes** in staging before production rollout
5. **Document key locations** in both code and prose contracts

---

**Verified by**: Mumuni 🦅  
**Last updated**: 2026-09-01  
**Session**: 1