prose-contracts/agent-zero-fix-summary.md

Agent Zero Issue Fix Summary

Date: 2026-09-01
Agent: Agent Zero (Docker container on kagentz CT105)
Issue: AuthenticationError + Telegram conflicts
Status: ✅ RESOLVED

---

Problems Identified

1. OpenRouter Authentication Error (CRITICAL)

litellm.exceptions.AuthenticationError: OpenrouterException - 
{"error":{"message":"User not found.","code":401}}

Root Cause: The OpenRouter API key in /a0/usr/.env belonged to a different OpenRouter user.

Old Key: «vault: agents/production OPENROUTER_API_KEY»
New Key: «vault: agents/production OPENROUTER_API_KEY»
New User: user_2rt9lCqcd5d7Vk1t18DHsvWdPTT

2. Telegram Bot Conflict (CRITICAL)

TelegramConflictError: Conflict: terminated by other getUpdates request

Root Cause: Two Telegram bot instances were competing for the same token:

1. Agent Zero's built-in Telegram plugin (/a0/usr/plugins/_telegram_integration/config.json)
2. Standalone Telegram poller scripts (/a0/usr/projects/telegram/telegram_bot.py)

Both were using token 8476855065:*** in polling mode.

Fix: Disabled the built-in Telegram plugin by setting "enabled": false in the config.

3. MCP Service Connectivity Issues (SEVERE)

McpError: Timed out while waiting for response to ClientRequest. Waited 10.0 seconds.

Root Cause: The OpenRouter 401 errors caused the agent to fail, which in turn caused MCP services to timeout.

Status: ✅ RESOLVED with OpenRouter key fix.

---

Fixes Applied

Fix 1: Update OpenRouter Key

# Container .env update
sudo docker exec agent-zero bash -c '
sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»|" /a0/usr/.env
'

Verification:

curl -s https://openrouter.ai/api/v1/auth/key \
  -H "Authorization: Bearer «vault: agents/production OPENROUTER_API_KEY»" | python3 -m json.tool

Result: HTTP 200, user user_2rt9lCqcd5d7Vk1t18DHsvWdPTT, not free tier.

Fix 2: Disable Telegram Plugin

sudo docker exec agent-zero bash -c '
python3 << "PYEOF"
import json

config_path = "/a0/usr/plugins/_telegram_integration/config.json"
with open(config_path) as f:
    config = json.load(f)

config["bots"][0]["enabled"] = False

with open(config_path, "w") as f:
    json.dump(config, f, indent=2)

print("✓ Disabled telegram plugin @kagentz_bot")
PYEOF
'

Fix 3: Restart Agent Zero UI

sudo docker exec agent-zero supervisorctl restart run_ui

Result: Process restarted (PID 3320), services running.

Fix 4: Full Container Restart (Required)

sudo docker restart agent-zero

Why needed: The run_ui process was caching the old API key in memory. A full container restart was required to force Agent Zero to reload the .env file with the new OpenRouter key.

Result: All services restarted cleanly, no more 401 errors.

Fix 5: Update Stale .env.clobbered-by-new-image (Critical)

Root cause: Agent Zero was loading the key from /a0/usr/.env.clobbered-by-new-image (line 28) instead of the main /a0/usr/.env (line 72). The clobbered file still had the old, stale key.

Fix:

KEY=$(grep "^API_KEY_OPENROUTER=" /a0/usr/.env | cut -d"=" -f2-)
sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=$KEY|" /a0/usr/.env.clobbered-by-new-image

Lesson: When updating Agent Zero's .env, check BOTH files:

• /a0/usr/.env (main)
• /a0/usr/.env.clobbered-by-new-image (backup, but loaded by Agent Zero)

The clobbered file is the one Agent Zero actually uses for LLM calls.

---

Infrastructure Documentation

New Contract Created

File: /home/hermes/syslog/prose-contracts/agent-zero-openrouter-key.prose.md

Contains:

• Key management procedures
• Rotation instructions
• Verification steps
• Current key inventory
• Related contracts

Updated Contract

File: /home/home/syslog/prose-contracts/litellm-api-keys.prose.md

Added section:

• Agent Zero OpenRouter integration
• Key storage locations
• Model configuration
• Why not LiteLLM proxy
• Rotation procedure

---

Current State

Component	Status	Details
OpenRouter Key	✅ Valid	`«vault: agents/production OPENROUTER_API_KEY» user verified
Telegram Bot	✅ Resolved	Plugin disabled, conflicts cleared
MCP Services	✅ Working	No timeouts after key fix
Container	✅ Running	PID 3320, uptime 16+ hours
Services	✅ All UP	run_ui, run_tunnel_api, run_searxng, run_cron, the_listener

---

Related Files

Path	Purpose
/a0/usr/.env	Container key storage
/a0/usr/plugins/_telegram_integration/config.json	Telegram plugin config
/a0/usr/plugins/_model_config/presets.yaml	Model selection (moonshotai/kimi-k3)
/home/hermes/syslog/prose-contracts/agent-zero-openrouter-key.prose.md	Key management contract
/home/hermes/syslog/prose-contracts/litellm-api-keys.prose.md	Fleet key inventory

---

Next Steps

1. Sync key to Infisical vault (optional, currently .env fallback only)
2. Monitor usage — Check OpenRouter dashboard for daily/weekly spend
3. Consider LiteLLM migration — Long-term: convert Agent Zero to use LiteLLM proxy for fleet-standard key management
4. Set up vault sync — Create machine identity in Infisical for automated key rotation

---

Prevention

To prevent similar issues:

1. Always verify API keys against their providers before using
2. Keep fleet-wide key inventory updated in prose contracts
3. Rotate keys on schedule (quarterly hygiene, not on-demand only)
4. Test key changes in staging before production rollout
5. Document key locations in both code and prose contracts

---

Verified by: Mumuni 🦅
Last updated: 2026-09-01
Session: 1