Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 root
|
15e8998289 | ||
|
root
|
4819247e39 | ||
|
root
|
1dc4155d9a | ||
|
root
|
09efbdf4e5 | ||
 jerome
|
4058010e57 | ||
|
jerome
|
c0191c9edc |
@@ -224,32 +224,69 @@ dangling images and orphaned build cache. No automated GC was in place.
|
||||
- This contract now runs disk GC fleet-wide every 6 hours
|
||||
- Docker hosts get `docker system prune` on amber, `-a --force` on red
|
||||
- kagentz is flagged as HIGH risk due to development activity
|
||||
- amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
|
||||
```
|
||||
|
||||
Done. Here's what we have:
|
||||
## Incident Log: 2026-07-09 — amdpve docker bloat
|
||||
|
||||
## Storage Summary
|
||||
### Discovery
|
||||
Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
|
||||
amdpve (.15) flagged at 78% (AMBER threshold: 75%).
|
||||
|
||||
| System | Used | Free | Health |
|
||||
|--------|------|------|--------|
|
||||
| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
|
||||
| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
|
||||
| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
|
||||
| **/root/go** (abiba) | 773M | — | Fine |
|
||||
### Diagnosis
|
||||
- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
|
||||
- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
|
||||
- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
|
||||
- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
|
||||
(`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
|
||||
- Not a running service — zero containers, zero active volumes
|
||||
|
||||
## New Contract: `disk-gc-threat-response.prose.md`
|
||||
### Resolution
|
||||
```
|
||||
docker system prune -a --force
|
||||
→ Reclaimed 11.56GB
|
||||
→ Post: 55G / 94G (62%), 35G free
|
||||
→ 1 image removed (llama-strix-hip:latest, 15.5GB)
|
||||
→ 8 build cache layers removed
|
||||
→ amdpve now GREEN
|
||||
```
|
||||
|
||||
Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
|
||||
### Root Cause
|
||||
Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
|
||||
artifacts. Docker on amdpve serves no running purpose — it's only used for
|
||||
one-off GPU builds. No automated post-migration cleanup was in place.
|
||||
|
||||
### Three-in-One
|
||||
### Preventive Measures
|
||||
- amdpve added to Docker GC scan list
|
||||
- Post-migration cleanup step added: after any GPU backend migration, prune
|
||||
the old backend's Docker images within 24 hours
|
||||
- Contract now scans GPU bare-metal hosts alongside CTs
|
||||
- Access via `pct-run` script for all CTs (no hardcoded IPs)
|
||||
|
||||
1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
|
||||
## Access Matrix (documented 2026-07-09)
|
||||
|
||||
2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
|
||||
### CT Access (via pct-run)
|
||||
| CT | Name | Node | Status |
|
||||
|----|------|------|--------|
|
||||
| 100 | abiba | amdpve | local |
|
||||
| 102 | adguard | acerpve | ✅ reachable |
|
||||
| 104 | authentik | minipve | ✅ reachable |
|
||||
| 105 | kagentz | amdpve | ✅ reachable |
|
||||
| 106 | ra-h-os | storepve | ✅ reachable |
|
||||
| 107 | pbs | storepve | ✅ reachable |
|
||||
| 108 | media | storepve | ✅ reachable |
|
||||
| 110 | gitea | minipve | ✅ reachable |
|
||||
| 111 | tdunna | amdpve | ✅ reachable |
|
||||
| 112 | tanko | amdpve | ✅ reachable |
|
||||
| 113 | baggy | amdpve | ✅ reachable |
|
||||
| 114 | mumuni | minipve | ✅ reachable |
|
||||
| 115 | scottdenya | amdpve | ✅ reachable |
|
||||
| 116 | syslog-api | minipve | ✅ reachable |
|
||||
| 117 | zulip | storepve | ✅ reachable |
|
||||
|
||||
3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
|
||||
|
||||
### The incident root cause
|
||||
Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
|
||||
|
||||
Want me to push this to the prose-contracts repo?
|
||||
### GPU Bare Metal (via direct SSH)
|
||||
| Host | IP | GPU | Status |
|
||||
|------|-----|-----|--------|
|
||||
| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
|
||||
| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
|
||||
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
|
||||
@@ -0,0 +1,237 @@
|
||||
---
|
||||
kind: function
|
||||
name: hermes-zulip-plugin
|
||||
description: >
|
||||
Installs or updates the Zulip platform plugin for any Hermes agent from the
|
||||
canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
|
||||
the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
|
||||
recovery). Verifies the installation, restarts the gateway, and sends a relay
|
||||
success signal.
|
||||
agent: abiba
|
||||
version: 1.0.0
|
||||
status: active
|
||||
runtime_contract: 2
|
||||
---
|
||||
|
||||
# Hermes Zulip Plugin — Install & Repair
|
||||
|
||||
Single-shot function that pulls the latest zulip-platform plugin from the
|
||||
canonical git repo, installs it to the correct Hermes bundled plugin path,
|
||||
verifies the installation, and signals completion.
|
||||
|
||||
Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
|
||||
layer and a targeted gateway restart. It does NOT verify env credentials or
|
||||
run a live Zulip connection test. Use `hermes-zulip-restore` for full
|
||||
connectivity recovery including end-to-end DM validation.
|
||||
|
||||
## Parameters
|
||||
|
||||
| Param | Type | Required | Default | Description |
|
||||
|-------|------|----------|---------|-------------|
|
||||
| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
|
||||
| `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
|
||||
|
||||
## Maintains
|
||||
|
||||
- plugin_installed: bool — Whether all three adapter files exist at the bundled path
|
||||
- plugin_version: string — Git commit SHA of the installed version
|
||||
- strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
|
||||
- signal_sent: bool — Whether relay success message was dispatched
|
||||
|
||||
### Postconditions
|
||||
|
||||
- All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
|
||||
- `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
|
||||
- Installed version matches HEAD of the requested branch
|
||||
- Relay success signal sent to Hermes agent's inbox
|
||||
|
||||
## Requires
|
||||
|
||||
- SSH access to target host (direct or via amdpve for CTs)
|
||||
- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
|
||||
- Python 3 with `httpx` installed on target
|
||||
|
||||
## Live-State Fields
|
||||
|
||||
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
||||
|------|-----|---------|-------------|-------------|------|
|
||||
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
|
||||
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
|
||||
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
|
||||
|
||||
| Field | Value | Trust |
|
||||
|-------|-------|-------|
|
||||
| Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
|
||||
| Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
|
||||
| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
|
||||
| Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
|
||||
| Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
|
||||
|
||||
## Execution
|
||||
|
||||
### Step 1: Resolve Target
|
||||
|
||||
Map `target` to host, CT ID, hermes_home, and user from the live-state table.
|
||||
For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
|
||||
|
||||
### Step 2: Pull Latest Plugin Source
|
||||
|
||||
On the target host:
|
||||
|
||||
```bash
|
||||
# Ensure deploy scratch space
|
||||
mkdir -p /tmp/zulip-deploy
|
||||
cd /tmp/zulip-deploy
|
||||
|
||||
# Clone or pull
|
||||
if [ -d zulip-platform-plugins ]; then
|
||||
cd zulip-platform-plugins
|
||||
git fetch origin
|
||||
git checkout {{branch}}
|
||||
git pull origin {{branch}}
|
||||
else
|
||||
git clone --branch {{branch}} \
|
||||
https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
|
||||
cd zulip-platform-plugins
|
||||
fi
|
||||
|
||||
# Capture installed version
|
||||
INSTALLED_SHA=$(git rev-parse HEAD)
|
||||
echo "Installed SHA: $INSTALLED_SHA"
|
||||
|
||||
# Verify we're at HEAD
|
||||
HEAD_SHA=$(git rev-parse origin/{{branch}})
|
||||
if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
|
||||
echo "At latest commit on {{branch}}"
|
||||
else
|
||||
echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
|
||||
fi
|
||||
```
|
||||
|
||||
### Step 3: Install Plugin Files
|
||||
|
||||
```bash
|
||||
# Ensure target directory exists
|
||||
mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
|
||||
|
||||
# Copy adapter files
|
||||
cp plugins/platforms/zulip/adapter.py \
|
||||
plugins/platforms/zulip/__init__.py \
|
||||
plugins/platforms/zulip/plugin.yaml \
|
||||
{{hermes_home}}/hermes-agent/plugins/platforms/zulip/
|
||||
|
||||
# Fix ownership (Tanko only — runs as jerome user)
|
||||
[ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
|
||||
{{hermes_home}}/hermes-agent/plugins/platforms/zulip/
|
||||
|
||||
echo "Plugin files installed"
|
||||
```
|
||||
|
||||
### Step 4: Verify Installation
|
||||
|
||||
```bash
|
||||
# Check all three files exist
|
||||
for f in __init__.py adapter.py plugin.yaml; do
|
||||
if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
|
||||
echo "✅ $f present"
|
||||
else
|
||||
echo "❌ $f MISSING"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Verify _strip_html fix
|
||||
grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
|
||||
&& echo "✅ _strip_html fix present" \
|
||||
|| echo "❌ _strip_html MISSING — plugin may be stale"
|
||||
|
||||
# Show installed plugin.yaml version
|
||||
grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
|
||||
```
|
||||
|
||||
### Step 5: Restart Gateway
|
||||
|
||||
Plugin changes require a gateway restart to take effect:
|
||||
|
||||
```bash
|
||||
cd {{hermes_home}}/hermes-agent
|
||||
# Use venv if available
|
||||
python3 -m hermes_cli.main gateway restart 2>&1 || \
|
||||
venv/bin/python -m hermes_cli.main gateway restart 2>&1
|
||||
```
|
||||
|
||||
Wait for restart to complete (up to 45s), then confirm:
|
||||
|
||||
```bash
|
||||
grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
|
||||
```
|
||||
|
||||
Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
|
||||
|
||||
Quick smoke check — confirm zulip platform loaded:
|
||||
|
||||
```bash
|
||||
grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
|
||||
```
|
||||
|
||||
If gateway fails to restart, check logs for the crash cause before proceeding.
|
||||
|
||||
### Step 6: Send Relay Success Signal
|
||||
|
||||
On the Abiba host (local), dispatch a relay message to the target agent:
|
||||
|
||||
```
|
||||
ra-h-os-createRelayNode:
|
||||
title: "Zulip plugin updated — {{target}}"
|
||||
source: |
|
||||
Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
|
||||
All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
|
||||
_strip_html fix: PRESENT
|
||||
Timestamp: {{timestamp}}
|
||||
|
||||
description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
|
||||
```
|
||||
|
||||
### Step 7: Report
|
||||
|
||||
Compile results into a single status block:
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Target | `{{target}}` |
|
||||
| Branch | `{{branch}}` |
|
||||
| Commit SHA | `{{INSTALLED_SHA}}` |
|
||||
| Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
|
||||
| `_strip_html` | `{{present|missing}}` |
|
||||
| Gateway restarted | `{{yes|no}}` |
|
||||
| Signal sent | `{{yes|no}}` |
|
||||
|
||||
## Known Failure Modes
|
||||
|
||||
| Symptom | Root Cause | Recovery |
|
||||
|---------|-----------|----------|
|
||||
| Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
|
||||
| Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
|
||||
| `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
|
||||
| Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
|
||||
| Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
|
||||
|
||||
## Edge Differences from hermes-zulip-restore
|
||||
|
||||
| Concern | hermes-zulip-restore | hermes-zulip-plugin |
|
||||
|---------|---------------------|---------------------|
|
||||
| Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
|
||||
| Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
|
||||
| Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
|
||||
| Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
|
||||
| Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
|
||||
| Signal dispatch | ❌ None | ✅ Relay message to target |
|
||||
|
||||
For full connectivity recovery after a plugin install, chain this contract
|
||||
with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
|
||||
|
||||
---
|
||||
|
||||
**Last updated**: 2026-07-08 — Switched default branch to `master`; added
|
||||
gateway restart step. If outstanding unmerged feature branches exist, address
|
||||
them in a follow-up merge after this contract completes.
|
||||
@@ -0,0 +1,188 @@
|
||||
---
|
||||
kind: function
|
||||
name: hermes-zulip-restore
|
||||
description: >
|
||||
Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
|
||||
Koby CT111). Deploys the zulip-platform adapter to the correct bundled plugin
|
||||
path, verifies env credentials, restarts the gateway, and confirms Zulip
|
||||
connects. Run this whenever a Hermes agent stops responding on Zulip or after
|
||||
a fresh agent deployment.
|
||||
agent: abiba
|
||||
version: 1.0.0
|
||||
status: active
|
||||
runtime_contract: 2
|
||||
---
|
||||
|
||||
# Hermes Zulip Restore — Bring Any Agent Back to Good State
|
||||
|
||||
Single-shot function that restores full Zulip connectivity for a Hermes agent.
|
||||
Covers adapter deployment, HTML stripping (slash command fix), env verification,
|
||||
gateway restart, and connection validation.
|
||||
|
||||
## Parameters
|
||||
|
||||
| Param | Type | Required | Default | Description |
|
||||
|-------|------|----------|---------|-------------|
|
||||
| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
|
||||
|
||||
## Maintains
|
||||
|
||||
- adapter_deployed: bool — Whether `_strip_html` adapter is at correct bundled path
|
||||
- zulip_connected: bool — Whether gateway_state shows zulip.state = "connected"
|
||||
- env_valid: bool — Whether .env has ZULIP_SITE, ZULIP_EMAIL, ZULIP_API_KEY
|
||||
- gateway_running: bool — Whether gateway process is running
|
||||
|
||||
### Postconditions
|
||||
|
||||
- `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
|
||||
- All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
|
||||
- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
|
||||
- Gateway restarted and zulip platform reports state `connected`
|
||||
- HTML stripping enabled for `/approve` and `/deny` slash command support
|
||||
|
||||
## Requires
|
||||
|
||||
- SSH access to target host (direct or via amdpve for CTs)
|
||||
- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
|
||||
- Python 3 with `httpx` installed on target
|
||||
- Zulip server accessible at `https://chat.sysloggh.net`
|
||||
|
||||
## Live-State Fields
|
||||
|
||||
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
||||
|------|-----|---------|-------------|-------------|------|
|
||||
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
|
||||
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
|
||||
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
|
||||
|
||||
| Field | Value | Trust |
|
||||
|-------|-------|-------|
|
||||
| Zulip server | https://chat.sysloggh.net | ✅ Verified |
|
||||
| Git repo (zulip-platform) | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
|
||||
| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
|
||||
| Git branch | `feat/zulip-streaming` | ✅ Verified (contains _strip_html fix) |
|
||||
|
||||
## Execution
|
||||
|
||||
### Step 1: Locate Target
|
||||
|
||||
Map `target` to connectivity parameters from the live-state table above.
|
||||
For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
|
||||
|
||||
### Step 2: Deploy Zulip Adapter
|
||||
|
||||
On the target host:
|
||||
|
||||
```bash
|
||||
# Clone or update the plugin repo
|
||||
mkdir -p /tmp/zulip-deploy
|
||||
cd /tmp/zulip-deploy
|
||||
if [ -d zulip-platform-plugins ]; then
|
||||
cd zulip-platform-plugins && git pull origin feat/zulip-streaming
|
||||
else
|
||||
git clone --branch feat/zulip-streaming \
|
||||
https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
|
||||
fi
|
||||
|
||||
# Ensure bundled plugin directory exists
|
||||
mkdir -p <HERMES_HOME>/hermes-agent/plugins/platforms/zulip
|
||||
|
||||
# Copy adapter files
|
||||
cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
|
||||
zulip-platform-plugins/plugins/platforms/zulip/__init__.py \
|
||||
zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
|
||||
<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
|
||||
|
||||
# Fix ownership (Tanko only)
|
||||
chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/ # Tanko only
|
||||
|
||||
# Clean up
|
||||
rm -rf /tmp/zulip-deploy
|
||||
```
|
||||
|
||||
### Step 3: Verify _strip_html is Present
|
||||
|
||||
```bash
|
||||
grep -q "_strip_html" <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/adapter.py
|
||||
```
|
||||
Expected: exit code 0. If not found → adapter is stale, re-run Step 2 with fresh clone.
|
||||
|
||||
### Step 4: Verify Env Credentials
|
||||
|
||||
```bash
|
||||
grep -E "ZULIP_SITE|ZULIP_EMAIL|ZULIP_API_KEY" <HERMES_HOME>/.env
|
||||
```
|
||||
|
||||
Expected: all three variables set with non-empty values. If any missing:
|
||||
- ZULIP_SITE: `https://chat.sysloggh.net`
|
||||
- ZULIP_EMAIL: `<agent>-bot@chat.sysloggh.net`
|
||||
- ZULIP_API_KEY: obtain from Zulip admin panel (Bots → show API key)
|
||||
|
||||
### Step 5: Restart Gateway
|
||||
|
||||
```bash
|
||||
cd <HERMES_HOME>/hermes-agent
|
||||
# Use venv if available
|
||||
python3 -m hermes_cli.main gateway restart # or: venv/bin/python -m hermes_cli.main gateway restart
|
||||
```
|
||||
|
||||
Wait for the restart to complete (up to 45s). Check:
|
||||
|
||||
```bash
|
||||
grep "Gateway running" <HERMES_HOME>/logs/gateway.log | tail -1
|
||||
```
|
||||
|
||||
Expected: "Gateway running with N platform(s)" where N > 1 (includes zulip).
|
||||
|
||||
### Step 6: Validate Zulip Connection
|
||||
|
||||
```bash
|
||||
python3 -c "
|
||||
import json
|
||||
d = json.load(open('$HERMES_HOME/gateway_state.json'))
|
||||
print('zulip:', d.get('platforms', {}).get('zulip', {}).get('state', 'NOT FOUND'))
|
||||
"
|
||||
```
|
||||
|
||||
Expected: `zulip: connected`. If not connected, check gateway log:
|
||||
|
||||
```bash
|
||||
grep -E "zulip|Zulip|ZULIP" <HERMES_HOME>/logs/gateway.log | tail -10
|
||||
```
|
||||
|
||||
### Step 7: Report
|
||||
|
||||
Compile results: `{ adapter_deployed, zulip_connected, env_valid, gateway_running }`.
|
||||
|
||||
| State | Action |
|
||||
|-------|--------|
|
||||
| All true | ✅ Agent restored — relay success to user |
|
||||
| `adapter_deployed: false` | Re-run Step 2 |
|
||||
| `env_valid: false` | Prompt for missing credentials |
|
||||
| `zulip_connected: false` | Check Zulip server reachability, verify API key |
|
||||
| `gateway_running: false` | Check process logs for crash cause |
|
||||
|
||||
## Known Failure Modes
|
||||
|
||||
| Symptom | Root Cause | Recovery |
|
||||
|---------|-----------|----------|
|
||||
| Gateway running with 1 platform(s) | Adapter at wrong path (user plugins vs bundled) | Deploy to `<hermes-agent>/plugins/platforms/zulip/` not `~/.hermes/plugins/` |
|
||||
| Queue expired / BAD_EVENT_QUEUE_ID | Idle for 10+ minutes → normal | Auto-reconnects — no action needed |
|
||||
| No events received for N seconds | No DMs or @mentions sent to this bot | Normal if nobody messaged the agent |
|
||||
| `httpx` not found | Missing dependency | `pip install httpx` in the Hermes venv or system Python |
|
||||
| Slash commands not matching | Missing `_strip_html` — Zulip sends `<p>/approve</p>` | Verify `_strip_html` in adapter (Step 3) |
|
||||
| Permission denied on gateway restart | Running as wrong user | Use `su - jerome` for Tanko; root for others |
|
||||
|
||||
## Git Branch Reference
|
||||
|
||||
The `_strip_html` fix lives on `feat/zulip-streaming` branch:
|
||||
```
|
||||
https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/zulip-streaming
|
||||
```
|
||||
|
||||
Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
|
||||
Pull request #33 is the primary integration branch.
|
||||
|
||||
---
|
||||
|
||||
**Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: zulip-oidc-redirect-fix
|
||||
status: active
|
||||
description: >
|
||||
Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
|
||||
uses the internal IP (192.168.68.19) instead of the public domain
|
||||
(chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
|
||||
Survives container restarts through compose.override.yaml.
|
||||
agent: abiba
|
||||
triggers:
|
||||
- Zulip OIDC login returns "Redirect URI Error" from Authentik
|
||||
- redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
|
||||
- After Zulip server restart, Authentik SSO login broken
|
||||
---
|
||||
|
||||
## Maintains
|
||||
|
||||
- zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
|
||||
- authentik-acceptance: { status: "accepted" | "rejected" }
|
||||
- patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
|
||||
|
||||
## Detection
|
||||
|
||||
### Rule 1: Wrong redirect_uri host
|
||||
- **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
|
||||
- **Status**: CRITICAL — Authentik will reject
|
||||
- **Trigger fix** → Execute self-heal
|
||||
|
||||
### Rule 2: OIDC flow broken
|
||||
- **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
|
||||
- **Diagnose**: Check redirect_uri in the 302 Location header chain
|
||||
|
||||
## Fix (Self-Heal)
|
||||
|
||||
Two layers applied:
|
||||
|
||||
### Layer 1: Live patch (inside container, immediate)
|
||||
```bash
|
||||
# Add to /home/zulip/deployments/current/zproject/computed_settings.py:
|
||||
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
|
||||
|
||||
import urllib.parse
|
||||
from social_core.strategy import BaseStrategy
|
||||
_original_absolute_uri = BaseStrategy.absolute_uri
|
||||
def _patched_absolute_uri(self, path=None):
|
||||
from django.conf import settings
|
||||
root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
|
||||
if path is not None:
|
||||
return urllib.parse.urljoin(root, path)
|
||||
return root
|
||||
BaseStrategy.absolute_uri = _patched_absolute_uri
|
||||
|
||||
# Restart Django
|
||||
supervisorctl restart zulip-django
|
||||
```
|
||||
|
||||
### Layer 2: Persistent fix (compose.override.yaml)
|
||||
The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
|
||||
`/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
|
||||
|
||||
### Verification
|
||||
```bash
|
||||
STEP1=$(curl -sk -w "%{redirect_url}" \
|
||||
"https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
|
||||
curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
|
||||
# Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
|
||||
# Wrong: redirect_uri=https://192.168.68.19/complete/oidc/
|
||||
```
|
||||
|
||||
### Rollback
|
||||
Remove the patch block from `compose.override.yaml` and restart the container:
|
||||
```bash
|
||||
docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
|
||||
After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
|
||||
Django's `request.build_absolute_uri()` → `request.get_host()`. The upstream
|
||||
Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
|
||||
`Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
|
||||
Django falls back to the server's IP.
|
||||
|
||||
The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
|
||||
`ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
|
||||
Host header.
|
||||
|
||||
## Related Contracts
|
||||
|
||||
- `zulip-health.prose.md` — General Zulip health monitoring
|
||||
- `zulip-self-heal.prose.md` — RETIRED (pi extension removed)
|
||||
Reference in New Issue
Block a user