Compare commits

...
Author SHA1 Message Date
root 97977f320c fix: contract improvements from 2026-07-09 run log review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m43s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
- pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs
- disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope,
  remove jitsi (decommissioned), update access matrix with KVM VM section
- infrastructure-control: docker-vm container count 11→16, add Trove agents stack
- infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)
2026-07-09 05:32:00 +00:00
abiba-bot a3786ab289 Merge pull request 'disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB)' (#6) from feat/disk-gc-amdpve-2026-07-09 into master 2026-07-09 05:23:30 +00:00
root 15e8998289 disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-07-09 05:20:42 +00:00
root 25c6c76fb3 disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-07-09 05:17:05 +00:00
root 4819247e39 Add zulip-oidc-redirect-fix prose contract
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
Fixes Zulip OIDC authentication where redirect_uri used internal IP
(192.168.68.19) instead of public domain (chat.sysloggh.net) after
server restart. Monkey-patches social_core.strategy.BaseStrategy
to use ROOT_DOMAIN_URI for all redirect URI construction.
2026-07-09 00:02:18 +00:00
root 1dc4155d9a fix(hermes-zulip-plugin): default to master branch, add gateway restart
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
- Switched default branch from feat/zulip-streaming to master
- Added Step 5: gateway restart with smoke check after plugin install
- Plugin changes require restart to take effect for a valid test
- Outstanding feature branch merges should be addressed post-install
2026-07-08 17:51:19 +00:00
root 09efbdf4e5 feat: add hermes-zulip-plugin (install & repair) and relocate hermes-zulip-restore to repo
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
- hermes-zulip-plugin.prose.md: installs latest zulip-platform plugin from
  canonical repo, verifies _strip_html fix, sends relay success signal.
  Distinct from restore — plugin-layer only, no env/gateway/connection checks.
- hermes-zulip-restore.prose.md: relocated from /root/ into prose-contracts repo.
- Removed duplicate zulip-platform-verification from /root/vault/contracts/.
2026-07-08 17:47:23 +00:00
6 changed files with 422 additions and 29 deletions
+75 -24
View File
@@ -3,10 +3,11 @@ kind: responsibility
name: disk-gc-threat-response name: disk-gc-threat-response
description: > description: >
Recurring disk health scan, garbage collection, and threat response Recurring disk health scan, garbage collection, and threat response
across all 19 Proxmox CTs and Docker hosts. Triggered by incident across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from 2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
Docker image bloat — 5 dangling images, 15 build cache layers. Docker image bloat — 5 dangling images, 15 build cache layers.
Recovered 35.67GB via `docker system prune -a --force`. Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
id: 067NV8KJ03ZG71S44N41F31022 id: 067NV8KJ03ZG71S44N41F31022
version: 1.0.0 version: 1.0.0
--- ---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
## Scope ## Scope
All CTs in the Proxmox cluster, with special attention to Docker hosts: All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
Docker hosts get special attention:
| Host | CT | Disk Risk | GC Strategy | | Host | CT | Disk Risk | GC Strategy |
|------|----|-----------|-------------| |------|----|-----------|-------------|
| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` | | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate | | syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts | | docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune | | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
| All others | — | LOW — no Docker | apt clean, log rotate | | All other CTs | — | LOW — no Docker | apt clean, log rotate |
| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
## Threat Levels ## Threat Levels
@@ -123,7 +130,7 @@ call summary-reporter
## GC Strategies by Host Type ## GC Strategies by Host Type
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109) ### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
```bash ```bash
# Phase 1: Safe prune (won't touch running containers' images) # Phase 1: Safe prune (won't touch running containers' images)
@@ -224,32 +231,76 @@ dangling images and orphaned build cache. No automated GC was in place.
- This contract now runs disk GC fleet-wide every 6 hours - This contract now runs disk GC fleet-wide every 6 hours
- Docker hosts get `docker system prune` on amber, `-a --force` on red - Docker hosts get `docker system prune` on amber, `-a --force` on red
- kagentz is flagged as HIGH risk due to development activity - kagentz is flagged as HIGH risk due to development activity
- amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
``` ```
Done. Here's what we have: ## Incident Log: 2026-07-09 — amdpve docker bloat
## Storage Summary ### Discovery
Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
amdpve (.15) flagged at 78% (AMBER threshold: 75%).
| System | Used | Free | Health | ### Diagnosis
|--------|------|------|--------| - amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green | - Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) | - Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
| **/var/lib/docker** (abiba) | 2.8G | — | Fine | - Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
| **/root/go** (abiba) | 773M | — | Fine | (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
- Not a running service — zero containers, zero active volumes
## New Contract: `disk-gc-threat-response.prose.md` ### Resolution
```
docker system prune -a --force
→ Reclaimed 11.56GB
→ Post: 55G / 94G (62%), 35G free
→ 1 image removed (llama-strix-hip:latest, 15.5GB)
→ 8 build cache layers removed
→ amdpve now GREEN
```
Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does: ### Root Cause
Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
artifacts. Docker on amdpve serves no running purpose — it's only used for
one-off GPU builds. No automated post-migration cleanup was in place.
### Three-in-One ### Preventive Measures
- amdpve added to Docker GC scan list
- Post-migration cleanup step added: after any GPU backend migration, prune
the old backend's Docker images within 24 hours
- Contract now scans GPU bare-metal hosts alongside CTs
- Access via `pct-run` script for all CTs (no hardcoded IPs)
1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline. ## Access Matrix (documented 2026-07-09)
2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum. ### CT Access (via pct-run)
| CT | Name | Node | Status |
|----|------|------|--------|
| 100 | abiba | amdpve | local |
| 102 | adguard | acerpve | ✅ reachable |
| 104 | authentik | minipve | ✅ reachable |
| 105 | kagentz | amdpve | ✅ reachable |
| 106 | ra-h-os | storepve | ✅ reachable |
| 107 | pbs | storepve | ✅ reachable |
| 108 | media | storepve | ✅ reachable |
| 110 | gitea | minipve | ✅ reachable |
| 111 | tdunna | amdpve | ✅ reachable |
| 112 | tanko | amdpve | ✅ reachable |
| 113 | baggy | amdpve | ✅ reachable |
| 114 | mumuni | minipve | ✅ reachable |
| 115 | scottdenya | amdpve | ✅ reachable |
| 116 | syslog-api | minipve | ✅ reachable |
| 117 | zulip | storepve | ✅ reachable |
3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns. ### GPU Bare Metal (via direct SSH)
| Host | IP | GPU | Status |
|------|-----|-----|--------|
| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
### The incident root cause ### KVM VM (via direct SSH)
Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`. | Host | IP | Role | Status |
|------|-----|------|--------|
| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
Want me to push this to the prose-contracts repo? > **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
+237
View File
@@ -0,0 +1,237 @@
---
kind: function
name: hermes-zulip-plugin
description: >
Installs or updates the Zulip platform plugin for any Hermes agent from the
canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
recovery). Verifies the installation, restarts the gateway, and sends a relay
success signal.
agent: abiba
version: 1.0.0
status: active
runtime_contract: 2
---
# Hermes Zulip Plugin — Install & Repair
Single-shot function that pulls the latest zulip-platform plugin from the
canonical git repo, installs it to the correct Hermes bundled plugin path,
verifies the installation, and signals completion.
Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
layer and a targeted gateway restart. It does NOT verify env credentials or
run a live Zulip connection test. Use `hermes-zulip-restore` for full
connectivity recovery including end-to-end DM validation.
## Parameters
| Param | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
| `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
## Maintains
- plugin_installed: bool — Whether all three adapter files exist at the bundled path
- plugin_version: string — Git commit SHA of the installed version
- strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
- signal_sent: bool — Whether relay success message was dispatched
### Postconditions
- All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
- `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
- Installed version matches HEAD of the requested branch
- Relay success signal sent to Hermes agent's inbox
## Requires
- SSH access to target host (direct or via amdpve for CTs)
- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
- Python 3 with `httpx` installed on target
## Live-State Fields
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------|
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Field | Value | Trust |
|-------|-------|-------|
| Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
| Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
| Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
| Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
## Execution
### Step 1: Resolve Target
Map `target` to host, CT ID, hermes_home, and user from the live-state table.
For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
### Step 2: Pull Latest Plugin Source
On the target host:
```bash
# Ensure deploy scratch space
mkdir -p /tmp/zulip-deploy
cd /tmp/zulip-deploy
# Clone or pull
if [ -d zulip-platform-plugins ]; then
cd zulip-platform-plugins
git fetch origin
git checkout {{branch}}
git pull origin {{branch}}
else
git clone --branch {{branch}} \
https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
cd zulip-platform-plugins
fi
# Capture installed version
INSTALLED_SHA=$(git rev-parse HEAD)
echo "Installed SHA: $INSTALLED_SHA"
# Verify we're at HEAD
HEAD_SHA=$(git rev-parse origin/{{branch}})
if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
echo "At latest commit on {{branch}}"
else
echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
fi
```
### Step 3: Install Plugin Files
```bash
# Ensure target directory exists
mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
# Copy adapter files
cp plugins/platforms/zulip/adapter.py \
plugins/platforms/zulip/__init__.py \
plugins/platforms/zulip/plugin.yaml \
{{hermes_home}}/hermes-agent/plugins/platforms/zulip/
# Fix ownership (Tanko only — runs as jerome user)
[ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
{{hermes_home}}/hermes-agent/plugins/platforms/zulip/
echo "Plugin files installed"
```
### Step 4: Verify Installation
```bash
# Check all three files exist
for f in __init__.py adapter.py plugin.yaml; do
if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
echo "$f present"
else
echo "$f MISSING"
exit 1
fi
done
# Verify _strip_html fix
grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
&& echo "✅ _strip_html fix present" \
|| echo "❌ _strip_html MISSING — plugin may be stale"
# Show installed plugin.yaml version
grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
```
### Step 5: Restart Gateway
Plugin changes require a gateway restart to take effect:
```bash
cd {{hermes_home}}/hermes-agent
# Use venv if available
python3 -m hermes_cli.main gateway restart 2>&1 || \
venv/bin/python -m hermes_cli.main gateway restart 2>&1
```
Wait for restart to complete (up to 45s), then confirm:
```bash
grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
```
Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
Quick smoke check — confirm zulip platform loaded:
```bash
grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
```
If gateway fails to restart, check logs for the crash cause before proceeding.
### Step 6: Send Relay Success Signal
On the Abiba host (local), dispatch a relay message to the target agent:
```
ra-h-os-createRelayNode:
title: "Zulip plugin updated — {{target}}"
source: |
Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
_strip_html fix: PRESENT
Timestamp: {{timestamp}}
description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
```
### Step 7: Report
Compile results into a single status block:
| Field | Value |
|-------|-------|
| Target | `{{target}}` |
| Branch | `{{branch}}` |
| Commit SHA | `{{INSTALLED_SHA}}` |
| Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
| `_strip_html` | `{{present|missing}}` |
| Gateway restarted | `{{yes|no}}` |
| Signal sent | `{{yes|no}}` |
## Known Failure Modes
| Symptom | Root Cause | Recovery |
|---------|-----------|----------|
| Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
| Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
| `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
| Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
| Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
## Edge Differences from hermes-zulip-restore
| Concern | hermes-zulip-restore | hermes-zulip-plugin |
|---------|---------------------|---------------------|
| Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
| Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
| Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
| Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
| Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
| Signal dispatch | ❌ None | ✅ Relay message to target |
For full connectivity recovery after a plugin install, chain this contract
with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
---
**Last updated**: 2026-07-08 — Switched default branch to `master`; added
gateway restart step. If outstanding unmerged feature branches exist, address
them in a follow-up merge after this contract completes.
+4 -1
View File
@@ -144,7 +144,7 @@ description: >
### Ecosystem A: docker-vm (192.168.68.7) ### Ecosystem A: docker-vm (192.168.68.7)
11 containers across 4 compose stacks: 16 containers across 4 compose stacks + trove agents:
| Stack | Path | Containers | | Stack | Path | Containers |
|-------|------|-----------| |-------|------|-----------|
@@ -152,6 +152,9 @@ description: >
| **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey | | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse | | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
| **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf | | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
**Last verified:** 2026-07-09 — 16/16 containers running healthy.
### Home Stack Services ### Home Stack Services
+8 -4
View File
@@ -7,10 +7,14 @@ description: >
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
via existing /metrics Prometheus endpoint. via existing /metrics Prometheus endpoint.
STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters) DEPLOYMENT STATUS (2026-07-09):
already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15) ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed. (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
This contract defines the target state; proxmox-monitor is the as-built. ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
NVIDIA sidecar exporters (.8/.110:9400) never installed.
Router falls back to direct GPU /health probes.
⚠️ This contract is target-state aspirational — not as-built.
As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
version: 1.0.0 version: 1.0.0
--- ---
+6
View File
@@ -30,6 +30,12 @@ declare -A CT_NODES=(
# acerpve (192.168.68.9) # acerpve (192.168.68.9)
[102]=acerpve # adguard [102]=acerpve # adguard
# ocupve (192.168.68.5) — no CTs (bare metal GPU .110) # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
#
# REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
# 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090)
# 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070)
# 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH)
# 118 jitsi → stopped, not in service
) )
# Each node must be root-accessible via SSH hostname # Each node must be root-accessible via SSH hostname
+92
View File
@@ -0,0 +1,92 @@
---
kind: responsibility
name: zulip-oidc-redirect-fix
status: active
description: >
Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
uses the internal IP (192.168.68.19) instead of the public domain
(chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
Survives container restarts through compose.override.yaml.
agent: abiba
triggers:
- Zulip OIDC login returns "Redirect URI Error" from Authentik
- redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
- After Zulip server restart, Authentik SSO login broken
---
## Maintains
- zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
- authentik-acceptance: { status: "accepted" | "rejected" }
- patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
## Detection
### Rule 1: Wrong redirect_uri host
- **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
- **Status**: CRITICAL — Authentik will reject
- **Trigger fix** → Execute self-heal
### Rule 2: OIDC flow broken
- **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
- **Diagnose**: Check redirect_uri in the 302 Location header chain
## Fix (Self-Heal)
Two layers applied:
### Layer 1: Live patch (inside container, immediate)
```bash
# Add to /home/zulip/deployments/current/zproject/computed_settings.py:
SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
import urllib.parse
from social_core.strategy import BaseStrategy
_original_absolute_uri = BaseStrategy.absolute_uri
def _patched_absolute_uri(self, path=None):
from django.conf import settings
root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
if path is not None:
return urllib.parse.urljoin(root, path)
return root
BaseStrategy.absolute_uri = _patched_absolute_uri
# Restart Django
supervisorctl restart zulip-django
```
### Layer 2: Persistent fix (compose.override.yaml)
The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
`/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
### Verification
```bash
STEP1=$(curl -sk -w "%{redirect_url}" \
"https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
# Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
# Wrong: redirect_uri=https://192.168.68.19/complete/oidc/
```
### Rollback
Remove the patch block from `compose.override.yaml` and restart the container:
```bash
docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
```
## Root Cause
After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
Django's `request.build_absolute_uri()``request.get_host()`. The upstream
Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
`Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
Django falls back to the server's IP.
The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
`ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
Host header.
## Related Contracts
- `zulip-health.prose.md` — General Zulip health monitoring
- `zulip-self-heal.prose.md` — RETIRED (pi extension removed)