Compare commits

...
Author SHA1 Message Date
root 68f9ebff74 Ship: monitoring contract fixes (2026-08-09)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-08-09 21:57:44 +00:00
root 93fb0d8e1e feat(contracts): add MCP URL validation and verify virtual keys on LiteLLM
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 2s
2026-08-07 21:33:59 +00:00
mumuni-bot cb6efa81c3 Merge pull request 'fix(contracts): memory-fixer executes decisions to completion (v2.0.0)' (#42) from fix/memory-fixer-execution-v3 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-08-04 12:32:56 +00:00
mumuni-bot 2dc77ee7da fix(contracts): memory-fixer executes decisions to completion
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Rewrite memory-fixer to v3 semantics: when Kwame replies to an escalation,
execute the action fully - set state (archived/active), clear or rename the
[REVIEW:] tag, and bump updated_at so the node exits the stale window and
is not re-flagged on the next run. Bump version 1.1.0 -> 2.0.0.
2026-08-04 12:30:37 +00:00
jerome 561c4d98c9 Merge pull request 'fix: correct SearXNG endpoint from dead storepve hostname to live VM109 IP' (#40) from fix/searxng-endpoint into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
Reviewed-on: #40
2026-08-02 16:20:49 +00:00
root 8f719ca7c7 fix(contracts): Pulse port 3001→7655 (verified), jdownloader decommissioned from docker-vm → CT 118 LXC (.20)
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Skipped
2026-08-01 20:28:50 +00:00
root c243fcddc3 no-mistakes(document): docs: fix stale SearXNG endpoint in monitoring checks
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-08-01 15:19:15 +00:00
root 50114f32c0 fix: correct SearXNG endpoint from dead storepve hostname to live VM109 IP (192.168.68.7:8888)
storepve (192.168.68.6) has no service on :8888. SearXNG runs on VM 109
(docker-vm) at 192.168.68.7:8888 (verified HTTP 200). Aligns with every other
contract reference. Missed in PR #39.
2026-08-01 14:56:45 +00:00
jerome 4dc59633b3 Merge pull request 'fix(agent-health): vault token fallback + strix-server service name' (#39) from fix/agent-health-check into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
Reviewed-on: #39
2026-08-01 14:20:02 +00:00
9 changed files with 201 additions and 68 deletions
+1 -1
View File
@@ -584,7 +584,7 @@ contracts:
verify: curl -sf https://git.sysloggh.net/api/v1/version verify: curl -sf https://git.sysloggh.net/api/v1/version
expect: 200 OK expect: 200 OK
- check: SearXNG reachable - check: SearXNG reachable
verify: curl -sf http://192.168.68.17:8080 verify: curl -sf http://192.168.68.7:8888
expect: 200 OK expect: 200 OK
artifact: infrastructure health report artifact: infrastructure health report
receipt: receipt:
+1 -1
View File
@@ -356,7 +356,7 @@ Postconditions to verify:
}, },
{ {
"check": "SearXNG reachable", "check": "SearXNG reachable",
"verify": "curl -sf http://192.168.68.17:8080", "verify": "curl -sf http://192.168.68.7:8888",
"expect": "200 OK" "expect": "200 OK"
} }
] ]
+15 -3
View File
@@ -5,8 +5,7 @@ description: >
Standard Hermes configuration template for Syslog Solution LLC agents. Standard Hermes configuration template for Syslog Solution LLC agents.
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
RA-H OS MCP) while keeping agent-specific API keys and model choices. RA-H OS MCP) while keeping agent-specific API keys and model choices.
UPDATED 2026-07-16: Compression model is the stable alias `strix-moe` (NOT `ornith-1.0-35b`, UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident.
which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
2026-07-16 Mumuni root-cause investigation (WAL #1300). 2026-07-16 Mumuni root-cause investigation (WAL #1300).
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13). UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
@@ -48,7 +47,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Component | Endpoint | Purpose | | Component | Endpoint | Purpose |
|---|---|---| |---|---|---|
| Firecrawl | `http://192.168.68.7:3002/` | Web content extraction | | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
| SearXNG | `http://storepve:8888` | Privacy-respecting web search | | SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search |
| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) | | LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) | | LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
| RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge | | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
@@ -101,6 +100,10 @@ model:
api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper
max_tokens: 4096 # ⚠️ CRITICAL: Prevents unbounded generation max_tokens: 4096 # ⚠️ CRITICAL: Prevents unbounded generation
context_length: 131072 # For syslog-auto (all GPUs at 128K for stability). context_length: 131072 # For syslog-auto (all GPUs at 128K for stability).
# ⚠️ MANDATORY: Hermes probes unknown models from 256K
# and falls back to 256K when /v1/models lacks a context
# field (llama-server does). Without this override, agents
# silently run syslog-auto at 256K (verified 2026-08-09).
# Set 65536 if using gemma-4-12b directly (tight VRAM). # Set 65536 if using gemma-4-12b directly (tight VRAM).
fallback_providers: fallback_providers:
@@ -396,6 +399,15 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
- **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>` - **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>`
before and after any config change to catch this and all other rule violations. before and after any config change to catch this and all other rule violations.
### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07)
- Every MCP server entry must point at the correct endpoint:
- ra-h-os = http://192.168.68.65:3100/mcp
- litellm = https://litellm.sysloggh.net/mcp
- MCP entries must carry a REAL key value in the header.
- Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP
endpoints and result in "Malformed API Key" floods.
- Ensure the header value is the actual key (e.g., `sk-...`).
## Execution ## Execution
1. **Check current config** — Read the target agent's config.yaml 1. **Check current config** — Read the target agent's config.yaml
+3 -3
View File
@@ -190,10 +190,10 @@ litellm_settings:
|-------|-----|-----|---------------|------------|--------|-----------------|---------------| |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 | | Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 | | Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 | | Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 | | Koonimo | 113 | .113 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 | | Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 | | Kagenz0 | 105 | .14 | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
> **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo). > **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
> LiteLLM key aliases use agent identity, not CT hostname. > LiteLLM key aliases use agent identity, not CT hostname.
+11 -7
View File
@@ -161,7 +161,7 @@ description: >
|-------|------|-----------| |-------|------|-----------|
| **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis | | **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis |
| **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey | | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse | | **Home stack** | `/opt/home_stack/` | stirling-pdf, pulse (jdownloader decommissioned 2026-08-01 → dedicated CT 118 LXC) |
| **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf | | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats | | **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
@@ -178,11 +178,15 @@ description: >
- Compose: `/opt/home_stack/docker-compose.yml` - Compose: `/opt/home_stack/docker-compose.yml`
- Control script: `/opt/home_stack/infra-control.sh` - Control script: `/opt/home_stack/infra-control.sh`
**JDownloader**: **JDownloader** (decommissioned from docker-vm 2026-08-01 — moved to dedicated CT 118 LXC):
- URL: `http://192.168.68.7:5800` (web UI via VNC) - LXC: CT 118 on storepve, `192.168.68.20` (JDownloader + VNC 5900 + web UI 6080)
- Web UI: `http://192.168.68.20:6080` (noVNC via websockify)
- Docker container `jdownloader-2` on .7 removed; compose entry stripped
**Pulse** (Uptime Kuma): **Pulse**:
- URL: `http://192.168.68.7:3001` - URL: `http://192.168.68.7:7655` (direct LAN)
- Public: `https://pulse.sysloggh.net` (NetBird CNAME proxy)
- Container: `rcourtman/pulse:5.1.35` in `/opt/home_stack` (port 7655, was 3001)
### Ecosystem B: CT 116 syslog-api (192.168.68.116) ### Ecosystem B: CT 116 syslog-api (192.168.68.116)
@@ -368,7 +372,7 @@ fine. Services that resolve directly to a LAN IP are NetBird-independent.
| Authentik | auth.sysloggh.net:443 | 192.168.68.11:9000 | CNAME → netbird | **Yes** | ⚠️ | | Authentik | auth.sysloggh.net:443 | 192.168.68.11:9000 | CNAME → netbird | **Yes** | ⚠️ |
| Gitea | git.sysloggh.net:443 | 192.168.68.17:3000 | CNAME → netbird | **Yes** | ⚠️ | | Gitea | git.sysloggh.net:443 | 192.168.68.17:3000 | CNAME → netbird | **Yes** | ⚠️ |
| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE | | Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ | | Pulse | pulse.sysloggh.net:443 | 192.168.68.7:7655 | CNAME → netbird | **Yes** | ✅ verified 2026-08-01 |
| DNS UI | dns.sysloggh.net:443 | 192.168.68.10:80 | CNAME → netbird | **Yes** | ⚠️ | | DNS UI | dns.sysloggh.net:443 | 192.168.68.10:80 | CNAME → netbird | **Yes** | ⚠️ |
| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ | | SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ | | Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
@@ -606,7 +610,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ | | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ | | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
| 117 | zulip | storepve | .19 | Chat | ❌ | | 117 | zulip | storepve | .19 | Chat | ❌ |
| 118 | jdownloader | storepve | — | JDownloader container | ❌ | | 118 | jdownloader | storepve | .20 | JDownloader LXC (dedicated, migrated from docker-vm 2026-08-01) | ✅ |
| 119 | infisical-vault | minipve | — | Vault | ❌ | | 119 | infisical-vault | minipve | — | Vault | ❌ |
## Appendix C: Docker Compose Files Location ## Appendix C: Docker Compose Files Location
+8 -6
View File
@@ -7,13 +7,15 @@ description: >
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
via existing /metrics Prometheus endpoint. via existing /metrics Prometheus endpoint.
DEPLOYMENT STATUS (2026-07-09): DEPLOYMENT STATUS (2026-08-09):
✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
(via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active. (via proxmox-monitor contract). Grafana at :3001, all scrape targets active.
❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15, ✅ GPU exporters DEPLOYED: all 3 GPU hosts (.8/.110/.15) run exporters on
NVIDIA sidecar exporters (.8/.110:9400) never installed. :9400 (nvidia_gpu_exporter / amdgpu exporter) — verified 200 on 2026-08-09.
Router falls back to direct GPU /health probes. ✅ LiteLLM /metrics scraping live (success_callback: prometheus; auth via
⚠️ This contract is target-state aspirational — not as-built. master key) + Alertmanager + Zulip bridge (alerts-infra) added 2026-08-09.
⚠️ This contract is target-state aspirational — but GPU export + alerting
are now as-built (verified 2026-08-09).
As-built GPU monitoring is via gpu-monitor contract (port 9100 poll). As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
version: 1.0.0 version: 1.0.0
--- ---
+1 -1
View File
@@ -78,7 +78,7 @@ Before ANY update wave:
|------|-------|---------| |------|-------|---------|
| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` | | VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` | | VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` | | VM 109 (.7) | Home stack (Pulse, Stirling PDF) — JDownloader moved to CT 118 LXC 2026-08-01 | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` | | VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` | | CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` | | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
+9
View File
@@ -18,6 +18,15 @@ description: >
Designed as a reusable contract for any Syslog agent. Designed as a reusable contract for any Syslog agent.
Source of truth: gpu-fleet.prose.md Source of truth: gpu-fleet.prose.md
## Monitoring / Alerting (as-built 2026-08-09)
- LiteLLM /metrics scrape job: requires `litellm_settings.success_callback: [prometheus]`
(failure_callback alone does NOT mount /metrics — verified 2026-08-09).
Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/.
- Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver
firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09.
- Prometheus node job covers ALL 6 PVE nodes (.4/.5/.6/.9/.12/.15:9100).
--- ---
## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU) ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
+152 -46
View File
@@ -2,69 +2,175 @@
kind: pattern kind: pattern
name: memory-fixer name: memory-fixer
description: > description: >
Auto-fix low-hanging fruit in the graph. No judgment calls — only deterministic Level 1 operations. Auto-fix low-hanging fruit in the RA-H OS knowledge graph. No judgment calls — only deterministic Level 1 operations.
Escalate anything that needs Kwame's input. Escalate anything that needs Kwame's input. Executes confirmed Kwame decisions to completion (state + updated_at).
version: 1.1.0 version: 2.0.0
--- ---
# Memory Fixer # Memory Fixer
> **Canonical copy:** `/root/.hermes/contracts/memory-fixer-v3.md` (used by the `memory-fixer-daily` cron job). This file is the institutional record of the same contract. When the two diverge, treat the v3 source in `/root/.hermes/contracts/` as executable truth.
## Purpose ## Purpose
Auto-fix low-hanging fruit in the graph. No judgment calls — only deterministic Level 1 operations. Escalate anything that needs Kwame's input. Auto-fix low-hanging fruit in the graph. No judgment calls — only deterministic Level 1 operations. Escalate anything that needs Kwame's input. When Kwame replies to an escalation, **execute the decision to completion** (update state and timestamps), never leaving a node in review-pending forever.
## Level 0 Auto-Deletes (Allowed Without Approval) **Type:** Write-only (Level 1 fixes only)
Ephemeral heartbeat and log nodes that violate "Logs NEVER go in the graph": **Scope:** RA-H OS knowledge graph (192.168.68.65)
**Schedule:** Daily at 8 AM ET
**Escalation:** Level 2+ to Kwame as task items
- `[LITELLM-HEALTH]`, `[GPU-SELF-HEAL]`, `[PM2-SELF-HEAL]` ## Key Design Decision
- `[PROXMOX-MONITOR]`, `[GPU-MONITOR]`, `[INFRA-MONITOR]`, `[AGENT-HEALTH]`, `[DISK-GC]`
- `[WAL]` entries older than 30 days
**Condition:** node must be an orphan (no edges). Deleting a connected node risks breaking other nodes. The `updateNode` tool's `metadata` field performs a **restricted merge** — the `state` key only accepts `'processed'` or `'not_processed'`. Additionally, new metadata keys cannot be added via the merge.
**Method:** direct SQLite on `.65` (MCP has no delete tool): **Solution:** Use the `description` field to tag stale nodes with review actions, since `description` is a simple string overwritable via `updateNode`.
```bash
ssh root@192.168.68.65 "sqlite3 /root/.local/share/RA-H/db/rah.sqlite \"
DELETE FROM nodes WHERE id IN (
SELECT id FROM nodes WHERE id NOT IN (SELECT from_node_id FROM edges)
AND id NOT IN (SELECT to_node_id FROM edges)
AND title LIKE '[LITELLM-HEALTH]%' -- add more prefixes as needed
);\""
```
## Level 1 Auto-Fixes (No Judgment Required) **Tag Format:** `[REVIEW: action] original description text...`
### 1. Missing `type` Field Where `action` is one of:
For nodes with content but no `metadata.type`: - `archive` — node is stale and should be archived
- Title contains "Proxmox" or "infrastructure" → `type: infrastructure` - `refresh` — node is stale and should be refreshed (infrastructure)
- Title contains "skill" or "how to" or "guide" → `type: skill` - `keep` — node has been confirmed as current
- Title contains "doc" or "template" or "brand" → `type: documentation` - `merge` — node is a duplicate candidate
- Title starts with "WAL:" or "TASK:" → `type: note`
- Title starts with "[LEARN]" → `type: documentation`
- Otherwise → `type: note` (default)
### 2. Missing `tenant` / `namespace` **Query for finding review-tagged nodes:**
For any node with NULL tenant or namespace:
```sql ```sql
UPDATE nodes SELECT id, title, description
SET metadata = json_set( FROM nodes
COALESCE(metadata, '{}'), WHERE description LIKE '[REVIEW:%';
'$.tenant', 'syslogsolution',
'$.namespace', 'syslogsolution'
)
WHERE json_extract(metadata, '$.tenant') IS NULL
OR json_extract(metadata, '$.namespace') IS NULL;
``` ```
### 3. Staleness State Transitions ## Level 1 Auto-Fixes (No Kwame Decision Needed)
Using the type-based windows from the memory-monitor contract:
- Nodes stale > their window → transition to `state: review_pending` ### 1. Missing `type` Auto-Classification
- Nodes in `review_pending` for >7 days → escalate to Kwame (Level 2)
```sql
SELECT id, title,
CASE
WHEN title LIKE '%infrastructure%' OR title LIKE '%proxmox%' OR title LIKE '%setup%' THEN 'infrastructure'
WHEN title LIKE '%skill%' OR title LIKE '%how to%' OR title LIKE '%guide%' THEN 'skill'
WHEN title LIKE '%doc%' OR title LIKE '%template%' OR title LIKE '%brand%' THEN 'documentation'
WHEN title LIKE 'WAL:%' OR title LIKE 'TASK:%' THEN 'note'
WHEN title LIKE '%[LEARN]%' THEN 'documentation'
ELSE 'note'
END as auto_type
FROM nodes
WHERE json_extract(metadata, '$.type') IS NULL;
```
### 2. Missing `namespace` Auto-Population
```sql
SELECT id, title, json_extract(metadata, '$.tenant') as tenant
FROM nodes
WHERE json_extract(metadata, '$.namespace') IS NULL;
```
### 3. Staleness Review Tagging
Using the type-based windows from the memory-monitor contract, tag nodes stale beyond their window. **Only process a maximum of 10 nodes per run** to avoid overwhelming Kwame. Prioritize infrastructure first, then dynamic, then ephemeral.
**Exclusion Rules:**
- Nodes with `state` = `review_pending`, `deprecated`, `archived`, or `not_processed` are NOT processed
- Nodes whose `description` already starts with `[REVIEW:` are NOT re-processed
```sql
SELECT id, title, json_extract(metadata, '$.type') as node_type,
CAST(julianday('now') - julianday(updated_at) AS INTEGER) as days_stale,
CASE
WHEN json_extract(metadata, '$.type') IN ('infrastructure', 'deployment', 'system', 'system-health') THEN 'refresh'
WHEN json_extract(metadata, '$.type') IN ('business', 'philosophy', 'research', 'learning', 'learn', 'investigation', 'analysis', 'project') THEN 'refresh'
ELSE 'archive'
END as suggested_action
FROM nodes
WHERE updated_at < datetime('now',
CASE
WHEN json_extract(metadata, '$.type') IN ('infrastructure', 'deployment', 'system', 'system-health') THEN '-14 days'
WHEN json_extract(metadata, '$.type') IN ('skill', 'documentation', 'template', 'protocol-enforcement', 'prd', 'architecture') THEN '-90 days'
WHEN json_extract(metadata, '$.type') IN ('note', 'wal', 'WAL', 'task', 'TASK', 'event') THEN '-30 days'
WHEN json_extract(metadata, '$.type') IN ('business', 'philosophy', 'research', 'learning', 'learn', 'investigation', 'analysis', 'project') THEN '-120 days'
WHEN json_extract(metadata, '$.type') IN ('deprecated-relay', 'audit', 'audit-report', 'incident', 'incident-report') THEN '-3650 days'
ELSE '-45 days'
END
)
AND json_extract(metadata, '$.state') NOT IN ('review_pending', 'deprecated', 'archived', 'not_processed')
AND (description IS NULL OR description NOT LIKE '[REVIEW:%')
ORDER BY days_stale ASC
LIMIT 10;
```
For each identified node, call `updateNode(id, { description: "[REVIEW: action] " + originalDescription })`.
## Level 2 Escalations (Kwame Decision Required) ## Level 2 Escalations (Kwame Decision Required)
1. **Nodes in `review_pending` >7 days** — Archive, refresh, or keep?
2. **Orphan Nodes >90 days old** — Delete or Connect? 1. **Stale nodes** flagged with `[REVIEW: …]` — Archive, refresh, or keep?
3. **Potential Duplicate Nodes** — Same title or >70% overlap. Merge or Keep? 2. **Duplicate Nodes** (same title or >70% title overlap) — Merge or keep?
4. **Conflicting Metadata** — Content suggests one tenant but metadata says another. 3. **Orphan Nodes >90 days old** — Archive or connect?
## Reporting Format
The fixer reports to Kwame via this Zulip DM:
```
🦅 Memory Fixer — [HH:MM UTC]
Level 1 fixes applied:
- Missing type: X nodes classified
- Missing namespace: Y nodes populated
Stale nodes needing review (max 10):
1. [Node #XXX] Title — X days stale, SUGGEST: refresh
2. [Node #YYY] Title — Y days stale, SUGGEST: archive
...
Duplicates needing decision:
1. [Node #AAA] vs [Node #BBB] — Same title
Orphans >90 days:
1. [Node #EEE] Title — X days stale, orphaned
Reply with:
- "archive #XXX, #YYY" to mark for archive
- "archive all" to archive all stale nodes listed
- "keep #XXX" to confirm a node is current
- "merge #AAA into #BBB" to merge duplicates
- "refresh #XXX" to mark as current
```
## Execution on Next Run
The fixer reads Kwame's previous response and **executes the decision to completion** — it must not leave a node in review-pending forever. Tagging alone is NOT enough; each confirmed decision must also update `state` and `updated_at` so the node drops out of the stale window on the next run.
> ⚠️ `updateNode` cannot set `state` to non-standard values (restricted to `processed`/`not_processed`) and cannot add metadata keys. For state transitions and `updated_at` bumps, use **direct SSH + SQLite** on the bridge host:
> ```bash
> ssh root@192.168.68.65 "sqlite3 /root/.local/share/RA-H/db/rah.sqlite \"UPDATE nodes SET metadata = json_set(metadata, '$.state', '<state>'), updated_at = datetime('now') WHERE id = <id>;\""
> ```
> Use `updateNode` only for description/source/title/link edits.
Decision → completed action mapping:
| Kwame reply | Description change | State | `updated_at` |
|---|---|---|---|
| `archive #XXX` | replace `[REVIEW: archive] ` → `[ARCHIVED] ` prefix | `archived` | bumped to now |
| `keep #XXX` / `refresh #XXX` | **clear the `[REVIEW: …]` tag entirely** | `active` | bumped to now |
| `merge #AAA into #BBB` | set `[REVIEW: merge_into #BBB]` on #AAA, then follow manual merge workflow | handled manually | bumped to now |
| `archive all` | apply the archive row to every node listed in the prior report | `archived` | bumped to now |
**Why `updated_at` must be bumped (critical):** the Level-1 staleness query keys off `updated_at < now - window`. If the fixer clears the tag but leaves a stale `updated_at`, the node is immediately re-flagged on the very next run and the cycle repeats forever. Bumping `updated_at` to now pushes the node back to the front of the window.
**Exclusion after action:** once an action is applied, the node's description no longer starts with `[REVIEW:` (archive → `[ARCHIVED]`, refresh/keep → original text), so it is not re-processed.
After all actions are applied, verify with:
```sql
SELECT id, json_extract(metadata, '$.state') FROM nodes WHERE description LIKE '[REVIEW:%';
```
The result must be 0 rows when all decisions are executed. Report what was done.
## Checks
- **State integrity:** archived nodes have `state: archived` + `[ARCHIVED]` prefix; kept nodes are `state: active` without a `[REVIEW:]` tag.
- **No review-pending forever:** after executing Kwame's decisions, `[REVIEW:%` node count must be 0.
- **Timestamps:** every executed decision bumps `updated_at`, so the node exits the stale window on the next run.
## Logging ## Logging
Every Level 1 fix logged to `~/.hermes/logs/memory-fixer/YYYY-MM-DD.md` Every Level 1 fix logged to `~/.hermes/logs/memory-fixer/YYYY-MM-DD.md`