Merge pull request 'Ship: monitoring contract fixes (2026-08-09)' (#43) from ship/monitoring-contract-fixes into master

Reviewed-on: #43

contract-registry.yaml

@@ -584,7 +584,7 @@ contracts:
       verify: curl -sf https://git.sysloggh.net/api/v1/version
       expect: 200 OK
     - check: SearXNG reachable
-      verify: curl -sf http://192.168.68.17:8080
+      verify: curl -sf http://192.168.68.7:8888
       expect: 200 OK
     artifact: infrastructure health report
   receipt:

cron-prompts-review.md

@@ -356,7 +356,7 @@ Postconditions to verify:
   },
   {
     "check": "SearXNG reachable",
-    "verify": "curl -sf http://192.168.68.17:8080",
+    "verify": "curl -sf http://192.168.68.7:8888",
     "expect": "200 OK"
   }
 ]

hermes-config-template.prose.md

@@ -5,8 +5,7 @@ description: >
   Standard Hermes configuration template for Syslog Solution LLC agents.
   Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
   RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  UPDATED 2026-07-16: Compression model is the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
-  which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
+  UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident.
   Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
   2026-07-16 Mumuni root-cause investigation (WAL #1300).
   UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
@@ -48,7 +47,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
 | Component | Endpoint | Purpose |
 |---|---|---|
 | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
-| SearXNG | `http://storepve:8888` | Privacy-respecting web search |
+| SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search |
 | LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
 | LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
 | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
@@ -101,6 +100,10 @@ model:
   api_key_env: LITELLM_API_KEY    # Injected via infisical run -- wrapper
   max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
   context_length: 131072          # For syslog-auto (all GPUs at 128K for stability).
+                                  # ⚠️ MANDATORY: Hermes probes unknown models from 256K
+                                  # and falls back to 256K when /v1/models lacks a context
+                                  # field (llama-server does). Without this override, agents
+                                  # silently run syslog-auto at 256K (verified 2026-08-09).
                                   # Set 65536 if using gemma-4-12b directly (tight VRAM).
 
 fallback_providers:
@@ -396,6 +399,15 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
 - **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>`
   before and after any config change to catch this and all other rule violations.
 
+### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07)
+- Every MCP server entry must point at the correct endpoint:
+  - ra-h-os = http://192.168.68.65:3100/mcp
+  - litellm = https://litellm.sysloggh.net/mcp
+- MCP entries must carry a REAL key value in the header.
+  - Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP
+    endpoints and result in "Malformed API Key" floods.
+  - Ensure the header value is the actual key (e.g., `sk-...`).
+
 ## Execution
 
 1. **Check current config** — Read the target agent's config.yaml

hermes-key-enforcement.prose.md

@@ -190,10 +190,10 @@ litellm_settings:
 |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
 | Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
 | Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
-| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
-| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
+| Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
+| Koonimo | 113 | .113 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
 | Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
-| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
+| Kagenz0 | 105 | .14 | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
 
 > **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
 > LiteLLM key aliases use agent identity, not CT hostname.

infrastructure-monitoring.prose.md

@@ -7,13 +7,15 @@ description: >
   from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
   via existing /metrics Prometheus endpoint.
   
-  DEPLOYMENT STATUS (2026-07-09):
+  DEPLOYMENT STATUS (2026-08-09):
   ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
-     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
-  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
-     NVIDIA sidecar exporters (.8/.110:9400) never installed.
-     Router falls back to direct GPU /health probes.
-  ⚠️ This contract is target-state aspirational — not as-built.
+     (via proxmox-monitor contract). Grafana at :3001, all scrape targets active.
+  ✅ GPU exporters DEPLOYED: all 3 GPU hosts (.8/.110/.15) run exporters on
+     :9400 (nvidia_gpu_exporter / amdgpu exporter) — verified 200 on 2026-08-09.
+  ✅ LiteLLM /metrics scraping live (success_callback: prometheus; auth via
+     master key) + Alertmanager + Zulip bridge (alerts-infra) added 2026-08-09.
+  ⚠️ This contract is target-state aspirational — but GPU export + alerting
+     are now as-built (verified 2026-08-09).
      As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---

litellm-health.prose.md

@@ -18,6 +18,15 @@ description: >
   Designed as a reusable contract for any Syslog agent.
   
   Source of truth: gpu-fleet.prose.md
+
+  ## Monitoring / Alerting (as-built 2026-08-09)
+
+  - LiteLLM /metrics scrape job: requires `litellm_settings.success_callback: [prometheus]`
+    (failure_callback alone does NOT mount /metrics — verified 2026-08-09).
+    Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/.
+  - Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver
+    firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09.
+  - Prometheus node job covers ALL 6 PVE nodes (.4/.5/.6/.9/.12/.15:9100).
 ---
 
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)

memory-fixer.prose.md

@@ -2,69 +2,175 @@
 kind: pattern
 name: memory-fixer
 description: >
-  Auto-fix low-hanging fruit in the graph. No judgment calls — only deterministic Level 1 operations. 
-  Escalate anything that needs Kwame's input.
-version: 1.1.0
+  Auto-fix low-hanging fruit in the RA-H OS knowledge graph. No judgment calls — only deterministic Level 1 operations.
+  Escalate anything that needs Kwame's input. Executes confirmed Kwame decisions to completion (state + updated_at).
+version: 2.0.0
 ---
 
 # Memory Fixer
 
+> **Canonical copy:** `/root/.hermes/contracts/memory-fixer-v3.md` (used by the `memory-fixer-daily` cron job). This file is the institutional record of the same contract. When the two diverge, treat the v3 source in `/root/.hermes/contracts/` as executable truth.
+
 ## Purpose
-Auto-fix low-hanging fruit in the graph. No judgment calls — only deterministic Level 1 operations. Escalate anything that needs Kwame's input.
+Auto-fix low-hanging fruit in the graph. No judgment calls — only deterministic Level 1 operations. Escalate anything that needs Kwame's input. When Kwame replies to an escalation, **execute the decision to completion** (update state and timestamps), never leaving a node in review-pending forever.
 
-## Level 0 Auto-Deletes (Allowed Without Approval)
-Ephemeral heartbeat and log nodes that violate "Logs NEVER go in the graph":
+**Type:** Write-only (Level 1 fixes only)
+**Scope:** RA-H OS knowledge graph (192.168.68.65)
+**Schedule:** Daily at 8 AM ET
+**Escalation:** Level 2+ to Kwame as task items
 
-- `[LITELLM-HEALTH]`, `[GPU-SELF-HEAL]`, `[PM2-SELF-HEAL]`
-- `[PROXMOX-MONITOR]`, `[GPU-MONITOR]`, `[INFRA-MONITOR]`, `[AGENT-HEALTH]`, `[DISK-GC]`
-- `[WAL]` entries older than 30 days
+## Key Design Decision
 
-**Condition:** node must be an orphan (no edges). Deleting a connected node risks breaking other nodes.
+The `updateNode` tool's `metadata` field performs a **restricted merge** — the `state` key only accepts `'processed'` or `'not_processed'`. Additionally, new metadata keys cannot be added via the merge.
 
-**Method:** direct SQLite on `.65` (MCP has no delete tool):
-```bash
-ssh root@192.168.68.65 "sqlite3 /root/.local/share/RA-H/db/rah.sqlite \"
-DELETE FROM nodes WHERE id IN (
-  SELECT id FROM nodes WHERE id NOT IN (SELECT from_node_id FROM edges)
-  AND id NOT IN (SELECT to_node_id FROM edges)
-  AND title LIKE '[LITELLM-HEALTH]%' -- add more prefixes as needed
-);\""
-```
+**Solution:** Use the `description` field to tag stale nodes with review actions, since `description` is a simple string overwritable via `updateNode`.
 
-## Level 1 Auto-Fixes (No Judgment Required)
+**Tag Format:** `[REVIEW: action] original description text...`
 
-### 1. Missing `type` Field
-For nodes with content but no `metadata.type`:
-- Title contains "Proxmox" or "infrastructure" → `type: infrastructure`
-- Title contains "skill" or "how to" or "guide" → `type: skill`
-- Title contains "doc" or "template" or "brand" → `type: documentation`
-- Title starts with "WAL:" or "TASK:" → `type: note`
-- Title starts with "[LEARN]" → `type: documentation`
-- Otherwise → `type: note` (default)
+Where `action` is one of:
+- `archive` — node is stale and should be archived
+- `refresh` — node is stale and should be refreshed (infrastructure)
+- `keep` — node has been confirmed as current
+- `merge` — node is a duplicate candidate
 
-### 2. Missing `tenant` / `namespace`
-For any node with NULL tenant or namespace:
+**Query for finding review-tagged nodes:**
 ```sql
-UPDATE nodes 
-SET metadata = json_set(
-  COALESCE(metadata, '{}'),
-  '$.tenant', 'syslogsolution',
-  '$.namespace', 'syslogsolution'
-)
-WHERE json_extract(metadata, '$.tenant') IS NULL 
-   OR json_extract(metadata, '$.namespace') IS NULL;
+SELECT id, title, description
+FROM nodes
+WHERE description LIKE '[REVIEW:%';
 ```
 
-### 3. Staleness State Transitions
-Using the type-based windows from the memory-monitor contract:
-- Nodes stale > their window → transition to `state: review_pending`
-- Nodes in `review_pending` for >7 days → escalate to Kwame (Level 2)
+## Level 1 Auto-Fixes (No Kwame Decision Needed)
+
+### 1. Missing `type` Auto-Classification
+
+```sql
+SELECT id, title,
+  CASE
+    WHEN title LIKE '%infrastructure%' OR title LIKE '%proxmox%' OR title LIKE '%setup%' THEN 'infrastructure'
+    WHEN title LIKE '%skill%' OR title LIKE '%how to%' OR title LIKE '%guide%' THEN 'skill'
+    WHEN title LIKE '%doc%' OR title LIKE '%template%' OR title LIKE '%brand%' THEN 'documentation'
+    WHEN title LIKE 'WAL:%' OR title LIKE 'TASK:%' THEN 'note'
+    WHEN title LIKE '%[LEARN]%' THEN 'documentation'
+    ELSE 'note'
+  END as auto_type
+FROM nodes
+WHERE json_extract(metadata, '$.type') IS NULL;
+```
+
+### 2. Missing `namespace` Auto-Population
+
+```sql
+SELECT id, title, json_extract(metadata, '$.tenant') as tenant
+FROM nodes
+WHERE json_extract(metadata, '$.namespace') IS NULL;
+```
+
+### 3. Staleness Review Tagging
+
+Using the type-based windows from the memory-monitor contract, tag nodes stale beyond their window. **Only process a maximum of 10 nodes per run** to avoid overwhelming Kwame. Prioritize infrastructure first, then dynamic, then ephemeral.
+
+**Exclusion Rules:**
+- Nodes with `state` = `review_pending`, `deprecated`, `archived`, or `not_processed` are NOT processed
+- Nodes whose `description` already starts with `[REVIEW:` are NOT re-processed
+
+```sql
+SELECT id, title, json_extract(metadata, '$.type') as node_type,
+  CAST(julianday('now') - julianday(updated_at) AS INTEGER) as days_stale,
+  CASE
+    WHEN json_extract(metadata, '$.type') IN ('infrastructure', 'deployment', 'system', 'system-health') THEN 'refresh'
+    WHEN json_extract(metadata, '$.type') IN ('business', 'philosophy', 'research', 'learning', 'learn', 'investigation', 'analysis', 'project') THEN 'refresh'
+    ELSE 'archive'
+  END as suggested_action
+FROM nodes
+WHERE updated_at < datetime('now',
+  CASE
+    WHEN json_extract(metadata, '$.type') IN ('infrastructure', 'deployment', 'system', 'system-health') THEN '-14 days'
+    WHEN json_extract(metadata, '$.type') IN ('skill', 'documentation', 'template', 'protocol-enforcement', 'prd', 'architecture') THEN '-90 days'
+    WHEN json_extract(metadata, '$.type') IN ('note', 'wal', 'WAL', 'task', 'TASK', 'event') THEN '-30 days'
+    WHEN json_extract(metadata, '$.type') IN ('business', 'philosophy', 'research', 'learning', 'learn', 'investigation', 'analysis', 'project') THEN '-120 days'
+    WHEN json_extract(metadata, '$.type') IN ('deprecated-relay', 'audit', 'audit-report', 'incident', 'incident-report') THEN '-3650 days'
+    ELSE '-45 days'
+  END
+)
+AND json_extract(metadata, '$.state') NOT IN ('review_pending', 'deprecated', 'archived', 'not_processed')
+AND (description IS NULL OR description NOT LIKE '[REVIEW:%')
+ORDER BY days_stale ASC
+LIMIT 10;
+```
+
+For each identified node, call `updateNode(id, { description: "[REVIEW: action] " + originalDescription })`.
 
 ## Level 2 Escalations (Kwame Decision Required)
-1. **Nodes in `review_pending` >7 days** — Archive, refresh, or keep?
-2. **Orphan Nodes >90 days old** — Delete or Connect?
-3. **Potential Duplicate Nodes** — Same title or >70% overlap. Merge or Keep?
-4. **Conflicting Metadata** — Content suggests one tenant but metadata says another.
+
+1. **Stale nodes** flagged with `[REVIEW: …]` — Archive, refresh, or keep?
+2. **Duplicate Nodes** (same title or >70% title overlap) — Merge or keep?
+3. **Orphan Nodes >90 days old** — Archive or connect?
+
+## Reporting Format
+
+The fixer reports to Kwame via this Zulip DM:
+
+```
+🦅 Memory Fixer — [HH:MM UTC]
+
+Level 1 fixes applied:
+- Missing type: X nodes classified
+- Missing namespace: Y nodes populated
+
+Stale nodes needing review (max 10):
+1. [Node #XXX] Title — X days stale, SUGGEST: refresh
+2. [Node #YYY] Title — Y days stale, SUGGEST: archive
+...
+
+Duplicates needing decision:
+1. [Node #AAA] vs [Node #BBB] — Same title
+
+Orphans >90 days:
+1. [Node #EEE] Title — X days stale, orphaned
+
+Reply with:
+- "archive #XXX, #YYY" to mark for archive
+- "archive all" to archive all stale nodes listed
+- "keep #XXX" to confirm a node is current
+- "merge #AAA into #BBB" to merge duplicates
+- "refresh #XXX" to mark as current
+```
+
+## Execution on Next Run
+
+The fixer reads Kwame's previous response and **executes the decision to completion** — it must not leave a node in review-pending forever. Tagging alone is NOT enough; each confirmed decision must also update `state` and `updated_at` so the node drops out of the stale window on the next run.
+
+> ⚠️ `updateNode` cannot set `state` to non-standard values (restricted to `processed`/`not_processed`) and cannot add metadata keys. For state transitions and `updated_at` bumps, use **direct SSH + SQLite** on the bridge host:
+> ```bash
+> ssh root@192.168.68.65 "sqlite3 /root/.local/share/RA-H/db/rah.sqlite \"UPDATE nodes SET metadata = json_set(metadata, '$.state', '<state>'), updated_at = datetime('now') WHERE id = <id>;\""
+> ```
+> Use `updateNode` only for description/source/title/link edits.
+
+Decision → completed action mapping:
+
+| Kwame reply | Description change | State | `updated_at` |
+|---|---|---|---|
+| `archive #XXX` | replace `[REVIEW: archive] ` → `[ARCHIVED] ` prefix | `archived` | bumped to now |
+| `keep #XXX` / `refresh #XXX` | **clear the `[REVIEW: …]` tag entirely** | `active` | bumped to now |
+| `merge #AAA into #BBB` | set `[REVIEW: merge_into #BBB]` on #AAA, then follow manual merge workflow | handled manually | bumped to now |
+| `archive all` | apply the archive row to every node listed in the prior report | `archived` | bumped to now |
+
+**Why `updated_at` must be bumped (critical):** the Level-1 staleness query keys off `updated_at < now - window`. If the fixer clears the tag but leaves a stale `updated_at`, the node is immediately re-flagged on the very next run and the cycle repeats forever. Bumping `updated_at` to now pushes the node back to the front of the window.
+
+**Exclusion after action:** once an action is applied, the node's description no longer starts with `[REVIEW:` (archive → `[ARCHIVED]`, refresh/keep → original text), so it is not re-processed.
+
+After all actions are applied, verify with:
+```sql
+SELECT id, json_extract(metadata, '$.state') FROM nodes WHERE description LIKE '[REVIEW:%';
+```
+The result must be 0 rows when all decisions are executed. Report what was done.
+
+## Checks
+
+- **State integrity:** archived nodes have `state: archived` + `[ARCHIVED]` prefix; kept nodes are `state: active` without a `[REVIEW:]` tag.
+- **No review-pending forever:** after executing Kwame's decisions, `[REVIEW:%` node count must be 0.
+- **Timestamps:** every executed decision bumps `updated_at`, so the node exits the stale window on the next run.
 
 ## Logging
 Every Level 1 fix logged to `~/.hermes/logs/memory-fixer/YYYY-MM-DD.md`