- litellm-self-heal: merged health check as §Health Check section, includes full architecture diagram (v4.0.0), GPU topology, timeout tables, container list, and execution flow. Eliminates duplication between the two contracts. - litellm-health: marked DEPRECATED, retained for reference - README.md: updated with consolidated contract, deprecation notice, disk-gc-threat-response, and infrastructure-monitoring status
208 lines
10 KiB
Markdown
208 lines
10 KiB
Markdown
# Prose Contracts — Syslog Solution LLC
|
|
|
|
Operating contracts for Syslog agents. Each `.prose.md` file defines a **function** (callable helper), **responsibility** (recurring duty with state), or **pattern** (instantiable knowledge/template) that agents can execute via OpenProse or follow manually.
|
|
|
|
## ⚠️ Operating Doctrine — Read Before Acting
|
|
|
|
Two rules govern how agents interact with this repo and the infrastructure it
|
|
describes. Internalize both before running or relying on any contract.
|
|
|
|
### 1. Verify-before-mutate (the `.117` rule)
|
|
|
|
**Contracts are leads to investigate, NOT facts to act on.** Live-state fields
|
|
(IPs, ports, hostnames, credentials, container names, PIDs) drift. Before any
|
|
mutating operation on infrastructure, verify the current state against the live
|
|
system. On drift, halt and ask the user which value is correct — do NOT "fix"
|
|
the live state to match a stale contract.
|
|
|
|
Origin: a contract stated Zulip CT 117 was at `.117`; the live IP was `.19`.
|
|
An agent ran `pct set` without checking `pct config` first and changed the IP
|
|
to the wrong value, breaking Zulip. The staleness was harmless until acted on.
|
|
|
|
**Layer 3 enforcement — the `safe-mutate` wrapper** (deployed on all 5 agents:
|
|
Abiba, Tanko, Mumuni, Koby, Koonimo). ALL infrastructure mutations MUST go
|
|
through `safe-mutate`. Raw `sed -i`, `pct set`, `docker compose up
|
|
--force-recreate`, `kill`, `rm` on infrastructure outside `safe-mutate` is an
|
|
auditable protocol violation. The wrapper runs a verify command, optionally
|
|
checks an `--expect` pattern, refuses on mismatch, and logs every call to
|
|
`/root/.safe-mutate/audit.log`.
|
|
|
|
```
|
|
safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
|
|
safe-mutate --verify "CMD" --dry-run --mutate "CMD" # verify + show, no execute
|
|
safe-mutate --verify "CMD" --verify-only # record a verification only
|
|
```
|
|
|
|
Load the `verify-before-mutate` skill (local pi skill + RA-H shared) for the
|
|
full protocol. See knowledge graph node #604 for the post-mortem.
|
|
|
|
### 2. Field trust levels
|
|
|
|
Not all contract content carries the same trust:
|
|
|
|
| Field type | Trust level | Examples |
|
|
|-----------|-------------|----------|
|
|
| **Policy** | Authoritative — trust the contract | "hardcoded harness keys forbidden", "use api_key_env" |
|
|
| **Live state** | **Never trust — always verify** | IPs, ports, hostnames, credentials, container names, PIDs |
|
|
| **Procedure** | Trust but adapt | command patterns are right, values may be stale |
|
|
|
|
Contracts label live-state fields with `VERIFY-BEFORE-USE`. Treat any such
|
|
field as a hint to confirm against the live system, not a fact to apply.
|
|
|
|
### 3. Verify-before-fix (contract-native)
|
|
|
|
Never `prose run` a fix-contract directly. Run its verifier first, read the
|
|
verdict, lint the fixer, then run:
|
|
|
|
```
|
|
prose run <verifier> # e.g. zulip-platform-verification → verdict
|
|
prose lint <fixer> # validate structure without executing
|
|
prose preflight <fixer> # check deps/env, no execute
|
|
prose run <fixer> # only after verdict + lint pass
|
|
```
|
|
|
|
`kind: responsibility` contracts VERIFY and return a verdict without making
|
|
changes. `kind: function` contracts DO things. Forme `### Requires` →
|
|
`### Maintains` wiring enforces this at the DAG level: a fixer that requires a
|
|
fresh verdict cannot fire until the verifier has run.
|
|
|
|
## How Agents Run These Contracts
|
|
|
|
### Option A: Via OpenProse CLI (preferred)
|
|
|
|
```bash
|
|
prose run <contract-name> [param1=value1 param2=value2 ...]
|
|
```
|
|
|
|
**Examples:**
|
|
|
|
```bash
|
|
# Run a memory audit with default parameters
|
|
prose run memory-audit-maintenance
|
|
|
|
# Run a memory audit with custom threshold
|
|
prose run memory-audit-maintenance memory_threshold=90 verify_configs=true
|
|
|
|
# Configure a new agent with the template
|
|
prose run hermes-config-template agent_name=syslog-devops default_model=claude-sonnet-4
|
|
|
|
# Configure an agent with a different auxiliary model
|
|
prose run hermes-config-template agent_name=syslog-code default_model=qwen3.6-27B-code auxiliary_model=gemma-4-12b
|
|
```
|
|
|
|
### Option B: Manual Execution
|
|
|
|
If `prose` CLI isn't available, any agent can:
|
|
1. Fetch the raw contract: `curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/<filename>`
|
|
2. Read the **Execution** section
|
|
3. Follow the steps
|
|
|
|
**Example for fetching:**
|
|
```bash
|
|
curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/memory-audit-maintenance.prose.md
|
|
```
|
|
|
|
## Available Contracts
|
|
|
|
### Responsibilities (Recurring Duties — `kind: responsibility`)
|
|
|
|
Run on trigger or schedule. Maintain persistent world-model state across runs.
|
|
|
|
| Contract | Domain | Description |
|
|
|---|---|---|
|
|
| `memory-audit-maintenance` | Memory | Audits & reorganizes an agent's native memory (MEMORY.md, USER.md). Categorizes entries, moves rules to skills, verifies configs. |
|
|
| `build-zulip-plugin` | Zulip | Generates and iteratively improves a Hermes Zulip platform plugin. Each run produces a new version. |
|
|
| `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
|
|
| `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
|
|
| `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
|
|
| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
|
|
| `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
|
|
| `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
|
|
| `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
|
|
| `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
|
|
| `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |
|
|
|
|
### Instantiable Templates (`kind: pattern` — run with `prose run`)
|
|
|
|
Generate configs or perform a transformation on demand. No persistent state.
|
|
|
|
| Contract | Description | Key Parameters |
|
|
|---|---|---|
|
|
| `hermes-config-template` | Standard Hermes config for any Syslog agent. Enforces shared infra (Firecrawl, SearXNG, RA-H OS MCP, LiteLLM) + standardized `api_key_env` key indirection. | `agent_name` (required), `default_model`, `auxiliary_model` |
|
|
| `hermes-key-enforcement` | **Enforcement contract** — all harness/LiteLLM providers MUST use `api_key_env`, never hardcoded keys. Includes detection query, rotation procedure, violation response. External providers (DeepSeek, OpenAI) exempt. | (none — doctrine reference) |
|
|
|
|
### Reference Documents (`kind: pattern` — read for context, not run)
|
|
|
|
Encode topology, architectural decisions, and lessons. You read them before planning; you don't `prose run` them.
|
|
|
|
| Contract | Description |
|
|
|---|---|
|
|
| `infrastructure-control` | Full topology and control pattern: 5-node Proxmox cluster, 3 Docker ecosystems, NFS storage, network verification, IP-first configuration doctrine. Live-state fields marked `VERIFY-BEFORE-USE`. |
|
|
| `pi-approval-architecture` | pi's approval model vs Hermes, available commands, architectural constraints. |
|
|
| `zulip-adapter-lessons` | Failure modes, fixes, and patterns from building the pi Zulip extension and Hermes Zulip plugin. |
|
|
|
|
### Functions (`kind: function` — callable helpers, no state)
|
|
|
|
Called on-demand as single-render tools.
|
|
|
|
| Contract | Description |
|
|
|---|---|
|
|
| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
|
|
| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
|
|
| `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
|
|
| `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
|
|
| `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
|
|
|
|
### Scripts (`scripts/`)
|
|
|
|
Companion shell scripts that contracts delegate to.
|
|
|
|
| File | Description |
|
|
|---|---|
|
|
| `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
|
|
| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
|
|
| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |
|
|
|
|
## Contract Structure
|
|
|
|
Kinds determine the section set.
|
|
|
|
**`kind: responsibility`** (recurring duty with persistent state):
|
|
```
|
|
## Maintains — World-model state this contract tracks
|
|
## Parameters — Tunable inputs (with defaults)
|
|
## Requires — External dependencies and preconditions
|
|
## Continuity — When to run (triggers & cadence)
|
|
## Remembers — What to learn between runs
|
|
## Invariants — Inviolable rules for execution
|
|
## Execution — Step-by-step instructions
|
|
## Example Output — What a good run looks like
|
|
```
|
|
|
|
**`kind: function`** (stateless callable helper):
|
|
```
|
|
## Parameters — Tunable inputs (with defaults)
|
|
## Returns — Output schema
|
|
## Requires — External dependencies
|
|
## Execution — Step-by-step instructions
|
|
```
|
|
|
|
**`kind: pattern`** (instantiable knowledge):
|
|
```
|
|
Variable — templates use Parameters + Execution; reference docs are prose-driven.
|
|
```
|
|
|
|
**`kind: gateway`** / **`kind: test`** — not currently used in this repo.
|
|
|
|
## Adding New Contracts
|
|
|
|
1. Create a `.prose.md` file following the structure above
|
|
2. Push to this repo
|
|
3. The contract is immediately available for any agent to `prose run`
|
|
|
|
## Repository
|
|
|
|
- **URL:** https://git.sysloggh.net/SyslogSolution/prose-contracts
|
|
- **Branch:** master
|
|
- **Auth:** mumuni-bot (write access via PAT)
|