Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8215b84f88 | ||
|
|
b7e23e2592 |
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,612 @@
|
|||||||
|
# Cron Prompts Review — All 10 Scheduled Contracts
|
||||||
|
|
||||||
|
Generated: 2026-07-13 20:59:18 ET
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## hermes-key-enforcement
|
||||||
|
|
||||||
|
**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 6 * * *
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: hermes-key-enforcement
|
||||||
|
|
||||||
|
Category: compliance
|
||||||
|
Domain: hermes-agent
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Daily compliance scan at 6am ET
|
||||||
|
|
||||||
|
This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: hermes-key-enforcement.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/hermes-key-enforcement/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "no plaintext API keys in config",
|
||||||
|
"verify": "grep -rc 'api_key: sk-' /root/.hermes/config.yaml",
|
||||||
|
"expect": "0 matches"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "api_key_env used for harness/litellm providers",
|
||||||
|
"verify": "grep -c 'api_key_env.*LITELLM_API_KEY' /root/.hermes/config.yaml",
|
||||||
|
"expect": "count > 0"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert + pause
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/hermes-key-enforcement/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-key-enforcement, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## hermes-config-template
|
||||||
|
|
||||||
|
**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 4 * * 1
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: hermes-config-template
|
||||||
|
|
||||||
|
Category: compliance
|
||||||
|
Domain: hermes-agent
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Weekly config drift check Monday at 4am ET
|
||||||
|
|
||||||
|
This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: hermes-config-template.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/hermes-config-template/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "agent config template_version matches template file",
|
||||||
|
"verify": "grep -q 'template_version' /root/.hermes/config.yaml && diff <(grep 'template_version' /root/.hermes/config.yaml | cut -d: -f2 | xargs) <(grep 'template_version' /root/prose-contracts/hermes-config-template.prose.md | cut -d: -f2 | xargs) && echo match || echo mismatch",
|
||||||
|
"expect": "match"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "config file is valid YAML",
|
||||||
|
"verify": "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
|
||||||
|
"expect": "valid"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/hermes-config-template/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-config-template, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## hermes-agent-baseline
|
||||||
|
|
||||||
|
**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 5 * * 1
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: hermes-agent-baseline
|
||||||
|
|
||||||
|
Category: compliance
|
||||||
|
Domain: hermes-agent
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Weekly baseline verification Monday at 5am ET
|
||||||
|
|
||||||
|
This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: hermes-agent-baseline.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/hermes-agent-baseline/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "Hermes agent process running",
|
||||||
|
"verify": "pgrep -f 'hermes' > /dev/null && echo running || echo stopped",
|
||||||
|
"expect": "running"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "agent config file exists and valid YAML",
|
||||||
|
"verify": "test -f /root/.hermes/config.yaml && python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
|
||||||
|
"expect": "valid"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "no uncommitted changes in hermes directory",
|
||||||
|
"verify": "cd /root/.hermes && git status --porcelain | wc -l",
|
||||||
|
"expect": "0"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/hermes-agent-baseline/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-agent-baseline, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## proxmox-monitor
|
||||||
|
|
||||||
|
**Category:** monitoring | **Domain:** proxmox | **Owner:** abiba | **Schedule:** */15 * * * *
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: proxmox-monitor
|
||||||
|
|
||||||
|
Category: monitoring
|
||||||
|
Domain: proxmox
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Every 15 minutes
|
||||||
|
|
||||||
|
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: proxmox-monitor.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/proxmox-monitor/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "all Proxmox nodes reachable",
|
||||||
|
"verify": "curl -sf http://192.168.68.10:8006/api2/json/status | jq '.status'",
|
||||||
|
"expect": "healthy"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "no VMs in crashed state",
|
||||||
|
"verify": "pvesh get /nodes -output-format=json | jq '.[] | select(.status==\"Crashed\")'",
|
||||||
|
"expect": "empty"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "backups running on schedule",
|
||||||
|
"verify": "pbs-info --check",
|
||||||
|
"expect": "last_backup < 24h ago"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/proxmox-monitor/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=proxmox-monitor, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## gpu-monitor
|
||||||
|
|
||||||
|
**Category:** monitoring | **Domain:** gpu | **Owner:** abiba | **Schedule:** */15 * * * *
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: gpu-monitor
|
||||||
|
|
||||||
|
Category: monitoring
|
||||||
|
Domain: gpu
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Every 15 minutes — polls all GPU subsystems
|
||||||
|
|
||||||
|
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: gpu-monitor.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/gpu-monitor/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "GPU metrics accessible",
|
||||||
|
"verify": "curl -sf http://localhost:9100/gpu-data",
|
||||||
|
"expect": "200 OK, populated data"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "dashboard serving",
|
||||||
|
"verify": "curl -sf http://localhost:9100/gpu-fleet.html",
|
||||||
|
"expect": "200 OK, HTML returned"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "health endpoint responsive",
|
||||||
|
"verify": "curl -sf http://localhost:9100/health",
|
||||||
|
"expect": "200 OK"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/gpu-monitor/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=gpu-monitor, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## infrastructure-monitoring
|
||||||
|
|
||||||
|
**Category:** monitoring | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** */30 * * * *
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: infrastructure-monitoring
|
||||||
|
|
||||||
|
Category: monitoring
|
||||||
|
Domain: infrastructure
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Every 30 minutes
|
||||||
|
|
||||||
|
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: infrastructure-monitoring.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/infrastructure-monitoring/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "Proxmox API reachable",
|
||||||
|
"verify": "curl -sf http://192.168.68.10:8006/api2/json",
|
||||||
|
"expect": "200 OK"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "Zulip API reachable",
|
||||||
|
"verify": "curl -sf https://chat.sysloggh.net/api/v1/me",
|
||||||
|
"expect": "200 OK"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "LiteLLM proxy reachable",
|
||||||
|
"verify": "curl -sf http://192.168.68.116/litellm/v1/models",
|
||||||
|
"expect": "200 OK"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "Gitea API reachable",
|
||||||
|
"verify": "curl -sf https://git.sysloggh.net/api/v1/version",
|
||||||
|
"expect": "200 OK"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "SearXNG reachable",
|
||||||
|
"verify": "curl -sf http://192.168.68.17:8080",
|
||||||
|
"expect": "200 OK"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/infrastructure-monitoring/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-monitoring, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## zulip-health
|
||||||
|
|
||||||
|
**Category:** monitoring | **Domain:** zulip | **Owner:** abiba | **Schedule:** */15 * * * *
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: zulip-health
|
||||||
|
|
||||||
|
Category: monitoring
|
||||||
|
Domain: zulip
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Every 15 minutes — monitors all Zulip-connected agents
|
||||||
|
|
||||||
|
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: zulip-health.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/zulip-health/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "bot registration active",
|
||||||
|
"verify": "curl -sf https://chat.sysloggh.net/api/v1/me | jq '.user_id'",
|
||||||
|
"expect": "bot_id present"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "DM delivery working",
|
||||||
|
"verify": "curl -sf https://chat.sysloggh.net/api/v1/users/me/is-online",
|
||||||
|
"expect": "online: true"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/zulip-health/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=zulip-health, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## litellm-health
|
||||||
|
|
||||||
|
**Category:** monitoring | **Domain:** litellm | **Owner:** abiba | **Schedule:** */10 * * * *
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: litellm-health
|
||||||
|
|
||||||
|
Category: monitoring
|
||||||
|
Domain: litellm
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Every 10 minutes — LiteLLM proxy health
|
||||||
|
|
||||||
|
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: litellm-health.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/litellm-health/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "LiteLLM proxy reachable",
|
||||||
|
"verify": "curl -sf http://192.168.68.116/litellm/v1/models",
|
||||||
|
"expect": "200 OK, models returned"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "router deprecated, nginx routes work",
|
||||||
|
"verify": "curl -sf https://litellm.sysloggh.net/v1/models",
|
||||||
|
"expect": "200 OK (via nginx)"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/litellm-health/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=litellm-health, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## memory-audit-maintenance
|
||||||
|
|
||||||
|
**Category:** maintenance | **Domain:** memory | **Owner:** mumuni | **Schedule:** 0 3 * * *
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: memory-audit-maintenance
|
||||||
|
|
||||||
|
Category: maintenance
|
||||||
|
Domain: memory
|
||||||
|
Owner: mumuni
|
||||||
|
Schedule: Daily at 3am ET
|
||||||
|
|
||||||
|
This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: memory-audit-maintenance.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/memory-audit-maintenance/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "memory files below 80% capacity",
|
||||||
|
"verify": "wc -l ~/.hermes/memories/*.md",
|
||||||
|
"expect": "total lines < threshold"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "no stale entries",
|
||||||
|
"verify": "grep -r 'STALE' ~/.hermes/memories/",
|
||||||
|
"expect": "0 matches"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify mumuni → action: relay_alert
|
||||||
|
- CRITICAL: notify mumuni, abiba → action: relay_alert
|
||||||
|
- FATAL: notify mumuni, abiba, kwame → action: relay_alert + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 3 per 60.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/memory-audit-maintenance/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=memory-audit-maintenance, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## infrastructure-update
|
||||||
|
|
||||||
|
**Category:** maintenance | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** 0 2 * * 0
|
||||||
|
|
||||||
|
```
|
||||||
|
Contract Enforcement: infrastructure-update
|
||||||
|
|
||||||
|
Category: maintenance
|
||||||
|
Domain: infrastructure
|
||||||
|
Owner: abiba
|
||||||
|
Schedule: Weekly system updates Sunday at 2am ET
|
||||||
|
|
||||||
|
This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Load contract from prose-contracts/main (file: infrastructure-update.prose.md)
|
||||||
|
2. Verify prerequisites (connectivity, tools, deps)
|
||||||
|
3. Execute contract per SOP
|
||||||
|
4. Run postconditions from contract registry
|
||||||
|
5. Generate receipt with status (pass/fail/escalated)
|
||||||
|
6. If any postcondition fails, escalate per contract escalation tiers
|
||||||
|
7. Log to ~/.hermes/runs/infrastructure-update/
|
||||||
|
|
||||||
|
Postconditions to verify:
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"check": "all services running after update",
|
||||||
|
"verify": "systemctl list-units --state=running",
|
||||||
|
"expect": "all critical services"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
Escalation Tiers:
|
||||||
|
- INFO: notify nobody → action: log_to_receipt
|
||||||
|
- WARNING: notify abiba → action: relay_alert
|
||||||
|
- CRITICAL: notify abiba, mumuni → action: relay_alert
|
||||||
|
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
|
||||||
|
|
||||||
|
Circuit Breaker:
|
||||||
|
- Max retries: 1 per 120.0min window
|
||||||
|
- On trip: escalate_to_fatal
|
||||||
|
|
||||||
|
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||||
|
Receipt storage: ~/.hermes/runs/infrastructure-update/receipt-{timestamp}.json
|
||||||
|
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-update, status=<status>
|
||||||
|
|
||||||
|
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||||
|
|
||||||
|
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
*End of review*
|
||||||
@@ -1,256 +0,0 @@
|
|||||||
---
|
|
||||||
kind: pattern
|
|
||||||
name: delegation-prose-contract
|
|
||||||
description: >
|
|
||||||
Manager (Mumuni) operating doctrine for task decomposition, worker
|
|
||||||
delegation, verification, and delivery. Defines when to delegate, which
|
|
||||||
worker to use for what, how to handle failures, and the kanban board
|
|
||||||
protocol. Enforces context-window discipline and separation of concerns.
|
|
||||||
Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
|
|
||||||
version: 1.0.0
|
|
||||||
---
|
|
||||||
|
|
||||||
## Maintains
|
|
||||||
|
|
||||||
- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
|
|
||||||
`syslog-research`, `syslog-review`, `syslog-writer`)
|
|
||||||
- Kanban board state at `~/.hermes/kanban/kanban.json`
|
|
||||||
- Context window budget: ~65K tokens per request (131K total, 60% threshold)
|
|
||||||
|
|
||||||
## Topology
|
|
||||||
|
|
||||||
**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
|
|
||||||
**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
|
|
||||||
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
|
|
||||||
|
|
||||||
This contract is infrastructure-agnostic in terms of which nodes are used.
|
|
||||||
Workers execute tasks on whatever infrastructure they're given — SSH to .6,
|
|
||||||
.pm, .9, .12, or .15 depending on the task. The contract defines the
|
|
||||||
**who** and **when** — not the **where**.
|
|
||||||
|
|
||||||
## Why This Matters
|
|
||||||
|
|
||||||
Without enforced delegation, the manager consumes the full iteration budget
|
|
||||||
(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
|
|
||||||
leaving no capacity for actual coordination. The result: context overflow
|
|
||||||
(59K tokens in system prompt), iteration exhaustion, and degraded response
|
|
||||||
quality. This contract exists because I blew through my budget checking
|
|
||||||
Proxmox node status instead of delegating to `syslog-devops`.
|
|
||||||
|
|
||||||
## Context Window Discipline
|
|
||||||
|
|
||||||
**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
|
|
||||||
**Volatile (MEMORY.md + USER.md): ~300 tokens.**
|
|
||||||
**Total base: ~6,800 tokens per request.**
|
|
||||||
|
|
||||||
The remaining budget is the conversation. Every tool call result adds to it.
|
|
||||||
If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
|
|
||||||
from multiple nodes), the context fills fast. That's why we delegate: workers
|
|
||||||
process in isolation and return compact results.
|
|
||||||
|
|
||||||
## Trigger Conditions
|
|
||||||
|
|
||||||
Delegation is **mandatory** when any of these apply:
|
|
||||||
|
|
||||||
| Condition | Threshold | Example |
|
|
||||||
|-----------|-----------|---------|
|
|
||||||
| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
|
|
||||||
| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
|
|
||||||
| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
|
|
||||||
| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
|
|
||||||
| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
|
|
||||||
| Code builds or changes | Writing or modifying code | Scripts, configs, patches |
|
|
||||||
| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
|
|
||||||
|
|
||||||
**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
|
|
||||||
`curl`, `hermes tools list` — these are decision-making tools. The manager
|
|
||||||
reads them directly.
|
|
||||||
|
|
||||||
## Worker Selection Matrix
|
|
||||||
|
|
||||||
| Worker | Model | Toolsets | Role | Use When |
|
|
||||||
|--------|-------|----------|------|----------|
|
|
||||||
| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
|
|
||||||
| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
|
|
||||||
| `syslog-email` | strix-moe | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
|
|
||||||
| `syslog-research` | strix-moe | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
|
|
||||||
| `syslog-review` | strix-moe | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
|
|
||||||
| `syslog-writer` | strix-moe | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
|
|
||||||
|
|
||||||
### Selection Rules
|
|
||||||
|
|
||||||
1. **Match specialty first.** A code task → `syslog-code`. An infra task →
|
|
||||||
`syslog-devops`. Don't put a `syslog-email` worker on a code review.
|
|
||||||
2. **Research tasks with browser needs → `syslog-research`.** Other workers
|
|
||||||
don't have the browser toolset.
|
|
||||||
3. **Verification → `syslog-review`.** Never deliver raw worker output.
|
|
||||||
4. **Documentation/content → `syslog-writer`.** Let them own the prose.
|
|
||||||
5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
|
|
||||||
(includes browser) and high reasoning effort.
|
|
||||||
|
|
||||||
## Delegation Protocol
|
|
||||||
|
|
||||||
### Step 1: Decompose
|
|
||||||
|
|
||||||
Break the task into lanes. Each lane does ONE thing. Workers are independent —
|
|
||||||
no lane depends on another's output mid-flight. If lanes depend on each other,
|
|
||||||
dispatch sequentially.
|
|
||||||
|
|
||||||
### Step 2: Dispatch
|
|
||||||
|
|
||||||
Fire workers via `delegate_task`:
|
|
||||||
|
|
||||||
**Parallel (independent lanes):**
|
|
||||||
```
|
|
||||||
delegate_task(
|
|
||||||
tasks=[
|
|
||||||
{"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
|
|
||||||
{"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
|
|
||||||
]
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Sequential (dependent lanes):**
|
|
||||||
Dispatch lane 1 → wait for result → dispatch lane 2.
|
|
||||||
|
|
||||||
### Step 3: Verify
|
|
||||||
|
|
||||||
**MANDATORY for:**
|
|
||||||
- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
|
|
||||||
- Code builds and modifications
|
|
||||||
- Research findings (web data, external sources)
|
|
||||||
- Any output that will reach the user
|
|
||||||
|
|
||||||
**Fire `syslog-review` to verify:**
|
|
||||||
```
|
|
||||||
delegate_task(
|
|
||||||
goal="Review the output of the devops worker. Verify the node status
|
|
||||||
report is accurate, check for inconsistencies, confirm all nodes were
|
|
||||||
reachable.",
|
|
||||||
context="Worker was syslog-devops. Output is at /tmp/node-report.md.
|
|
||||||
Verify against live system."
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
**If verification fails:**
|
|
||||||
1. Send work back to original worker with review feedback
|
|
||||||
2. Re-verify
|
|
||||||
3. Max 2 re-verify cycles before escalating to Kwame
|
|
||||||
|
|
||||||
### Step 4: Deliver
|
|
||||||
|
|
||||||
Only verified results reach Kwame. Format per channel:
|
|
||||||
- Telegram: Use `telegram-formatting` skill
|
|
||||||
- Zulip: Use Zulip Markdown (CommonMark)
|
|
||||||
- Email: Use `syslog-email` skill
|
|
||||||
|
|
||||||
## Kanban Board Protocol
|
|
||||||
|
|
||||||
**File:** `~/.hermes/kanban/kanban.json`
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"task_id": "unique-id",
|
|
||||||
"title": "Task description",
|
|
||||||
"created": "2026-07-09T01:00:00",
|
|
||||||
"status": "backlog|in_progress|review|done",
|
|
||||||
"lanes": [
|
|
||||||
{
|
|
||||||
"lane_id": "devops-check",
|
|
||||||
"worker": "syslog-devops",
|
|
||||||
"goal": "Check all 5 Proxmox nodes",
|
|
||||||
"status": "dispatched|completed|failed",
|
|
||||||
"output_file": "/tmp/node-report.md"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Update the board on every state change.**
|
|
||||||
|
|
||||||
## Failure Handling
|
|
||||||
|
|
||||||
### Worker Timeouts
|
|
||||||
|
|
||||||
- Child timeout: **900 seconds** (15 minutes)
|
|
||||||
- Worker model `syslog-auto` is slow — it can hit the timeout limit with
|
|
||||||
22+ API calls
|
|
||||||
- **If a worker times out:** Re-dispatch with a narrower scope. Break the
|
|
||||||
task into smaller pieces that fit in the timeout window.
|
|
||||||
- **Avoid delegating sequential SSH hops** — each SSH connection adds latency
|
|
||||||
that compounds quickly. Prefer API-based or local approaches when possible.
|
|
||||||
|
|
||||||
### Worker Selection Failures
|
|
||||||
|
|
||||||
- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
|
|
||||||
- `syslog-code` is best for code-level work (reading files, writing scripts)
|
|
||||||
- `syslog-research` has the browser toolset — use for web research
|
|
||||||
- `syslog-review` is the QA gate — always fire before delivery
|
|
||||||
- **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
|
|
||||||
- **Never nest delegation** (max_spawn_depth: 1)
|
|
||||||
|
|
||||||
### Context Overflow
|
|
||||||
|
|
||||||
- If a task requires >10K tokens of output, delegate the processing
|
|
||||||
- Workers return compact summaries, not raw data dumps
|
|
||||||
- Pass file paths and concrete goals — never dump raw data into context
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- ❌ Reading large files into your own context before deciding → delegate the read
|
|
||||||
- ❌ Carrying SSH/grep/output results in your context → delegate the analysis
|
|
||||||
- ❌ Doing work yourself and then "pretending" to delegate → the user can tell
|
|
||||||
- ❌ Skipping verification → raw worker output never reaches the user
|
|
||||||
- ❌ Delegating single tool calls → keep quick reads/writes at manager level
|
|
||||||
- ❌ Firing more than 3 workers in parallel → hard limit
|
|
||||||
|
|
||||||
## Emergency Exception
|
|
||||||
|
|
||||||
**In an emergency (server down, service must be restored immediately):**
|
|
||||||
- Delegate the diagnosis (find the problem)
|
|
||||||
- Execute the fix yourself (minimize handoff latency)
|
|
||||||
- Verify the fix after delivery
|
|
||||||
- Log the exception in the kanban board
|
|
||||||
|
|
||||||
The emergency exception exists because the user needs the service back NOW,
|
|
||||||
not after three worker round-trips. But it's an exception — not the rule.
|
|
||||||
|
|
||||||
## What This Contract Doesn't Cover
|
|
||||||
|
|
||||||
1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
|
|
||||||
2. **SSH key management** — covered by existing SSH/Proxmox contracts
|
|
||||||
3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
|
|
||||||
4. **Cron job management** — covered by individual cron contracts
|
|
||||||
5. **Infra verification** — covered by `verify-before-mutate` protocol
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
|
|
||||||
```
|
|
||||||
|
|
||||||
## References
|
|
||||||
|
|
||||||
- `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
|
|
||||||
- `worker-profile-audit.md` (skill reference): Worker configuration audit notes
|
|
||||||
- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
|
|
||||||
- `verify-before-mutate` protocol: Infrastructure change verification
|
|
||||||
- `hermes-config-template.prose.md`: Worker profile configuration
|
|
||||||
|
|
||||||
## Success Criteria
|
|
||||||
|
|
||||||
This contract succeeds when:
|
|
||||||
|
|
||||||
1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
|
|
||||||
2. **Workers do the work** — manager coordinates, doesn't execute
|
|
||||||
3. **Verification before delivery** — all output passes through `syslog-review`
|
|
||||||
4. **Kanban board is current** — every task has a lane, every lane has a status
|
|
||||||
5. **User gets verified results** — raw worker output never reaches Kwame
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Last updated:** 2026-07-09
|
|
||||||
**Author:** Mumuni (with Kwame's input on triggers and exception criteria)
|
|
||||||
**Status:** Draft — awaiting PR review and merge to prose-contracts main
|
|
||||||
+3
-3
@@ -7,7 +7,7 @@ description: >
|
|||||||
saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
|
saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
|
||||||
UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe,
|
UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe,
|
||||||
gpu-dense, gpu-light. These never change — only the underlying model does.
|
gpu-dense, gpu-light. These never change — only the underlying model does.
|
||||||
Strix Halo: ornith-1.0-35b → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
|
Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
|
||||||
RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
|
RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
|
||||||
RTX 5070 context: 131K → 256K. VRAM: 88% (10.8/12.2GB).
|
RTX 5070 context: 131K → 256K. VRAM: 88% (10.8/12.2GB).
|
||||||
Compression timeout: 300s (was 120s). Mumuni context: 128K (was 256K).
|
Compression timeout: 300s (was 120s). Mumuni context: 128K (was 256K).
|
||||||
@@ -186,7 +186,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
|
|||||||
3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
|
3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
|
||||||
4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
|
4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
|
||||||
5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
|
5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
|
||||||
- gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (ornith-1.0-35b does NOT exist — legacy name, do not use)
|
- gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (strix-moe does NOT exist — legacy name, do not use)
|
||||||
- global request_timeout: 300s, nginx proxy_read_timeout: 600s
|
- global request_timeout: 300s, nginx proxy_read_timeout: 600s
|
||||||
6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
|
6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
|
||||||
7. Check port conflicts: verify only one llama-server on :8080 per host
|
7. Check port conflicts: verify only one llama-server on :8080 per host
|
||||||
@@ -314,7 +314,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
|
|||||||
| `personalities` | `creative` | Creative assistant personality |
|
| `personalities` | `creative` | Creative assistant personality |
|
||||||
| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
|
| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
|
||||||
| Main model timeout | 300s | LiteLLM global timeout |
|
| Main model timeout | 300s | LiteLLM global timeout |
|
||||||
| Compression model timeout | 300s | ornith timeout increased from 120s |
|
| Compression model timeout | 300s | strix-moe timeout increased from 120s |
|
||||||
|
|
||||||
### Agent Update Status (2026-07-15)
|
### Agent Update Status (2026-07-15)
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ version: 1.0.0
|
|||||||
description: >
|
description: >
|
||||||
Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
|
Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
|
||||||
configuration state, keys, workarounds, and audit procedure. When an agent's
|
configuration state, keys, workarounds, and audit procedure. When an agent's
|
||||||
configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
|
configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
|
||||||
author: Abiba (pi agent)
|
author: Abiba (pi agent)
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -26,11 +26,11 @@ done
|
|||||||
|-------|-----|------|-----|---------------|------------|----------|
|
|-------|-----|------|-----|---------------|------------|----------|
|
||||||
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
||||||
| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
|
| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
|
||||||
| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||||
| Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
||||||
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes |
|
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
||||||
|
|
||||||
> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
|
> **Note**: CT hostnames (tdunna→CT129, baggy→CT114) differ from agent identities (koby, koonimo).
|
||||||
|
|
||||||
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
|
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
|
||||||
Keys are stored in Infisical vault (project=agents, env=production) and injected at
|
Keys are stored in Infisical vault (project=agents, env=production) and injected at
|
||||||
@@ -169,9 +169,9 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
|
|||||||
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
|
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
|
||||||
```
|
```
|
||||||
|
|
||||||
### For Koby (CT 111 / tdunna)
|
### For Koby (CT 129 / tdunna)
|
||||||
|
|
||||||
Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
|
Koby runs Hermes on CT 129 (tdunna). Config files at `/root/.hermes/config.yaml`.
|
||||||
Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
|
Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
|
||||||
|
|
||||||
**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
|
**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
|
||||||
|
|||||||
@@ -122,7 +122,11 @@ runs as user `jerome`, not systemd root, needs user-scope adaptation).
|
|||||||
| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram |
|
| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram |
|
||||||
| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
|
| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
|
||||||
> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent.
|
> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent.
|
||||||
|
<<<<<<< HEAD
|
||||||
|
| tanko | .122 | infisical-gateway.sh + service token (migrated 2026-07-16) | TANKO_LITELLM_API_KEY | ✅ vault-backed |
|
||||||
|
=======
|
||||||
| tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |
|
| tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |
|
||||||
|
>>>>>>> origin/master
|
||||||
|
|
||||||
### Tanko migration (pending)
|
### Tanko migration (pending)
|
||||||
|
|
||||||
@@ -183,6 +187,21 @@ TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth
|
|||||||
-d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
|
-d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
|
||||||
# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
|
# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
|
||||||
```
|
```
|
||||||
|
<<<<<<< HEAD
|
||||||
|
**2026-07-16 UPDATE — vault SYNCED + CLEANED.** The abiba service token (`st.8e848433…`, READ+WRITE)
|
||||||
|
can write to the vault. All session-13 rotated keys in vault and validate 200 against LiteLLM.
|
||||||
|
4 stale secrets deprecated, 5 personal creds flagged for separate project. Tanko migrated from
|
||||||
|
hardcoded keys to `infisical-gateway.sh` wrapper. Koonimo Zulip key restored.
|
||||||
|
|
||||||
|
**Machine Identity:** `8ddb9438-74fc-4ab6-bd74-929e7c47b53b` exists in Infisical UI.
|
||||||
|
Client secret needed to use universal auth for vault writes from automation.
|
||||||
|
|
||||||
|
**Service Token Inventory (2026-07-16):**
|
||||||
|
| Token ID | Name | Permissions | Used By |
|
||||||
|
|----------|------|-------------|---------|
|
||||||
|
| `st.8e848433…` | tanko-gateway | READ+WRITE | Abiba, Koonimo |
|
||||||
|
| `st.353699cd…` | tanko-agent | READ-only | Tanko |
|
||||||
|
=======
|
||||||
Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings →
|
Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings →
|
||||||
Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity
|
Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity
|
||||||
and store its credentials in the vault itself (or a root-only file).**
|
and store its credentials in the vault itself (or a root-only file).**
|
||||||
@@ -197,11 +216,18 @@ koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `
|
|||||||
`KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again.
|
`KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again.
|
||||||
Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix
|
Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix
|
||||||
so the shared service token isn't reused across hosts — but it is no longer blocking.
|
so the shared service token isn't reused across hosts — but it is no longer blocking.
|
||||||
|
>>>>>>> origin/master
|
||||||
|
|
||||||
## Key Rotation Log
|
## Key Rotation Log
|
||||||
|
|
||||||
| Date | Agent | Action | Notes |
|
| Date | Agent | Action | Notes |
|
||||||
|------|-------|--------|-------|
|
|------|-------|--------|-------|
|
||||||
|
<<<<<<< HEAD
|
||||||
|
| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault (Tt2bUL…). Updated wrapper to inject ZULIP_API_KEY + ZULIP_EMAIL. Restarted gateway → Zulip connected as koonimo-bot@ (bot_id=17). 3 platforms now. |
|
||||||
|
| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml key to infisical-gateway.sh wrapper + service token st.353699cd… (tanko-agent). Systemd user service updated, hardcoded key drop-ins removed. Verified LITELLM_API_KEY from vault, 3 platforms connected. |
|
||||||
|
| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated: KAGENZ0_LITELLM_API_KEY, HERMES_OPENROUTER_KEY, LITELLM_API_KEY (generic duplicate), ZULIP_API_KEY (generic duplicate). 5 personal creds flagged for separate project. |
|
||||||
|
=======
|
||||||
|
>>>>>>> origin/master
|
||||||
| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
|
| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
|
||||||
| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
|
| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
|
||||||
| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
|
| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
|
||||||
|
|||||||
Reference in New Issue
Block a user