Vault cleanup + contract sync (WAL #1316) #19
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,612 @@
|
||||
# Cron Prompts Review — All 10 Scheduled Contracts
|
||||
|
||||
Generated: 2026-07-13 20:59:18 ET
|
||||
|
||||
---
|
||||
|
||||
## hermes-key-enforcement
|
||||
|
||||
**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 6 * * *
|
||||
|
||||
```
|
||||
Contract Enforcement: hermes-key-enforcement
|
||||
|
||||
Category: compliance
|
||||
Domain: hermes-agent
|
||||
Owner: abiba
|
||||
Schedule: Daily compliance scan at 6am ET
|
||||
|
||||
This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: hermes-key-enforcement.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/hermes-key-enforcement/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "no plaintext API keys in config",
|
||||
"verify": "grep -rc 'api_key: sk-' /root/.hermes/config.yaml",
|
||||
"expect": "0 matches"
|
||||
},
|
||||
{
|
||||
"check": "api_key_env used for harness/litellm providers",
|
||||
"verify": "grep -c 'api_key_env.*LITELLM_API_KEY' /root/.hermes/config.yaml",
|
||||
"expect": "count > 0"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert + pause
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/hermes-key-enforcement/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-key-enforcement, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## hermes-config-template
|
||||
|
||||
**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 4 * * 1
|
||||
|
||||
```
|
||||
Contract Enforcement: hermes-config-template
|
||||
|
||||
Category: compliance
|
||||
Domain: hermes-agent
|
||||
Owner: abiba
|
||||
Schedule: Weekly config drift check Monday at 4am ET
|
||||
|
||||
This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: hermes-config-template.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/hermes-config-template/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "agent config template_version matches template file",
|
||||
"verify": "grep -q 'template_version' /root/.hermes/config.yaml && diff <(grep 'template_version' /root/.hermes/config.yaml | cut -d: -f2 | xargs) <(grep 'template_version' /root/prose-contracts/hermes-config-template.prose.md | cut -d: -f2 | xargs) && echo match || echo mismatch",
|
||||
"expect": "match"
|
||||
},
|
||||
{
|
||||
"check": "config file is valid YAML",
|
||||
"verify": "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
|
||||
"expect": "valid"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/hermes-config-template/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-config-template, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## hermes-agent-baseline
|
||||
|
||||
**Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 5 * * 1
|
||||
|
||||
```
|
||||
Contract Enforcement: hermes-agent-baseline
|
||||
|
||||
Category: compliance
|
||||
Domain: hermes-agent
|
||||
Owner: abiba
|
||||
Schedule: Weekly baseline verification Monday at 5am ET
|
||||
|
||||
This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: hermes-agent-baseline.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/hermes-agent-baseline/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "Hermes agent process running",
|
||||
"verify": "pgrep -f 'hermes' > /dev/null && echo running || echo stopped",
|
||||
"expect": "running"
|
||||
},
|
||||
{
|
||||
"check": "agent config file exists and valid YAML",
|
||||
"verify": "test -f /root/.hermes/config.yaml && python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
|
||||
"expect": "valid"
|
||||
},
|
||||
{
|
||||
"check": "no uncommitted changes in hermes directory",
|
||||
"verify": "cd /root/.hermes && git status --porcelain | wc -l",
|
||||
"expect": "0"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/hermes-agent-baseline/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-agent-baseline, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## proxmox-monitor
|
||||
|
||||
**Category:** monitoring | **Domain:** proxmox | **Owner:** abiba | **Schedule:** */15 * * * *
|
||||
|
||||
```
|
||||
Contract Enforcement: proxmox-monitor
|
||||
|
||||
Category: monitoring
|
||||
Domain: proxmox
|
||||
Owner: abiba
|
||||
Schedule: Every 15 minutes
|
||||
|
||||
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: proxmox-monitor.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/proxmox-monitor/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "all Proxmox nodes reachable",
|
||||
"verify": "curl -sf http://192.168.68.10:8006/api2/json/status | jq '.status'",
|
||||
"expect": "healthy"
|
||||
},
|
||||
{
|
||||
"check": "no VMs in crashed state",
|
||||
"verify": "pvesh get /nodes -output-format=json | jq '.[] | select(.status==\"Crashed\")'",
|
||||
"expect": "empty"
|
||||
},
|
||||
{
|
||||
"check": "backups running on schedule",
|
||||
"verify": "pbs-info --check",
|
||||
"expect": "last_backup < 24h ago"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/proxmox-monitor/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=proxmox-monitor, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## gpu-monitor
|
||||
|
||||
**Category:** monitoring | **Domain:** gpu | **Owner:** abiba | **Schedule:** */15 * * * *
|
||||
|
||||
```
|
||||
Contract Enforcement: gpu-monitor
|
||||
|
||||
Category: monitoring
|
||||
Domain: gpu
|
||||
Owner: abiba
|
||||
Schedule: Every 15 minutes — polls all GPU subsystems
|
||||
|
||||
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: gpu-monitor.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/gpu-monitor/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "GPU metrics accessible",
|
||||
"verify": "curl -sf http://localhost:9100/gpu-data",
|
||||
"expect": "200 OK, populated data"
|
||||
},
|
||||
{
|
||||
"check": "dashboard serving",
|
||||
"verify": "curl -sf http://localhost:9100/gpu-fleet.html",
|
||||
"expect": "200 OK, HTML returned"
|
||||
},
|
||||
{
|
||||
"check": "health endpoint responsive",
|
||||
"verify": "curl -sf http://localhost:9100/health",
|
||||
"expect": "200 OK"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/gpu-monitor/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=gpu-monitor, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## infrastructure-monitoring
|
||||
|
||||
**Category:** monitoring | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** */30 * * * *
|
||||
|
||||
```
|
||||
Contract Enforcement: infrastructure-monitoring
|
||||
|
||||
Category: monitoring
|
||||
Domain: infrastructure
|
||||
Owner: abiba
|
||||
Schedule: Every 30 minutes
|
||||
|
||||
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: infrastructure-monitoring.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/infrastructure-monitoring/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "Proxmox API reachable",
|
||||
"verify": "curl -sf http://192.168.68.10:8006/api2/json",
|
||||
"expect": "200 OK"
|
||||
},
|
||||
{
|
||||
"check": "Zulip API reachable",
|
||||
"verify": "curl -sf https://chat.sysloggh.net/api/v1/me",
|
||||
"expect": "200 OK"
|
||||
},
|
||||
{
|
||||
"check": "LiteLLM proxy reachable",
|
||||
"verify": "curl -sf http://192.168.68.116/litellm/v1/models",
|
||||
"expect": "200 OK"
|
||||
},
|
||||
{
|
||||
"check": "Gitea API reachable",
|
||||
"verify": "curl -sf https://git.sysloggh.net/api/v1/version",
|
||||
"expect": "200 OK"
|
||||
},
|
||||
{
|
||||
"check": "SearXNG reachable",
|
||||
"verify": "curl -sf http://192.168.68.17:8080",
|
||||
"expect": "200 OK"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/infrastructure-monitoring/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-monitoring, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## zulip-health
|
||||
|
||||
**Category:** monitoring | **Domain:** zulip | **Owner:** abiba | **Schedule:** */15 * * * *
|
||||
|
||||
```
|
||||
Contract Enforcement: zulip-health
|
||||
|
||||
Category: monitoring
|
||||
Domain: zulip
|
||||
Owner: abiba
|
||||
Schedule: Every 15 minutes — monitors all Zulip-connected agents
|
||||
|
||||
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: zulip-health.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/zulip-health/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "bot registration active",
|
||||
"verify": "curl -sf https://chat.sysloggh.net/api/v1/me | jq '.user_id'",
|
||||
"expect": "bot_id present"
|
||||
},
|
||||
{
|
||||
"check": "DM delivery working",
|
||||
"verify": "curl -sf https://chat.sysloggh.net/api/v1/users/me/is-online",
|
||||
"expect": "online: true"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/zulip-health/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=zulip-health, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## litellm-health
|
||||
|
||||
**Category:** monitoring | **Domain:** litellm | **Owner:** abiba | **Schedule:** */10 * * * *
|
||||
|
||||
```
|
||||
Contract Enforcement: litellm-health
|
||||
|
||||
Category: monitoring
|
||||
Domain: litellm
|
||||
Owner: abiba
|
||||
Schedule: Every 10 minutes — LiteLLM proxy health
|
||||
|
||||
This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: litellm-health.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/litellm-health/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "LiteLLM proxy reachable",
|
||||
"verify": "curl -sf http://192.168.68.116/litellm/v1/models",
|
||||
"expect": "200 OK, models returned"
|
||||
},
|
||||
{
|
||||
"check": "router deprecated, nginx routes work",
|
||||
"verify": "curl -sf https://litellm.sysloggh.net/v1/models",
|
||||
"expect": "200 OK (via nginx)"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba, mumuni → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/litellm-health/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=litellm-health, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## memory-audit-maintenance
|
||||
|
||||
**Category:** maintenance | **Domain:** memory | **Owner:** mumuni | **Schedule:** 0 3 * * *
|
||||
|
||||
```
|
||||
Contract Enforcement: memory-audit-maintenance
|
||||
|
||||
Category: maintenance
|
||||
Domain: memory
|
||||
Owner: mumuni
|
||||
Schedule: Daily at 3am ET
|
||||
|
||||
This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: memory-audit-maintenance.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/memory-audit-maintenance/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "memory files below 80% capacity",
|
||||
"verify": "wc -l ~/.hermes/memories/*.md",
|
||||
"expect": "total lines < threshold"
|
||||
},
|
||||
{
|
||||
"check": "no stale entries",
|
||||
"verify": "grep -r 'STALE' ~/.hermes/memories/",
|
||||
"expect": "0 matches"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify mumuni → action: relay_alert
|
||||
- CRITICAL: notify mumuni, abiba → action: relay_alert
|
||||
- FATAL: notify mumuni, abiba, kwame → action: relay_alert + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 3 per 60.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/memory-audit-maintenance/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=memory-audit-maintenance, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## infrastructure-update
|
||||
|
||||
**Category:** maintenance | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** 0 2 * * 0
|
||||
|
||||
```
|
||||
Contract Enforcement: infrastructure-update
|
||||
|
||||
Category: maintenance
|
||||
Domain: infrastructure
|
||||
Owner: abiba
|
||||
Schedule: Weekly system updates Sunday at 2am ET
|
||||
|
||||
This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
|
||||
|
||||
Steps:
|
||||
1. Load contract from prose-contracts/main (file: infrastructure-update.prose.md)
|
||||
2. Verify prerequisites (connectivity, tools, deps)
|
||||
3. Execute contract per SOP
|
||||
4. Run postconditions from contract registry
|
||||
5. Generate receipt with status (pass/fail/escalated)
|
||||
6. If any postcondition fails, escalate per contract escalation tiers
|
||||
7. Log to ~/.hermes/runs/infrastructure-update/
|
||||
|
||||
Postconditions to verify:
|
||||
[
|
||||
{
|
||||
"check": "all services running after update",
|
||||
"verify": "systemctl list-units --state=running",
|
||||
"expect": "all critical services"
|
||||
}
|
||||
]
|
||||
|
||||
Escalation Tiers:
|
||||
- INFO: notify nobody → action: log_to_receipt
|
||||
- WARNING: notify abiba → action: relay_alert
|
||||
- CRITICAL: notify abiba, mumuni → action: relay_alert
|
||||
- FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
|
||||
|
||||
Circuit Breaker:
|
||||
- Max retries: 1 per 120.0min window
|
||||
- On trip: escalate_to_fatal
|
||||
|
||||
Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
|
||||
Receipt storage: ~/.hermes/runs/infrastructure-update/receipt-{timestamp}.json
|
||||
Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-update, status=<status>
|
||||
|
||||
If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
|
||||
|
||||
IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
|
||||
```
|
||||
|
||||
|
||||
---
|
||||
*End of review*
|
||||
@@ -1,256 +0,0 @@
|
||||
---
|
||||
kind: pattern
|
||||
name: delegation-prose-contract
|
||||
description: >
|
||||
Manager (Mumuni) operating doctrine for task decomposition, worker
|
||||
delegation, verification, and delivery. Defines when to delegate, which
|
||||
worker to use for what, how to handle failures, and the kanban board
|
||||
protocol. Enforces context-window discipline and separation of concerns.
|
||||
Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
## Maintains
|
||||
|
||||
- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
|
||||
`syslog-research`, `syslog-review`, `syslog-writer`)
|
||||
- Kanban board state at `~/.hermes/kanban/kanban.json`
|
||||
- Context window budget: ~65K tokens per request (131K total, 60% threshold)
|
||||
|
||||
## Topology
|
||||
|
||||
**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
|
||||
**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
|
||||
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
|
||||
|
||||
This contract is infrastructure-agnostic in terms of which nodes are used.
|
||||
Workers execute tasks on whatever infrastructure they're given — SSH to .6,
|
||||
.pm, .9, .12, or .15 depending on the task. The contract defines the
|
||||
**who** and **when** — not the **where**.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Without enforced delegation, the manager consumes the full iteration budget
|
||||
(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
|
||||
leaving no capacity for actual coordination. The result: context overflow
|
||||
(59K tokens in system prompt), iteration exhaustion, and degraded response
|
||||
quality. This contract exists because I blew through my budget checking
|
||||
Proxmox node status instead of delegating to `syslog-devops`.
|
||||
|
||||
## Context Window Discipline
|
||||
|
||||
**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
|
||||
**Volatile (MEMORY.md + USER.md): ~300 tokens.**
|
||||
**Total base: ~6,800 tokens per request.**
|
||||
|
||||
The remaining budget is the conversation. Every tool call result adds to it.
|
||||
If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
|
||||
from multiple nodes), the context fills fast. That's why we delegate: workers
|
||||
process in isolation and return compact results.
|
||||
|
||||
## Trigger Conditions
|
||||
|
||||
Delegation is **mandatory** when any of these apply:
|
||||
|
||||
| Condition | Threshold | Example |
|
||||
|-----------|-----------|---------|
|
||||
| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
|
||||
| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
|
||||
| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
|
||||
| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
|
||||
| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
|
||||
| Code builds or changes | Writing or modifying code | Scripts, configs, patches |
|
||||
| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
|
||||
|
||||
**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
|
||||
`curl`, `hermes tools list` — these are decision-making tools. The manager
|
||||
reads them directly.
|
||||
|
||||
## Worker Selection Matrix
|
||||
|
||||
| Worker | Model | Toolsets | Role | Use When |
|
||||
|--------|-------|----------|------|----------|
|
||||
| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
|
||||
| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
|
||||
| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
|
||||
| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
|
||||
| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
|
||||
| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
|
||||
|
||||
### Selection Rules
|
||||
|
||||
1. **Match specialty first.** A code task → `syslog-code`. An infra task →
|
||||
`syslog-devops`. Don't put a `syslog-email` worker on a code review.
|
||||
2. **Research tasks with browser needs → `syslog-research`.** Other workers
|
||||
don't have the browser toolset.
|
||||
3. **Verification → `syslog-review`.** Never deliver raw worker output.
|
||||
4. **Documentation/content → `syslog-writer`.** Let them own the prose.
|
||||
5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
|
||||
(includes browser) and high reasoning effort.
|
||||
|
||||
## Delegation Protocol
|
||||
|
||||
### Step 1: Decompose
|
||||
|
||||
Break the task into lanes. Each lane does ONE thing. Workers are independent —
|
||||
no lane depends on another's output mid-flight. If lanes depend on each other,
|
||||
dispatch sequentially.
|
||||
|
||||
### Step 2: Dispatch
|
||||
|
||||
Fire workers via `delegate_task`:
|
||||
|
||||
**Parallel (independent lanes):**
|
||||
```
|
||||
delegate_task(
|
||||
tasks=[
|
||||
{"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
|
||||
{"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
**Sequential (dependent lanes):**
|
||||
Dispatch lane 1 → wait for result → dispatch lane 2.
|
||||
|
||||
### Step 3: Verify
|
||||
|
||||
**MANDATORY for:**
|
||||
- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
|
||||
- Code builds and modifications
|
||||
- Research findings (web data, external sources)
|
||||
- Any output that will reach the user
|
||||
|
||||
**Fire `syslog-review` to verify:**
|
||||
```
|
||||
delegate_task(
|
||||
goal="Review the output of the devops worker. Verify the node status
|
||||
report is accurate, check for inconsistencies, confirm all nodes were
|
||||
reachable.",
|
||||
context="Worker was syslog-devops. Output is at /tmp/node-report.md.
|
||||
Verify against live system."
|
||||
)
|
||||
```
|
||||
|
||||
**If verification fails:**
|
||||
1. Send work back to original worker with review feedback
|
||||
2. Re-verify
|
||||
3. Max 2 re-verify cycles before escalating to Kwame
|
||||
|
||||
### Step 4: Deliver
|
||||
|
||||
Only verified results reach Kwame. Format per channel:
|
||||
- Telegram: Use `telegram-formatting` skill
|
||||
- Zulip: Use Zulip Markdown (CommonMark)
|
||||
- Email: Use `syslog-email` skill
|
||||
|
||||
## Kanban Board Protocol
|
||||
|
||||
**File:** `~/.hermes/kanban/kanban.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"task_id": "unique-id",
|
||||
"title": "Task description",
|
||||
"created": "2026-07-09T01:00:00",
|
||||
"status": "backlog|in_progress|review|done",
|
||||
"lanes": [
|
||||
{
|
||||
"lane_id": "devops-check",
|
||||
"worker": "syslog-devops",
|
||||
"goal": "Check all 5 Proxmox nodes",
|
||||
"status": "dispatched|completed|failed",
|
||||
"output_file": "/tmp/node-report.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Update the board on every state change.**
|
||||
|
||||
## Failure Handling
|
||||
|
||||
### Worker Timeouts
|
||||
|
||||
- Child timeout: **900 seconds** (15 minutes)
|
||||
- Worker model `syslog-auto` is slow — it can hit the timeout limit with
|
||||
22+ API calls
|
||||
- **If a worker times out:** Re-dispatch with a narrower scope. Break the
|
||||
task into smaller pieces that fit in the timeout window.
|
||||
- **Avoid delegating sequential SSH hops** — each SSH connection adds latency
|
||||
that compounds quickly. Prefer API-based or local approaches when possible.
|
||||
|
||||
### Worker Selection Failures
|
||||
|
||||
- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
|
||||
- `syslog-code` is best for code-level work (reading files, writing scripts)
|
||||
- `syslog-research` has the browser toolset — use for web research
|
||||
- `syslog-review` is the QA gate — always fire before delivery
|
||||
- **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
|
||||
- **Never nest delegation** (max_spawn_depth: 1)
|
||||
|
||||
### Context Overflow
|
||||
|
||||
- If a task requires >10K tokens of output, delegate the processing
|
||||
- Workers return compact summaries, not raw data dumps
|
||||
- Pass file paths and concrete goals — never dump raw data into context
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- ❌ Reading large files into your own context before deciding → delegate the read
|
||||
- ❌ Carrying SSH/grep/output results in your context → delegate the analysis
|
||||
- ❌ Doing work yourself and then "pretending" to delegate → the user can tell
|
||||
- ❌ Skipping verification → raw worker output never reaches the user
|
||||
- ❌ Delegating single tool calls → keep quick reads/writes at manager level
|
||||
- ❌ Firing more than 3 workers in parallel → hard limit
|
||||
|
||||
## Emergency Exception
|
||||
|
||||
**In an emergency (server down, service must be restored immediately):**
|
||||
- Delegate the diagnosis (find the problem)
|
||||
- Execute the fix yourself (minimize handoff latency)
|
||||
- Verify the fix after delivery
|
||||
- Log the exception in the kanban board
|
||||
|
||||
The emergency exception exists because the user needs the service back NOW,
|
||||
not after three worker round-trips. But it's an exception — not the rule.
|
||||
|
||||
## What This Contract Doesn't Cover
|
||||
|
||||
1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
|
||||
2. **SSH key management** — covered by existing SSH/Proxmox contracts
|
||||
3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
|
||||
4. **Cron job management** — covered by individual cron contracts
|
||||
5. **Infra verification** — covered by `verify-before-mutate` protocol
|
||||
|
||||
## Verification
|
||||
|
||||
Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
|
||||
|
||||
```bash
|
||||
python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
|
||||
- `worker-profile-audit.md` (skill reference): Worker configuration audit notes
|
||||
- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
|
||||
- `verify-before-mutate` protocol: Infrastructure change verification
|
||||
- `hermes-config-template.prose.md`: Worker profile configuration
|
||||
|
||||
## Success Criteria
|
||||
|
||||
This contract succeeds when:
|
||||
|
||||
1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
|
||||
2. **Workers do the work** — manager coordinates, doesn't execute
|
||||
3. **Verification before delivery** — all output passes through `syslog-review`
|
||||
4. **Kanban board is current** — every task has a lane, every lane has a status
|
||||
5. **User gets verified results** — raw worker output never reaches Kwame
|
||||
|
||||
---
|
||||
|
||||
**Last updated:** 2026-07-09
|
||||
**Author:** Mumuni (with Kwame's input on triggers and exception criteria)
|
||||
**Status:** Draft — awaiting PR review and merge to prose-contracts main
|
||||
+115
-46
@@ -5,13 +5,12 @@ description: >
|
||||
Manages the GPU inference fleet across all hosts. Handles model deployment,
|
||||
registration, health checks, LiteLLM sync, agent key management, GPU
|
||||
saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
|
||||
UPDATED 2026-07-12: Architecture is DIRECT GPU — LiteLLM routes directly
|
||||
to llama-server on each GPU host (no router in inference path). Router
|
||||
(port 9000) is running but NOT in request path. All GPUs standardized on
|
||||
api-key 'not-needed'. RTX 5070 had api-key mismatch (sk-loc...5678) that
|
||||
caused cascading 401→timeout→401 fallback loops — fixed.
|
||||
Context: RTX 3090 verified at 256K (was documented as 128K — WRONG).
|
||||
Workload: Compression moved to Strix Halo, RTX 5070 → vision/web only.
|
||||
UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe,
|
||||
gpu-dense, gpu-light. These never change — only the underlying model does.
|
||||
Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
|
||||
RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
|
||||
RTX 5070 context: 131K → 256K. VRAM: 88% (10.8/12.2GB).
|
||||
Compression timeout: 300s (was 120s). Mumuni context: 128K (was 256K).
|
||||
agent: abiba
|
||||
triggers:
|
||||
- on model add/remove
|
||||
@@ -68,48 +67,72 @@ triggers:
|
||||
│ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │
|
||||
│ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │
|
||||
│ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │
|
||||
│ 256K ctx │ │ 131K ctx │ │ 256K ctx │ │ Watchdog │
|
||||
│ qwen3.6 │ │ gemma-4-12b │ │ ornith35B │ │ Prometheus │
|
||||
│ 27B-code │ │ :8080 │ │ :8080 │ │ exporter │
|
||||
│ 256K ctx │ │ 256K ctx │ │ 256K ctx │ │ Watchdog │
|
||||
│ qwen3.6 │ │ gemma-4-12b │ │ qwen3.6 │ │ Prometheus │
|
||||
│ 27B-code │ │ :8080 │ │ -35B-udq4 │ │ exporter │
|
||||
│ :8080 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │
|
||||
│ :9400 │ └─────────────┘ └───────────┘ └──────────────┘
|
||||
└──────────┘
|
||||
```
|
||||
|
||||
## Current Model Assignments (2026-07-12)
|
||||
## Stable Role-Based Aliases (Introduced 2026-07-15)
|
||||
|
||||
Agent configs, cron jobs, and workflows MUST use these aliases, never model-specific names.
|
||||
When a model is swapped on a GPU, ONLY the infrastructure layer changes — agent configs are untouched.
|
||||
|
||||
| Alias | GPU | Current Model | Will Route To |
|
||||
|-------|-----|---------------|---------------|
|
||||
| `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
|
||||
| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 |
|
||||
| `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
|
||||
|
||||
**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
|
||||
but are deprecated for agent configs. Only the stable aliases survive model swaps.
|
||||
|
||||
## Current Model Assignments (2026-07-15)
|
||||
|
||||
| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
|
||||
|-------|-----|------|------|-----|----------|----------|-------------|--------|
|
||||
| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.7/24GB (84%) | **256K** | turbo4 | 1 | default | ✅ healthy |
|
||||
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy |
|
||||
| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
|
||||
| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | 22.2/24.6GB (90%) | **256K** 🚀 | turbo4 | 2 | default | ✅ 63 tok/s |
|
||||
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 256K | q4_0 | 2 | 2048/1024 | ✅ healthy |
|
||||
| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
|
||||
|
||||
## Routing Configuration (LiteLLM — July 2026)
|
||||
|
||||
### syslog-auto Weighted Pool
|
||||
### syslog-auto Weighted Pool (Direct GPU — bypasses router)
|
||||
|
||||
| Model | GPU | Weight | RPM Cap | Purpose |
|
||||
| Model | GPU | Weight | RPM Cap | Timeout |
|
||||
|-------|-----|--------|---------|---------|
|
||||
| qwen3.6-27B-code | RTX 3090 | 0.55 | 500 | Heavy reasoning, code, long context |
|
||||
| ornith-1.0-35b | Strix Halo | 0.30 | **60** | Agentic workflows, tool calling |
|
||||
| gemma-4-12b | RTX 5070 | 0.15 | 200 | Overflow + vision |
|
||||
| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
|
||||
| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
|
||||
| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
|
||||
|
||||
Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path.
|
||||
|
||||
### Direct Model Endpoints
|
||||
|
||||
| Model | RPM Cap | Notes |
|
||||
|-------|---------|-------|
|
||||
| ornith-1.0-35b | 40 | Tight cap — prevents Strix overload |
|
||||
| qwen3.6-35B-udq4 | 40 | Tight cap — prevents Strix overload |
|
||||
| qwen3.6-27B-code | 500 | High cap — primary workhorse |
|
||||
| gemma-4-12b | 500 | High cap — fast 12B |
|
||||
| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
|
||||
|
||||
### Stable Aliases (for agent configs — never change)
|
||||
|
||||
| Alias | RPM Cap | Routes To | Purpose |
|
||||
|-------|---------|-----------|---------|
|
||||
| `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) |
|
||||
| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
|
||||
| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
|
||||
|
||||
### Fallback Chains
|
||||
- gemma → qwen
|
||||
- qwen → gemma
|
||||
- ornith → qwen → gemma
|
||||
- syslog-auto → qwen → gemma → ornith
|
||||
- qwen3.6-35B-udq4 → qwen → gemma
|
||||
- syslog-auto → qwen → gemma → qwen3.6-35B-udq4
|
||||
|
||||
### Why ornith RPM Is Capped
|
||||
- Direct: 40 RPM (tight) — Strix Halo is shared with compression tasks
|
||||
### Why Strix Halo RPM Is Capped
|
||||
- Direct (qwen3.6-35B-udq4): 40 RPM (tight) — Strix Halo is shared with compression tasks
|
||||
- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
|
||||
- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
|
||||
|
||||
@@ -163,7 +186,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
|
||||
3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
|
||||
4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
|
||||
5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
|
||||
- gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
|
||||
- gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (strix-moe does NOT exist — legacy name, do not use)
|
||||
- global request_timeout: 300s, nginx proxy_read_timeout: 600s
|
||||
6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
|
||||
7. Check port conflicts: verify only one llama-server on :8080 per host
|
||||
@@ -203,7 +226,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
|
||||
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
|
||||
| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
|
||||
| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
|
||||
| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
|
||||
| `/etc/systemd/system/strix-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo) running unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
|
||||
|
||||
## Prometheus & Grafana
|
||||
|
||||
@@ -220,14 +243,14 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
|
||||
- **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
|
||||
Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
|
||||
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
|
||||
- **VRAM (2026-07-12)**: RTX 3090 at 20.7/24GB (84%), RTX 5070 at 10.0/12.2GB (82%). RTX 3090 context increased to 256K (was incorrectly documented as 128K — verified via /proc/PID/cmdline).
|
||||
- **RTX 3090 runs `--parallel 1`** (verified 2026-07-12). RTX 5070 and Strix at parallel 2.
|
||||
- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching`. Service: `/home/llmuser/llama-wrapper.sh`.
|
||||
- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 1024 --parallel 2`. Api-key standardized to `not-needed` (was `sk-loc...5678` causing 401 loops). Service: `/home/llmuser/llama-wrapper.sh`.
|
||||
- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
|
||||
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
|
||||
- **VRAM (2026-07-15)**: RTX 3090 at 22.2/24.6GB (90%) with **256K context** (corrected from 131K). RTX 5070 at 10.8/12.2GB (88%) with 256K context + MTP. Strix Halo at ~9GB/64GB.
|
||||
- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
|
||||
- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context corrected to 256K (2026-07-15). VRAM: 90%. Service: `/home/llmuser/llama-wrapper.sh`.
|
||||
- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 256K context. Gen speed: 122 tok/s (was 70). VRAM: 10.8/12.2GB (88%). No draft model pre-upgrade due to VRAM constraints. Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 262144`.
|
||||
- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
|
||||
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080 (was `ornith-server.service`), model changed to `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (UD-Q4_K_M), alias `qwen3.6-35B-udq4`, 256K context, flash-attn + q8 KV. MTP support enabled for 1.4-2.2x faster inference.
|
||||
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
|
||||
- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
|
||||
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
|
||||
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
|
||||
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
|
||||
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
|
||||
@@ -237,11 +260,14 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
|
||||
|
||||
## GPU Inference Benchmarks (Current)
|
||||
|
||||
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
|
||||
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
|
||||
|-----|-------|-----------|--------------|----------|---------|
|
||||
| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 |
|
||||
| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 |
|
||||
| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |
|
||||
| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **256K** |
|
||||
| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **256K** |
|
||||
| Strix Halo (.15) | qwen3.6-35B-udq4 | **71** | — | — | **256K** |
|
||||
|
||||
Benchmarks from 2026-07-15 verification run. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
|
||||
All 3 GPUs now at 256K context (2026-07-15).
|
||||
|
||||
Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
|
||||
Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
|
||||
@@ -249,12 +275,55 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window.
|
||||
|
||||
**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
|
||||
|
||||
## Agent Config Implications (2026-07-12)
|
||||
## Agent Config Implications (2026-07-15)
|
||||
|
||||
With RTX 3090 at 256K context (verified July 2026):
|
||||
- Agents using `syslog-auto` (55/30/15 qwen+ornith+gemma): `context_length: 262144` — ornith and qwen both support it
|
||||
- Agents using `qwen3.6-27B-code` directly: `context_length: 262144` (256K ctx verified)
|
||||
- Agents using `gemma-4-12b` directly (auxiliary tasks): `context_length: 131072`
|
||||
- Compression threshold at 0.65: fires at ~170K for 262K context window on Strix Halo
|
||||
- All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
|
||||
- Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)
|
||||
### Stable Aliases — CRITICAL
|
||||
|
||||
All agent configs MUST use stable role-based aliases, never model-specific names:
|
||||
- `compression.model: strix-moe` (NOT `qwen3.6-35B-udq4`)
|
||||
- `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`)
|
||||
- `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`)
|
||||
- `auxiliary.web_extract.model: gpu-light`
|
||||
|
||||
When the underlying model is swapped, only the LiteLLM config changes — agent configs are untouched.
|
||||
|
||||
### Context Windows
|
||||
- RTX 3090: **256K** (was 131K, bumped 2026-07-15) | RTX 5070: **256K** (up from 131K) | Strix Halo: **256K**
|
||||
- **Mumuni compression context**: 128K (down from 256K) — ensures compression model doesn't timeout
|
||||
- Compression threshold 0.65: fires at ~85K for 128K context window
|
||||
- Mumuni compression model alias: `strix-moe` with 300s timeout
|
||||
|
||||
### Mumuni Agent Profile
|
||||
|
||||
Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs:
|
||||
|
||||
| Setting | Value | Notes |
|
||||
|---------|-------|-------|
|
||||
| `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) |
|
||||
| `model.provider` | `custom:litellm` | LiteLLM on CT116 |
|
||||
| `compression.model` | `strix-moe` | Stable alias — survives model swaps |
|
||||
| `aux.compression.model` | `strix-moe` | Compression auxiliary model |
|
||||
| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
|
||||
| `aux.web_extract.model` | `gpu-light` | Web extraction |
|
||||
| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
|
||||
| `context.max_context_window` | 262144 (256K) | Fixed 2026-07-16 (was 131072 — caused premature compression, WAL #1300) |
|
||||
| `compression.threshold` | 0.65 | Triggers at ~85K |
|
||||
| `compression.target_ratio` | 0.3 | Compresses to ~38K |
|
||||
| `compression.protect_last_n` | 40 | Preserves last 40 messages |
|
||||
| `memory.memory_char_limit` | 800 | Brief memory entries |
|
||||
| `personalities` | `creative` | Creative assistant personality |
|
||||
| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
|
||||
| Main model timeout | 300s | LiteLLM global timeout |
|
||||
| Compression model timeout | 300s | strix-moe timeout increased from 120s |
|
||||
|
||||
### Agent Update Status (2026-07-15)
|
||||
|
||||
| Agent | Host | Status |
|
||||
|-------|------|--------|
|
||||
| **Mumuni** | CT114 (.123) | ✅ Updated to stable aliases |
|
||||
| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
|
||||
| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
|
||||
| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
|
||||
| **Kagenz0** | CT105 | ❌ SSH unreachable — needs Zulip DM |
|
||||
|
||||
All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap.
|
||||
|
||||
@@ -5,7 +5,7 @@ version: 1.0.0
|
||||
description: >
|
||||
Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
|
||||
configuration state, keys, workarounds, and audit procedure. When an agent's
|
||||
configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
|
||||
configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
|
||||
author: Abiba (pi agent)
|
||||
---
|
||||
|
||||
@@ -26,11 +26,11 @@ done
|
||||
|-------|-----|------|-----|---------------|------------|----------|
|
||||
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
||||
| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
|
||||
| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||
| Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
||||
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes |
|
||||
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
||||
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
||||
|
||||
> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
|
||||
> **Note**: CT hostnames (tdunna→CT129, baggy→CT114) differ from agent identities (koby, koonimo).
|
||||
|
||||
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
|
||||
Keys are stored in Infisical vault (project=agents, env=production) and injected at
|
||||
@@ -169,9 +169,9 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
|
||||
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
|
||||
```
|
||||
|
||||
### For Koby (CT 111 / tdunna)
|
||||
### For Koby (CT 129 / tdunna)
|
||||
|
||||
Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
|
||||
Koby runs Hermes on CT 129 (tdunna). Config files at `/root/.hermes/config.yaml`.
|
||||
Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
|
||||
|
||||
**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
|
||||
|
||||
+153
-3
@@ -14,9 +14,15 @@ description: >
|
||||
expire/404. The .env fallback prevents agents from running without keys.
|
||||
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
|
||||
|
||||
UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
|
||||
token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
|
||||
infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
|
||||
Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
|
||||
Abiba's key is now a proper agent key (NOT the master key — stale note removed).
|
||||
|
||||
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
|
||||
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
|
||||
on CT 116. Last verified: 2026-07-12.
|
||||
on CT 116. Last verified: 2026-07-16.
|
||||
---
|
||||
|
||||
## Parameters
|
||||
@@ -51,8 +57,8 @@ description: >
|
||||
- Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
|
||||
- Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
|
||||
- Duration is null (permanent) — inherited from litellm default_key_generate_params
|
||||
- Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
|
||||
- Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
|
||||
- Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
|
||||
- Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
|
||||
- Return the new key
|
||||
5. **If action == "rotate"**:
|
||||
- Generate new key with same alias (LiteLLM replaces the old key)
|
||||
@@ -67,3 +73,147 @@ description: >
|
||||
- Test the key against LiteLLM /v1/models
|
||||
- Confirm key alias matches agent_name in LiteLLM key list
|
||||
- Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
|
||||
|
||||
## Production Vault Access Process (canonical, 2026-07-16)
|
||||
|
||||
The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending —
|
||||
runs as user `jerome`, not systemd root, needs user-scope adaptation).
|
||||
|
||||
### The canonical pattern
|
||||
|
||||
1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`).
|
||||
2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`).
|
||||
- Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project.
|
||||
- Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).
|
||||
3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`):
|
||||
```bash
|
||||
#!/bin/bash
|
||||
export INFISICAL_API_URL="https://vault.sysloggh.net"
|
||||
TOKEN=$(cat /root/.infisical-token)
|
||||
LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs
|
||||
while true; do
|
||||
infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
|
||||
--env=prod --domain=https://vault.sysloggh.net -- bash -c '
|
||||
. /root/.hermes/.env 2>/dev/null # [FALLBACK Rule 3] safety net only
|
||||
export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"
|
||||
exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run
|
||||
' >> $LOG 2>&1
|
||||
sleep 5 # restart on exit
|
||||
done
|
||||
```
|
||||
4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth.
|
||||
5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation.
|
||||
6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot).
|
||||
7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault.
|
||||
|
||||
### Why this is non-fail
|
||||
|
||||
- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
|
||||
- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable.
|
||||
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway.
|
||||
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source.
|
||||
|
||||
### Migration status (2026-07-16)
|
||||
|
||||
| Agent | Host | Pattern | Vault key | Status |
|
||||
|-------|------|---------|-----------|--------|
|
||||
| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed |
|
||||
| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed |
|
||||
| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram |
|
||||
| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
|
||||
> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent.
|
||||
| tanko | .122 | infisical-gateway.sh + service token (migrated 2026-07-16) | TANKO_LITELLM_API_KEY | ✅ vault-backed |
|
||||
|
||||
### Tanko migration (pending)
|
||||
|
||||
Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in
|
||||
`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced).
|
||||
Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`)
|
||||
with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering
|
||||
enabled (`loginctl enable-linger jerome`) so the user service runs without a login session.
|
||||
|
||||
### Koby migration lessons (2026-07-16)
|
||||
|
||||
Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
|
||||
**Two mistakes I made that broke the agent:**
|
||||
1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
|
||||
in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
|
||||
inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
|
||||
2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
|
||||
Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
|
||||
|
||||
**How Koby actually connects (2026-07-16):**
|
||||
- Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
|
||||
Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config).
|
||||
- Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026).
|
||||
Allowed users: 6679773481. Home channel: 6679773481.
|
||||
- Both platforms now connect through the wrapper's env injection.
|
||||
|
||||
**Golden rule for gateway restarts:** always `cat /proc/<pid>/environ` before killing the old
|
||||
process — captures the live env set. Especially important when migrating gateways between
|
||||
injection mechanisms.
|
||||
|
||||
### Key rotation procedure (one vault operation with this standard)
|
||||
|
||||
1. Generate new key: `POST /key/generate` (master key, admin).
|
||||
2. Update vault: `infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`.
|
||||
3. Update `.env` fallback: `echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`.
|
||||
4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
|
||||
5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
|
||||
|
||||
## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
|
||||
|
||||
**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
|
||||
In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace
|
||||
key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt
|
||||
repo only ships 0.38.0, so `apt upgrade` does not help.
|
||||
|
||||
**Proper fix — Machine Identity (Infisical automation best practice):**
|
||||
Create a machine identity with READ+WRITE scope on the `agents` project (project_id=
|
||||
`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely.
|
||||
Then vault writes work from any host:
|
||||
```bash
|
||||
# Get a machine-identity access token
|
||||
TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"clientId":"<CLIENT_ID>","clientSecret":"<CLIENT_SECRET>"}' | jq -r .accessToken)
|
||||
# Write a secret via REST API v3
|
||||
curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \
|
||||
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
||||
-d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
|
||||
# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
|
||||
```
|
||||
**2026-07-16 UPDATE — vault SYNCED + CLEANED.** The abiba service token (`st.8e848433…`, READ+WRITE)
|
||||
can write to the vault. All session-13 rotated keys in vault and validate 200 against LiteLLM.
|
||||
4 stale secrets deprecated, 5 personal creds flagged for separate project. Tanko migrated from
|
||||
hardcoded keys to `infisical-gateway.sh` wrapper. Koonimo Zulip key restored.
|
||||
|
||||
**Machine Identity:** `8ddb9438-74fc-4ab6-bd74-929e7c47b53b` exists in Infisical UI.
|
||||
Client secret needed to use universal auth for vault writes from automation.
|
||||
|
||||
**Service Token Inventory (2026-07-16):**
|
||||
| Token ID | Name | Permissions | Used By |
|
||||
|----------|------|-------------|---------|
|
||||
| `st.8e848433…` | tanko-gateway | READ+WRITE | Abiba, Koonimo |
|
||||
| `st.353699cd…` | tanko-agent | READ-only | Tanko |
|
||||
|
||||
## Key Rotation Log
|
||||
|
||||
| Date | Agent | Action | Notes |
|
||||
|------|-------|--------|-------|
|
||||
| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault (Tt2bUL…). Updated wrapper to inject ZULIP_API_KEY + ZULIP_EMAIL. Restarted gateway → Zulip connected as koonimo-bot@ (bot_id=17). 3 platforms now. |
|
||||
| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml key to infisical-gateway.sh wrapper + service token st.353699cd… (tanko-agent). Systemd user service updated, hardcoded key drop-ins removed. Verified LITELLM_API_KEY from vault, 3 platforms connected. |
|
||||
| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated: KAGENZ0_LITELLM_API_KEY, HERMES_OPENROUTER_KEY, LITELLM_API_KEY (generic duplicate), ZULIP_API_KEY (generic duplicate). 5 personal creds flagged for separate project. |
|
||||
| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
|
||||
| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
|
||||
| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
|
||||
|
||||
## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
|
||||
|
||||
- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
|
||||
- Used for /key/generate, /key/delete, /key/list (GET), DB queries
|
||||
- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
|
||||
It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
|
||||
The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference —
|
||||
see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference".
|
||||
- LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`
|
||||
|
||||
Reference in New Issue
Block a user