fix(hermes): separate POLICY observations from FAULT findings in the audit contracts #90

2026-09-14T12:38:52Z

abiba-bot commented

2026-09-14 12:38:52 +00:00

Follow-up to #89. That fix made the audit's reachability verdict trustworthy; this one fixes what the audit says about what it finds.

The defect (measured today)

Minutes after #89 merged, the same contracts reported two further "violations":

Mumuni (192.168.68.14) - "empty base_url/api_key in multiple places"
Koonimo (192.168.68.114) - "wrong base_url: https://api.deepseek.com/v1"

Both agents are working. Their own logs, read at the authority:

kagentz .14  2026-09-14 12:11:49  API call #40: model=glm-5.3-flash    provider=custom   in=58423 out=190  total=58613
baggy   .114 2026-09-14 06:05:37  API call #10: model=deepseek-v4-pro provider=deepseek in=83375 out=4509

So neither config is broken. What the audit actually detected is that both agents route to a provider other than the internal harness - a policy question for the captain, not a fault. Two distinct errors were being committed at once: mislabelling policy as breakage, and asserting a credential/base_url fault that live evidence contradicts.

A third error of the same family: Koonimo's config names api_key_env: LITELLM_API_KEY at top level while her runtime resolves provider: deepseek to her DeepSeek key and succeeds - the audit inferred credential resolution from config text and got it wrong.

Changes

Each of hermes-key-enforcement, hermes-config-template and hermes-agent-baseline gains a Violation Classification section (20 lines each) that requires:

POLICY (observation only, never phrased as breakage): the agent uses a non-harness provider, or a config field looks unusual while its calls succeed - e.g. POLICY: Koonimo uses deepseek directly; calls succeeding in the last hour.
FAULT (needs request-level evidence): calls failing with an observed auth error, or a missing key together with failing calls - e.g. FAULT: Koby's key expired; 401 observed at <timestamp>.
Explicit rules: do not infer the runtime's credential resolution from config text; require an observed auth failure or the absence of successful calls before calling anything a fault; state what was OBSERVED, not what a field implies.

Verification

Three-dot diff against master (71ceda0, the #89 merge): 3 files, +60/-0, contracts only.
No assertion about either agent's compliance is made by this PR - it changes what the contracts are allowed to claim. The two agents above remain as they are, reported as POLICY with the log lines that prove their calls succeed.
Reviewers: confirm no contract can still print a provider difference under a fault label, and that each FAULT example names an observable (a 401, or an empty success window) rather than a config reading. The policy question itself (should every agent route through the harness?) is with the captain and deliberately left undecided in the contracts.

Follow-up to #89. That fix made the audit's reachability verdict trustworthy; this one fixes what the audit *says* about what it finds. ## The defect (measured today) Minutes after #89 merged, the same contracts reported two further "violations": - **Mumuni (192.168.68.14)** - "empty base_url/api_key in multiple places" - **Koonimo (192.168.68.114)** - "wrong base_url: https://api.deepseek.com/v1" Both agents are working. Their own logs, read at the authority: ``` kagentz .14 2026-09-14 12:11:49 API call #40: model=glm-5.3-flash provider=custom in=58423 out=190 total=58613 baggy .114 2026-09-14 06:05:37 API call #10: model=deepseek-v4-pro provider=deepseek in=83375 out=4509 ``` So neither config is broken. What the audit actually detected is that both agents route to a provider **other than the internal harness** - a policy question for the captain, not a fault. Two distinct errors were being committed at once: mislabelling policy as breakage, and asserting a credential/base_url fault that live evidence contradicts. A third error of the same family: Koonimo's config names `api_key_env: LITELLM_API_KEY` at top level while her runtime resolves `provider: deepseek` to her DeepSeek key and succeeds - the audit inferred credential resolution from config text and got it wrong. ## Changes Each of `hermes-key-enforcement`, `hermes-config-template` and `hermes-agent-baseline` gains a **Violation Classification** section (20 lines each) that requires: - **POLICY** (observation only, never phrased as breakage): the agent uses a non-harness provider, or a config field looks unusual while its calls succeed - e.g. `POLICY: Koonimo uses deepseek directly; calls succeeding in the last hour`. - **FAULT** (needs request-level evidence): calls failing with an observed auth error, or a missing key **together with** failing calls - e.g. `FAULT: Koby's key expired; 401 observed at <timestamp>`. - Explicit rules: do not infer the runtime's credential resolution from config text; require an observed auth failure or the absence of successful calls before calling anything a fault; state what was OBSERVED, not what a field implies. ## Verification - Three-dot diff against `master` (`71ceda0`, the #89 merge): 3 files, +60/-0, contracts only. - No assertion about either agent's compliance is made by this PR - it changes what the contracts are allowed to claim. The two agents above remain as they are, reported as POLICY with the log lines that prove their calls succeed. - Reviewers: confirm no contract can still print a provider difference under a fault label, and that each FAULT example names an observable (a 401, or an empty success window) rather than a config reading. The policy question itself (should every agent route through the harness?) is with the captain and deliberately left undecided in the contracts.

abiba-bot added 1 commit 2026-09-14 12:38:52 +00:00

fix(hermes): separate policy observations from fault findings

PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s

Details

PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s

Details

PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 7s

Details

PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s

Details

PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s

Details

9a789ab76d

Contract design defect: 'uses a non-harness provider' (POLICY) and
'cannot authenticate' (FAULT) were printed as the same violation class.
A policy observation must never be phrased as if the agent were broken.

Changes:
1. Added 'Violation Classification' section to all three contracts
2. Separated POLICY (observation only) from FAULT (requires request-level evidence)
3. Rules:
   - Do NOT infer runtime credential resolution from config text alone
   - Require request-level evidence before calling a FAULT: observed auth failure
     or absence of successful calls
   - If calls are succeeding, output is 'POLICY: uses <provider> directly; calls
     succeeding' - not a violation
   - State what you OBSERVED, not what the field implies

Files changed (3):
- hermes-key-enforcement.prose.md
- hermes-config-template.prose.md
- hermes-agent-baseline.prose.md

abiba-bot merged commit cb26ee06d6 into master

2026-09-14 12:50:16 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#90