Governance That Survives the Agent's Own Confidence

Four incidents, three different systems inside this factory. Read separately, each is a specific answer to a specific incident. Read together, they’re the same lesson told four times: a governance rule that only applies when the agent agrees it should isn’t governance at all.

The actual problem

Most guardrails get designed around the case where the agent recognizes danger. Flag the risky action, ask for confirmation, log the decision. That works exactly as well as the agent’s own risk model. Which means it works right up until the agent is confident, well-tested, and wrong, and the guardrail doesn’t fire, because from the inside, nothing looked risky.

Real governance has to hold in that exact case. Not “the agent will probably notice.” A rule that binds independent of whether the agent, in the moment, thinks it’s necessary.

“The agent will probably notice” is a risk model, not a guardrail.

Four times the same lesson landed

Three different platform components inside Hekton hit this four separate ways, and none of the four fixes was “add more review.”

Start with the one where the stakes were highest and the temptation to skip the rule was strongest. This site’s own go-live day, the Infrastructure Gremlin’s deploy-authority contract requires a human-apply step whenever its own blast-radius classifier flags a Terraform plan RED. The agent flagged its own launch plan RED, was confident it was a false positive, and the human-apply step held anyway. It was a false positive. The step still held, because a rule that bends the moment the agent is confident isn’t a rule, it’s a suggestion with extra steps.

That held on the highest-visibility day, in infrastructure. A separate system, the engine gateway that decides where this factory’s tasks run and whether their content may leave the machine, hit the same discipline somewhere far less visible: which engine is even allowed to see a task’s data. Data-egress consent got keyed to how a model runs, local, CLI, API, or handoff, instead of which vendor owns it or what an automation flag claims. A non-interactive JSON mode had no way to ask for consent at all, so the fix wasn’t to let a flag imply consent that was never given. It was to refuse cloud runtime outright in that mode, and require an explicit interactive path when consent genuinely matters.

A rule that binds on a runtime fact still has to survive someone changing the rule itself, which is where the same gateway’s second incident landed. Every material change to its own control surfaces, routing, privacy, egress consent, the ledger, now has to pass through an architecture review run through the gateway’s own tooling before it ships. That review caught a tidy-looking refactor of the egress-consent gate quietly teaching it to fail open, a regression all 106 of the gateway’s existing tests missed completely. The review board caught it before the change ever shipped, in the same session it was written, which is the entire point: the gate never actually approved anything it shouldn’t have, because the rule that reviews changes to the gate doesn’t get waived just because the team judged this particular change as obviously safe.

The fourth incident is a third system entirely, and it’s the discipline that makes the other three checkable at all instead of just plausible. The local coding lab’s review-panel promotion work now logs a control’s real false-approval count and failure shape as the standard entry format, not a pass or fail summary. A promotion decision built on a clean-looking summary is a decision built on whichever part of the evidence was easiest to make look good.

A system that grades its own confidence is not exempt from the rule that confidence doesn’t grant.

The reusable pattern

Underneath all four is one question worth running against any guardrail you’re designing: does this rule still bind when the agent is confident it doesn’t need to? A flag that stops the pipeline even when the agent believes it’s wrong. A consent gate keyed to a fact about how the system is running, not a flag the system itself can set. A review board that includes changes to the review board. An eval report that states the failure rate instead of hiding it inside a summary.

None of those four mechanisms depend on the agent recognizing anything. That’s the actual design principle. Build the check so it fires on a structural fact, not on the agent’s own read of the situation, because the agent’s read is exactly the thing you can’t govern by asking it to be more careful.

The decision this factory made, across all four, was the same one each time: never let “the agent judged this safe” stand in for a rule holding. Every guardrail worth keeping got rewritten to key on something the agent can’t reason its way around, a classifier’s flag, a runtime property, a change to a control surface, a logged failure rate, rather than on whether the agent, in the moment, agreed it mattered.

The artefact

None of the three internal systems above are something I can hand you to clone, and none of them involve an agent at all, so be clear about what this is: not a reproduction of any of the four incidents, an illustration of the same underlying discipline in a system simple enough to fit in one file. github.com/dermdunc/green-ci-stale-deploy is a deploy pipeline that stays green while shipping stale content, because its sync step used a same-byte-length comparison that a same-length content change slips straight past. No agent, no confidence, just a cheap check standing in for a real one. The fix is the same shape as the four mechanisms above: check the fact that’s actually true, not the fact that’s cheap to check.

$ bash tests/test-repro.sh
=== Day 1: first deploy ===
upload: index.html

=== Day 2: a same-length date change ships ===
skip (size unchanged): index.html

=== Assertion 1: the bug is real - live still says '16 July 2026' ===
CONFIRMED: live/index.html is stale (size-only sync skipped the change).

=== Assertion 2: content-hash verification catches it ===
STALE (content mismatch): index.html
CONFIRMED: verify-deploy.sh correctly failed and named the stale file.

Both assertions passed: the bug reproduces, and the fix catches it.

Clone it, run that, and watch the pipeline report success on a deploy that didn’t actually deploy anything new, then watch the content-hash check catch what the size-only check missed. The fix is the same category of move as keying consent to a runtime fact instead of a flag: check the thing that’s actually true, not the thing that’s cheap to check.

What next

If you’re designing guardrails for anything that can modify infrastructure, move data across a boundary, or approve its own changes, run each proposed rule against one question before you call it sufficient: does it still bind when the agent is confident it doesn’t need to fire.

Next time: back to basics, and why routing every task to the most expensive model you can reach was never actually the safe choice it felt like.