Exit code zero is not a success signal: closing the remediation loop

A remediation script exiting zero means the script ran. It says nothing about the cluster. Across a large fleet that gap is where confidence quietly detaches from reality: dashboards go green, and the same issue class shows up again next upgrade cycle.

Verification with the same code path

Re-detection is a required stage in the pipeline, not an optional check. After execution, detection runs again against the same contract, and the finding that triggered the action must be gone.

• Success is measured by the disappearance of the original signal, not by process exit status.
• Residual findings feed a second planned phase instead of being dropped.
• Unresolved classes are routed to engineering review rather than retried blindly.

{
  "environment": "cluster-eu-prod-07",
  "actions_attempted": 14,
  "actions_applied": 13,
  "remaining_issues": 1,
  "unresolved_classes": ["custom_admission_webhook_timeout"]
}

Phased remediation beats one large sweep

Because re-detection sits between phases, remediation proceeds in reviewed increments: detect, plan phase 1, execute, re-detect, plan phase 2, execute, record result. Each phase is small enough to reason about and reversible enough to stop.

A fix counts as done only when the signal that raised it disappears.

That single rule is what turned a collection of upgrade scripts into a platform. It also happens to be the cheapest reliability improvement available to most infrastructure teams — you already have the detector; run it twice.

More notes

• Detection first: remediating Kubernetes upgrades without mutating clusters
• JSON contracts between pipeline stages: treating remediation as data