Detection first: remediating Kubernetes upgrades without mutating clusters

Most Kubernetes upgrade tooling fails in the same way: it discovers a problem and fixes it in the same breath. That works on one cluster with an engineer watching. Across a fleet of 1500+ environments it means every policy change is a change to code that writes to production.

The remediation platform I operate splits that in half. Detection answers what is true. Planning answers what should be done. Only execution is allowed to mutate, and it replays a plan that a human could have read first.

What a detection stage is allowed to do

Detection queries control-plane, node, workload and add-on state and classifies drift into known issue classes — currently 28+ modelled classes. It emits evidence. It does not decide, retry, or repair.

• Read-only API access, so the stage is safe to schedule continuously across every environment.
• One typed record per finding: issue class, object reference, severity, evidence fields.
• No policy embedded in the collector — classification is data, not control flow.

{
  "issue_class": "deprecated_api_version",
  "object": { "kind": "Ingress", "namespace": "payments", "name": "checkout" },
  "severity": "high",
  "evidence": { "apiVersion": "extensions/v1beta1", "removed_in": "1.22" }
}

Why the boundary matters at fleet scale

The dangerous failure at scale is not a broken script. It is a correct script applied to a wrong assumption, simultaneously, everywhere. Separating detection from action gives you three properties that are hard to retrofit:

• You can run detection fleet-wide before you trust any remediation, and get a real inventory instead of an estimate.
• Policy iterates without touching the mutating path, so a classification tweak is not a production change.
• Execution has a reviewable input. A plan is a diff, and a diff can be rejected.

If your automation cannot tell you what it found without also changing it, you do not have an inventory — you have a side effect.

Where teams usually get stuck

The common objection is speed: two stages feel slower than one. In practice the slow part was never the API calls, it was the rollback conversation after an unreviewed sweep. Detection-first automation trades a few seconds of pipeline time for the ability to authorise change deliberately.

If you are planning a multi-version Kubernetes upgrade across many environments, build the detector before the fixer. The detector is what makes the fixer safe to write.

More notes

• JSON contracts between pipeline stages: treating remediation as data
• Exit code zero is not a success signal: closing the remediation loop