A CRD upgrade can pass schema validation even while a rolling deployment leaves older controller pods reading or writing the previous object shape. The risky cases include renamed fields, changed defaults, status fields whose meaning has changed, and rollback after some objects have been rewritten. What compatibility contract do you use during this transition? I'm considering an additive-first approach, explicit conversion when versions differ, checks for storedVersions, fixtures that exercise both controller versions, and a rollback plan that doesn't assume newly stored objects remain readable by the old binary. What do you test before rollout, and when is a conversion webhook worth the operational cost compared with keeping one storage version and performing a staged migration?
4 Answers
For genuinely incompatible APIs, use separate versions that can coexist. The old controller should continue handling only the old version, while the new controller and conversion logic support the new one. If information exists in one version but cannot be represented in the other, rollback or round-trip conversion may be lossy, so that limitation needs to be part of the compatibility design.
Before rollout, run both controller binaries against the same fixtures. Include objects created by the old controller, objects rewritten by the new one, missing optional fields, changed defaults, and status values from both generations. Verify that the old binary can safely observe or reconcile anything it might encounter during the rollout, or prevent it from touching objects it cannot understand.
Treat this as an API migration rather than an in-place schema edit. Add a new served API version while keeping exactly one storage version. If the representations have different meanings, deploy the conversion service first and use webhook conversion; the default strategy is suitable only when the versions are structurally compatible. Run a controller that understands both versions, provide sensible defaults for missing fields, move clients to the new version, then switch storage and rewrite existing objects through the API. Changing the CRD alone does not rewrite persisted objects. Check storedVersions and old-version API usage before disabling the old version. Test round trips in both directions, preserve data that has no direct equivalent, and take backups before rewriting resources.
A single storage version with staged migrations can be simpler than operating a conversion webhook. It requires more coordination, but reduces another production dependency and makes failure modes easier to reason about. Either way, plan the rollback before switching storage: once objects have been rewritten, an old controller may not be able to read them safely.
That works cleanly when you introduce distinct API versions. The trouble starts when someone tries to rename fields or change their meaning inside one version without adding conversion logic, which creates exactly the mixed-controller window that is easy to underestimate.