A CRD upgrade can pass schema validation even while a rolling deployment leaves older controller pods reading or writing the previous object shape. The risky cases include renamed fields, changed defaults, status fields whose meaning has changed, and rollbacks after some objects have already been rewritten. What compatibility contract do you use during that transition? I'm considering an additive-first approach, explicit conversion when versions differ, checks around storedVersions, fixtures that exercise both controller versions, and a rollback plan that doesn't assume newly stored objects are readable by the old binary. What do you test before rollout, and when is a conversion webhook worth the operational cost compared with keeping one storage version and performing a staged migration?
4 Answers
Treat this as an API migration rather than an in-place schema edit. Add a new API version, keep the old version served while clients and stored objects transition, and choose exactly one storage version. If the representations have different semantics, deploy the conversion service first and use webhook conversion; a no-conversion strategy is suitable only when the representations are effectively identical apart from the API version. Run a controller that understands both forms, handle missing fields safely, move clients to the new version, then make it the storage version. Existing objects are not rewritten automatically, so explicitly rewrite them through the API. Monitor storedVersions and old-version API usage before disabling the old version. Test round trips in both directions, preserve data that has no direct equivalent, and take backups before rewriting objects.
A single storage version with staged migrations can be a good alternative to a conversion webhook. It requires more coordination, but it reduces operational complexity: introduce compatible fields, deploy readers and writers that understand both shapes, migrate existing objects, and only then remove the old form. A webhook is worth the cost when several served versions must remain compatible or when the representations have real semantic differences; otherwise, a carefully staged migration is often easier to operate.
Before rollout, run both controller binaries against the same fixtures, including objects that the newer controller has already modified. Verify that the older binary can still read, reconcile, and update those objects, and test missing defaults, changed status values, unknown fields, downgrade behavior, and conversion failures.
If the versions are genuinely incompatible, make them separate API versions and let them coexist during the transition. The old controller can continue handling only its version, while the newer controller supports the new version and conversion between representations. Data that exists in one version but has nowhere to go in the other must be preserved or the migration is lossy.
That clean versioned approach is much safer than changing one version in place. Renames and semantic changes without a new version and conversion logic are where rolling deployments tend to get dangerous.
The round-trip and rollback tests are especially important. If the newer representation drops information, converting back to the old version can silently produce a different object, so rollback is not actually safe.