I've been diving into using controller runtime, and I'm running into a pretty confusing issue — when a controller makes updates, it doesn't immediately see its own changes in the local cache. For instance, let's say FooController is reconciling a resource named "bar" and updates it using Patch(). After that, when the same resource is reconciled again, the updated version isn't showing up in the local cache. This behavior is problematic for certain use cases where I need the controller to see its own updates right away.
I considered having the controller wait for the update to appear in the cache before returning a response, but since a watch isn't feasible here, I thought polling could work. However, I'm struggling to figure out how to confirm that the new version is in the cache since I can't rely on the generation field, as it only changes when the spec does. Checking the resourceVersion seems risky because other changes could impact it after the controller has already done the patch. While I think checking if the resourceVersion has changed might solve the problem most of the time, I'm wondering if there's a more reliable method to ensure my update is confirmed in the local cache without running into stale reads from other updates. Any ideas?
1 Answer
You really can't base reconciliation on the current state of the fetched object since things can change while you're processing. What I suggest is to use a unique annotation or a specific status field tied to the operation you're trying to track. Polling isn't a bad approach either; many projects, including Crossplane, use polling for various reasons, especially since other clients might modify the object between your reconciliations.
But what I'm really focusing on is ensuring that when FooController runs its second reconcile, it acknowledges the updates from the first reconcile. Other client updates aren't my concern in this scenario.