I maintain an operator that runs distributed load tests as Kubernetes Jobs. For years, a metrics exporter sidecar kept the master Job from completing because Kubernetes required every regular container to exit. The workaround was to call the exporter's shutdown endpoint from the load test when it finished.
Kubernetes 1.29's native sidecars finally solve the lifecycle problem: an init container with restartPolicy: Always starts before the regular containers, stays running while they run, and receives SIGTERM when the main containers finish. However, this exposed another issue in my operator. The sidecar exits with code 143 or 137 after the intentional SIGTERM, and the controller was interpreting any non-zero terminated exit code as a failed test. That also needs to work for injected sidecars such as service-mesh or database proxies, so I don't want to special-case one container by name.
My current rule is to ignore exit codes from containers whose restartPolicy is Always, while still treating Waiting states such as CrashLoopBackOff, ImagePullBackOff, and CreateContainerConfigError as failures. Is this the usual approach for controller health logic, or is there a more standardized way to distinguish the main container's result from an intentionally terminated native sidecar?
3 Answers
The key distinction is structural rather than based on the container name. A native sidecar is an init container with a container-level restartPolicy of Always, so its SIGTERM-related exit code should not determine the Job result. Ignoring Terminated exit codes for those containers while continuing to report Waiting failures seems like a sensible general rule, including for sidecars injected by a mesh or another platform component.
Before native sidecars were available, a reliable workaround was to use a shared emptyDir as a completion signal. The main container writes a marker file from a shell EXIT trap, and the sidecar watches for that file before terminating its own process. It is awkward, but it keeps the lifecycle coordination inside the pod and works for proxies that cannot be replaced with a metrics push mechanism.
For an exporter that only exists to expose metrics, pushing the results to a metrics gateway can avoid the sidecar lifecycle problem entirely. That does not help with components that must stay next to the workload, such as authentication or database proxies. Those should handle SIGTERM and exit cleanly within terminationGracePeriodSeconds; the controller still needs to recognize that their termination is expected rather than treating it as the workload's result.
That was my conclusion too. The important part is not to ignore every non-zero status: only intentional termination of a recognized native sidecar should be exempted, while startup and waiting failures still need to surface.