I maintain an operator that runs distributed load tests as Kubernetes Jobs. For years, tests could hang because a Prometheus exporter ran beside the main container and kept serving metrics after the test completed. Kubernetes Jobs originally had no native way to express "start this helper early, but stop it when the main workload exits," so users had to call a shutdown endpoint from their load test.
Native sidecars now solve the lifecycle problem. An init container with `restartPolicy: Always` starts before regular containers, remains running while they do, and receives termination when the regular containers finish. The important detail is that this is expressed through the per-container restart policy—there is no separate `sidecar: true` field. If that field disappears during a chart or controller refactor, the container becomes an ordinary init container and can block the Pod indefinitely.
After switching to native sidecars, my controller began reporting some successful Jobs as failed. The sidecar receives SIGTERM during normal teardown and may exit with code 143, or 137 if it is force-killed. My health logic treated any non-zero terminated exit code as a failure, and races between Pod status updates and Job completion made the problem intermittent.
I now ignore terminated exit codes for any container whose restart policy is `Always`, rather than special-casing the metrics exporter by name. Waiting states such as `CrashLoopBackOff`, `ImagePullBackOff`, and `CreateContainerConfigError` are still treated as failures. This also handles injected proxies and other sidecars that the operator does not know about.
For anyone writing controllers that inspect container status: is this the usual approach for native sidecars, or is there a more standardized way to distinguish an intentional sidecar shutdown from a real failure?
3 Answers
Before native sidecars were available, a practical workaround was to share an `emptyDir` volume between containers. The main process can create a marker file from an EXIT trap, while the sidecar watches for that file and terminates its own process cleanly. It is awkward shell glue, but it works for proxies and other helpers that must run inside the Pod.
The sidecar itself should also handle SIGTERM and exit cleanly when it receives it. The termination grace period gives it time to stop accepting new work and finish active requests before Kubernetes sends SIGKILL. That improves shutdown behavior, but the controller still needs to recognize that the resulting termination code is expected for a native sidecar.
The main-container completion event should remain the source of truth for whether the workload succeeded. Sidecar termination is a consequence of that completion, not an independent test result.
Ignoring non-zero exit codes for containers declared with `restartPolicy: Always` seems like the right structural rule. A native sidecar is expected to receive SIGTERM during normal Job teardown, so 143 or 137 should not make the workload fail. I would still evaluate Waiting states and startup failures separately, since those indicate problems before normal shutdown.
That also avoids coupling the controller to a particular exporter name. An injected service-mesh or database proxy has the same lifecycle semantics and should be handled the same way.
For metrics specifically, pushing to a gateway can avoid keeping an exporter alive in the Job. That is less useful for authentication proxies or other components that need to be colocated with the application.