We're moving production workloads to shell-less, distroless images such as Chainguard, so the images contain no shell or troubleshooting tools. Ephemeral containers through kubectl debug work technically, but the auditing story is incomplete. Kubernetes audit logs and events already capture who attached a debug container, which pod was targeted, when it happened, the image used, and the security context. The missing piece is the actual commands and activity performed inside the session.
Teleport looks like a strong fit because it can record sessions at the access proxy and provide short-lived access requests, approvals, and moderated sessions, but the required features may only be available in its Enterprise offering. A bastion with terminal recording has also come up as a possible alternative.
For teams that permit kubectl debug in production, do you record the full session, or do you rely on the Kubernetes audit event plus an incident or ticket reference? If you have used Teleport or another access broker for Kubernetes, were the recordings useful during real investigations, or did they mainly become compliance artifacts? If you moved away from a solution, what caused that decision?
The environment is multi-account and multi-tenant EKS. I'm mainly interested in practical experience, including operational overhead and limitations, rather than vendor marketing.
5 Answers
If full recording is too expensive, a reasonable baseline is tightly scoped RBAC, short-lived credentials, mandatory approval, a ticket or incident reference, restricted debug images and security contexts, and complete Kubernetes audit logging. You can also require the entry command to launch a recorder such as script, but that only works if users cannot replace or bypass the command and if the resulting transcript is reliably exported. It is weaker than proxy-level recording, but may be enough for lower-risk environments.
Some Kubernetes access-broker projects can provide just-in-time access and approval workflows, but verify whether they record commands inside exec sessions or ephemeral containers. Capturing the API request that creates a debug container is not the same as recording activity after the container starts. That distinction is easy to miss when evaluating tools.
The main tradeoff is whether transcripts are useful enough to justify the operational burden. Teams often collect them for compliance but rarely review them unless an incident occurs. Before investing heavily, define which investigations need command-level evidence, how recordings will be indexed and retained, and who is allowed to access them. That usually makes it clearer whether a full broker, a bastion recorder, or event-and-ticket auditing is appropriate.
Teleport is one of the more complete solutions for this. It can record sessions, integrate with different storage backends, and support short-lived access and moderated sessions. The downside is cost, especially when the features needed for production governance are limited to higher tiers. Self-hosting may help with deployment control, but it doesn’t necessarily remove the licensing concern.
A bastion or jump host using terminal recording tools such as tlog can capture the interactive session, but it changes the access path. You would need to ensure every production debug workflow goes through that host and that users cannot bypass it with direct Kubernetes credentials. It can be a practical lower-cost option, though it usually requires more policy and integration work than an access broker designed for this use case.
The moderated-session feature is particularly useful in regulated environments because another person can observe the work live, stop the session, and review what happened afterward.