We're moving production workloads to shell-less, distroless images such as Chainguard, so the images contain no shell or troubleshooting tools. Ephemeral containers launched with kubectl debug are the practical replacement and work technically, but session auditing is the unresolved issue.
We already export Kubernetes audit logs and events, so we can determine who attached a debug container to which pod, when it happened, which image was used, and what security context was requested. What we cannot currently see is what the operator actually ran after entering the container.
Access brokers such as Teleport appear to address this with recorded sessions, short-lived approvals, and moderated access, but those capabilities may require an expensive tier. I'm trying to understand what other teams do in production: Do you record the actual terminal session, or do you consider the Kubernetes audit event plus an incident or ticket reference sufficient? Has anyone used Teleport or a similar Kubernetes-focused solution, and were the recordings useful during real investigations or mostly kept for compliance? If you moved away from one, what caused the change?
Other possibilities we're considering include a bastion or jump host using terminal recording, lightweight access brokers, and enforcing a wrapper such as script for debug commands. We operate multi-account, multi-tenant EKS, so isolation and the access-broker workflow matter. I'm mainly interested in practical experience rather than vendor recommendations.
4 Answers
Teleport is one of the few options that ties the approval workflow, short-lived access, Kubernetes connectivity, and session recording together without a lot of custom glue. It can also support self-hosting and multiple storage backends. In environments with stricter controls, moderated sessions are useful because another person can observe, approve, or terminate the session. The downside is cost and the effort involved in getting pricing, so it can feel excessive if recordings are rarely reviewed.
There are smaller Kubernetes access and approval tools that cover just-in-time permissions, but command-level recording is often the missing feature. Before adopting one, verify that it captures commands and terminal output inside both exec sessions and ephemeral containers, not merely the request to attach the container. That distinction is easy to miss in a product demo.
Some teams decide that Kubernetes audit events plus a required incident or change-ticket reference are enough. That is simpler and avoids collecting sensitive command output, but it does not prove what was actually executed. If you take this route, tightly restrict who can create ephemeral containers, use short-lived access, require an explicit reason, limit approved images and namespaces, and alert on production debugging activity.
A forced recording wrapper can help for ordinary exec sessions, but it is not a complete control by itself. It only works if every access route invokes the wrapper, and it may miss commands run through alternate shells, tools, or direct API calls.
A bastion or jump host with terminal logging, such as tlog, can provide a simpler recording layer. The important part is making sure every debug path goes through it; otherwise users can bypass the recording by calling the Kubernetes API directly. You would still want Kubernetes audit logs for identity, pod, image, and security-context details, then correlate those events with the terminal transcript.
The main value seems to be having a complete, searchable record when an incident happens, rather than expecting someone to inspect every recording routinely. The approval and moderation controls may be as important as the transcript itself.