We have a collection of Bash, Python, and Groovy scripts for operational tasks such as draining nodes, terminating stuck pods, performing database failovers, and handling other incidents. Standard playbooks cover the basics, but we still need to run custom logic from our own repositories. The difficult part is enforcing who can execute a particular script in production, requiring approval for sensitive actions, and retaining an immutable record of the command, identity, parameters, and output. Giving engineers unrestricted SSH or Kubernetes credentials is not acceptable. How have you built this execution boundary? Did you create an internal runner or CLI that handles authentication, approvals, and auditing before invoking scripts, or adopt a larger platform for that purpose?
3 Answers
We use pipelines with a dedicated pipeline identity, and only allow execution from specific, reviewed commit hashes so someone cannot quietly replace the script before running it. An operator must be temporarily eligible to start the job through just-in-time access, and the appropriate people approve it. During overnight coverage, the on-call manager can approve a limited set of low-risk actions during defined hours; anything outside that policy is escalated to the engineer on call. The most important design choice is making destructive operations impossible or requiring a much stronger approval path. Keep the procedure and policy in Git so the whole process is reviewable.
If you want to avoid building the whole control plane yourself, an access or session-management platform can wrap scripts and hosts with centralized RBAC and session auditing. That can provide identity-aware access and recordings across Kubernetes, virtual machines, and databases. I would still restrict it so operators can launch only approved playbooks through CI/CD, use separate executor identities, and apply least privilege on the target systems. The platform should be treated as another privileged component that needs hardening and independent review, not as a replacement for careful job design.
A practical approach is to keep every script in version control and execute it through your CI/CD system. Put RBAC around the jobs or folders, use dedicated service accounts with narrowly scoped permissions, and retrieve their credentials from a secrets manager rather than handing them to operators. Sensitive jobs can have an approval gate, while the pipeline records the commit, requester, approver, output, and result. This also makes it easier to remove permissions later. The RBAC may be provided by the CI system itself, such as permissions on individual jobs or folders, rather than only by Kubernetes.
That helps clarify it—the CI system’s job or folder permissions are part of the RBAC layer, while the service account controls what the operation can actually do.
The commit pinning and just-in-time access are useful details. I’m still interested in how teams represent time-based approval rules without turning the pipeline into a pile of brittle exceptions.