I'm trying to understand how teams handle production patching for EC2 at scale. Do you use AWS Systems Manager Patch Manager, patch policies, configuration management, or another approach? I'm especially interested in whether you patch instances in place or replace them with newly built AMIs, how you schedule maintenance and minimize downtime, how Auto Scaling Groups are handled, and what rollback looks like when a patch causes problems. I'd also like to know how thoroughly patches are tested in development or staging, and which steps are automated versus manually approved.
5 Answers
A common production workflow is to build AMIs with Packer or a similar tool, keep the image definition in version control, and deploy through Terraform or another infrastructure-as-code system. Test the image in development and staging, then use a blue-green or rolling deployment in production. For urgent security patches, trigger the same pipeline earlier instead of manually changing live instances. The old environment stays available long enough to support a quick rollback.
The preferred approach is immutable infrastructure: build a new, patched golden AMI from code, deploy it as a new instance group, move traffic over, and retire the old group. This avoids machines slowly turning into unique, undocumented systems. With Auto Scaling Groups, you can roll out a new launch template version using an instance refresh, health checks, and a conservative rollout across Availability Zones. If something breaks, point the group back to the previous launch template or AMI.
Some teams also replace instances on a regular schedule even when there is no new patch, such as every few weeks. That limits configuration drift and ensures the fleet continually comes from a recent, reproducible image. The important distinction is to treat servers as replaceable capacity rather than long-lived machines that are modified indefinitely.
Not every machine can be replaced easily. Stateless application fleets are good candidates for golden AMIs and blue-green deployments, but databases, legacy services, and systems that depend on local disk state may still require in-place patching. For those, Systems Manager Patch Manager with defined maintenance windows, reboot controls, and approval rules can work well. Those exceptions should be treated as technical debt and gradually made disposable where possible.
For systems that must be patched in place, use a rolling process rather than updating the entire fleet at once. Patch one instance or one Availability Zone first, verify health and application metrics, and then continue. Schedule routine updates during a low-traffic window, test patches before production, and keep a rollback plan such as restoring the previous kernel or replacing the instance from the last known-good image. Automation should handle the standard path, with manual approval reserved for risky or urgent changes.
How do you handle worker nodes in a production container cluster? Do you rely on managed node-group upgrades to replace them with updated AMIs, or do you run a separate patching process for the underlying instances while keeping downtime low?