I wrote a Bash script to retrofit a default-deny configuration onto existing application load balancers. It forces HTTPS, rejects invalid headers, enables defensive desynchronization protections, and sets the HTTPS listener's default action to return 403 so only explicitly configured host-header rules can forward traffic.
I tested it against an older HTTP-only load balancer with a listener on port 80 forwarding to a target group. The script created an HTTPS listener, warned that it could not find a default forward target group, and then reported success with exit code 0. In reality, the application became unreachable: allowed hosts received 403, and the original target group was no longer attached to a load balancer.
The problem was ordering. The script created the HTTPS listener with a 403 fixed-response default, then inspected that new listener to discover the target group. Since the listener had no forward action yet, the lookup returned nothing. The script skipped creation of the host-header rule, printed a warning, and made the deny action permanent.
The fix was to capture the target group from the existing HTTP listener before making changes, create the allow rule before switching the default to deny, and abort with a nonzero exit code if no target group can be identified. I also added dry-run support and external verification for allowed-host routing, unknown-host denial, and HTTP redirects, with automatic rollback if verification fails.
The incident was not caught by shell linting or failed API calls. Every command succeeded, but the resulting infrastructure state was still an outage. What testing strategies work well for finding sequencing bugs like this across many infrastructure shapes without manually recreating every production variant?
3 Answers
Treat any lookup that can legitimately return an empty result as a hard failure when the next operation depends on it. A warning is not protection if the script continues and then exits successfully.
For live listener changes, I would never switch the default action to deny until a request through the load balancer proves that the intended host rule works. A dry-run mode that prints the planned calls is also useful, especially for older infrastructure shapes. It would have exposed the missing target group before the first write.
The exit code should represent the verified invariant, not merely the success of the API mutations. After applying the changes, probe the load balancer with an allowed host, an unknown host, and the HTTP redirect, then compare the responses with an expected matrix. If verification fails, return nonzero and restore the previous default action.
A plan/apply split is the stronger structural fix. The plan phase should read all relevant state and calculate the complete desired configuration without changing anything. For an HTTP-only load balancer, it would visibly show a deny default with no allow rules, which should stop the apply phase before an outage occurs. The deny action should be the final mutation, after all allow rules are in place.
I implemented the probes and rollback. The script now checks an allowed host, an unknown host, and the HTTP redirect, and those results determine the exit code. It also snapshots the old default and restores it if verification fails.
The probe itself exposed more issues: curl failures could produce duplicated 000 status text, and an unreliable resolver caused false failures until requests were pinned to resolved load-balancer addresses. I still need to finish the full plan/apply split, but that should prevent this class of ordering error rather than only detecting it afterward.
This is primarily a state-transition testing problem, so static analysis will have limited reach. Build fixtures for infrastructure shapes that violate your assumptions: HTTP-only listeners, missing target groups, multiple listeners, existing fixed responses, partially configured host rules, and internal load balancers. Run the plan against each fixture and assert that it either produces a safe complete plan or refuses to apply.
For mutating tests, use black-box checks after every apply. Confirm both positive behavior, such as the allowed host reaching its target, and negative behavior, such as an unknown host receiving the deny response. Also assert ownership and attachment relationships, not just that individual API calls returned successfully.
That is now the order: apply the allow rule, verify it through the load balancer, and only then activate the deny default. Empty target-group results also stop the script with a nonzero exit instead of becoming warnings.
One caveat is that internal load balancers cannot be verified from outside their network. For those, the script requires an explicit no-verify option and clearly states that traffic flow was not proven rather than claiming success.