I wrote a Bash script to retrofit a default-deny posture onto existing AWS Application Load Balancers. It forces HTTPS, rejects invalid headers, enables defensive desynchronization protections, and changes the HTTPS default action to return 403 so only explicitly configured host-header rules can forward traffic.
I tested it against an older HTTP-only ALB with a listener on port 80 forwarding to a target group. The script created an HTTPS listener with a 403 fixed-response default, then tried to discover the existing forward target group by inspecting that new HTTPS listener. Since the new listener had no forward action, the target group lookup returned empty, the host-header rule was never created, and the script continued anyway. It printed a warning and exited 0, leaving the application unreachable.
The individual API calls all succeeded, and the final ALB state matched what the script technically requested, but it was still an outage. The fix was to capture the target group before making changes, use the HTTP listener as a fallback, create the allow rule before switching the default action to deny, and abort with a nonzero status if no target group is found. I also added dry-run support.
The broader lesson is that warnings after a destructive change are not safeguards, and success should be based on verifying the intended traffic behavior rather than merely confirming that API calls returned successfully. I'm now adding black-box checks for an allowed host, an unknown host, and the HTTP redirect, along with automatic rollback if verification fails.
What are good ways to test this class of infrastructure mutation? Running an ugly, production-like sandbox and probing it externally works, but I'd like something that scales across more starting-state permutations.
2 Answers
Treat any empty result that could cause a destructive decision as a hard failure, not a warning. A missing target group should stop the script before the default action changes, with a nonzero exit and a reason that explains what was missing.
For a live retrofit, I’d create the forwarding rule first, verify that an allowed-host request reaches the expected backend, and only then change the default action to deny. A dry-run or plan mode that prints the complete sequence of intended operations is useful as well; it would have exposed the empty rule set before anything was changed.
The verification needs to account for internal load balancers too. If they cannot be reached from the operator’s environment, the script should say that traffic was not verified rather than claiming success. An explicit bypass can exist, but it should be visible and produce a suitably cautious result.
The exit status should represent the post-change invariant, not just whether every cloud API call succeeded. After applying the changes, probe the load balancer for an allowed host, an unknown host, and the HTTP redirect, then compare the results with an expected matrix. Use that verification to determine the exit code.
A plan/apply split is the stronger structural fix. The planning phase should read all state and calculate the complete desired configuration without mutating anything. For an HTTP-only load balancer, the plan would clearly show a deny default with no allow rules, so it could fail before touching production. The deny action should be the final mutation, after all forwarding rules are in place.
It’s also worth snapshotting the old default action and automatically restoring it if verification fails. That turns a successful-but-wrong sequence into a short-lived failure with a clear nonzero result.
I implemented the probe and rollback approach. It now checks an allowed host, an unknown host, and the port 80 redirect, and restores the previous default action if the checks fail. I also found that probing by the load balancer hostname could produce false failures because of resolver behavior, so the verification resolves addresses separately and pins requests to them.
The plan/apply split is still the larger fix because it makes this ordering bug impossible instead of merely detectable. I’m applying the same end-state assertions to my other mutation scripts too; several of them were reporting success when they had only confirmed that commands ran.
That’s now the order in the script: snapshot the old default, install the rule, probe the allowed and denied paths, and only then keep the deny default. Empty lookups fail immediately instead of generating a warning and continuing.
The dry-run mode prints the planned operations, but the probe is still needed because a syntactically correct plan cannot prove that traffic actually follows the intended path. For internal load balancers, the script now reports that verification was skipped unless it can run from a suitable network location.