I'm looking at AWS Emulator for local integration tests across CloudFormation and several AWS services. What works well, and where could a passing test be misleading?
How useful is AWS Emulator for local CloudFormation testing?
Answers
For CloudFormation workflows, the reported results are pretty encouraging. Version 1.0.322 handled stack creation, updates and deletion, change sets, rollbacks with event reasons, and termination protection. Express mode and Early Validation also worked in the tests.
There were useful service integrations too, including SNS to SQS, EventBridge to SQS, inline Python Lambda, Step Functions, DynamoDB with a GSI, CloudWatch Logs, KMS and Secrets Manager. That makes it worth considering for fast local workflow tests, rather than as a replacement for every AWS integration test.
Reply to this answer
Check the service API, not just the CloudFormation status. The test found resources marked CREATE_COMPLETE that didn't actually exist through the service API, including CloudWatch alarms, ElastiCache, CloudFront, load balancers and ECS. CloudFront and load balancers also returned synthetic DNS names.
That can produce a green test without the thing you meant to test. For those resources, keep tests against real AWS and verify the behavior you care about.
Reply to this answer
Don't use it to validate IAM policies. The reported behavior with the emulator's own credentials is to allow requests regardless of the policies, so a passing authorization test could give you the wrong answer. Test allow and deny cases against real AWS instead.
Reply to this answer
There's a fair bit of uneven coverage around updates and integrations. The test got NotImplemented for updates involving API Gateway, ECR, WAFv2 and VPC gateway attachments. ACM DNS validation and StackSet instances weren't implemented either.
Also check what your Lambda code calls. The tested Lambda runtime only built SDK clients for a limited set of services, and that set didn't include SNS or S3. A stack can deploy successfully while the function's actual integration still isn't testable locally.
Reply to this answer
The usefulness depends on what a test is supposed to prove. Some see a partial first release as promising, while others find the gaps too limiting. I'd try it for quick functional feedback, then retain real AWS tests for behavior where fidelity matters.
One practical setting to check: telemetry was on by default in the tested version and could be disabled with the --telemetry off option.
Reply to this answer
Add your answer
Answers are reviewed before publication.
4z+brindlecrayon_738:
That distinction helps. A successful stack operation isn't necessarily evidence that the underlying service is behaving like AWS.