I'm struggling to get my AWS EC2 instance set up with Bindplane running on Docker to properly upload logs to S3. Here's my current setup:
1. I'm running an EC2 instance with Docker that hosts a Bindplane container.
2. The EC2 instance profile has permissions to perform S3 Get/Put operations.
3. I've configured the AWS CLI credentials on my local machine.
4. My ~/.aws/config file includes:
- role_arn = arn:aws:iam::xxxxxxxxxxxxx:role/xxxxxxxx-role
- credential_source = Ec2InstanceMetadata
- region = us-east-1
I can successfully run commands like "aws sts get-caller-identity" and "aws s3 ls" on my local machine. Inside the container, I can get the instance ID by running `curl http://169.254.169.254/latest/meta-data/instance-id`.
Despite all of this, I'm seeing a 403 error in the Bindplane logs when it tries to upload:
"operation error S3: PutObject, https response error StatusCode: 403, api error InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records."
I've also tried with both IMDS v1 and v2 settings in case that made a difference, though I can query instance metadata with v1 but not v2, despite having the hop count set to 2. Any insights would be really appreciated!
2 Answers
You might want to check CloudTrail for more details on the failed S3 call. That error usually suggests the container is using a different credential than expected. Try running "aws sts get-caller-identity" inside the container for a clearer debug.
It sounds like the access key isn't recognized because the container might not have the right credentials passed to it. Double-check if the container already has credentials set up. You should be looking for the ID that appears to be missing from their records to confirm if it's what you expect or not.
According to AWS documentation, my instance profile role on EC2 should automatically be assumed by the containers during S3 operations, so I haven’t passed any credentials directly to the container.
I'll give that command a try inside the container. We don't have data events enabled, so I'm not sure if I'll see the attempt in CloudTrail.