I'm working on some plugins for a system that utilizes OAuth2 with the client_credentials grant flow. When a plugin is installed, it registers via a webhook, allowing me to use an API gateway resource tied to a Lambda function for handling the registration. I need a reliable place to store client API credentials, specifically a client_id and client_secret, which remain unchanged unless the plugin is deleted and reinstalled. Since the generated bearer token lasts 12 hours, I typically cache it and use the cached version until it expires. When the token expires, I'll make a new request to generate it, cache the new token, and hold onto it for the duration of the job.
Initially, I stored these credentials in Secrets Manager, which seemed ideal, but my expenses increased significantly with the number of clients I have. I then switched to SSM Parameter Store based on various recommendations, which worked fine initially and cut costs down considerably. However, I've started facing API throttling issues since I have approximately 130 clients whose jobs can all trigger simultaneously. The current limits on the GetParameter and PutParameter requests are very restrictive, which creates a bottleneck.
For my new project, clients will set their schedules for job triggers, and I need a solution that ensures immediate start times without delays, possibly running every 5 minutes. I'm exploring other options like DynamoDB or S3 for storing these credentials safely, ensuring they remain encrypted at rest and are quickly accessible from Lambdas and Docker containers. Any suggestions on the best storage solution?
4 Answers
You might be overcomplicating things! Given your situation, simply using a JSON file stored in S3 and mounted to your Docker container might suffice. Remember, the data can be in plaintext in memory while your app runs anyway, so overthinking the encryption might not be necessary.
Check out this article on asking better questions! It might give you some insight on refining your queries better: https://medium.com/@rickharrison_/the-art-of-asking-better-questions-4312b5d469e0
Actually, the initial question is well articulated, covering most of the essentials.
DynamoDB is a solid choice for your needs. It’s built to handle high concurrency with key-value lookups, has great throughput options, offers encryption at rest, and could save you some costs compared to working around SSM limits.
I’d also recommend DynamoDB based on what you've described. It's well-suited for high-frequency access that'll be necessary for your setup.
How do you think you could improve the original question?