Are Step Functions the Right Choice for Managing API Data Ingestion?
Asked by
TechWhiz123
I'm working on an API that's integrated with Cognito, where internal users may add data around 10 times a year. The GET methods are public, but any method that modifies data requires authentication. We're setting up a process to ingest metadata related to files that are stored in S3. Initially, I considered using a Lambda function to call our API's POST endpoints directly, but my early research suggests that AWS Step Functions might be better suited for this task. However, I'm unsure if a Step Function can bypass the authentication process if it's operating within the same AWS account and region. The data we need to ingest is structured in collections, where items can't be added to the database before their respective collections exist. We want to automate this process to load data into the database as files are uploaded to S3, and since the data might come out of order, we also need the ability to retry item loading. I've written some Lambda functions that utilize SQS for passing metadata, but I ran into limitations since some of our metadata exceeds 1MB, making SQS not ideal for this scenario. Now, I'm considering invoking the API directly through a Step Function. Is this the right approach?
Answers
To handle authentication against your API, consider setting the API Gateway endpoint authentication to IAM instead of Cognito. Then, grant the role linked with your process's Lambda or Step Functions permissions to invoke the API. Make sure that when you invoke the API from Step Functions or Lambda, the role used has permission to invoke it. If you go the Lambda route, use the AWS SDK to manage the authentication during the invoke process.
Reply to this answer
It's a mixed bag when it comes to using Step Functions for this. On one hand, Step Functions can’t bypass Cognito’s authentication, so you'll need that set up correctly. However, if the process becomes more complex, Step Functions can be helpful, although they might struggle with retries if something goes wrong. A more straightforward approach could be to trigger a Lambda function directly from an S3 event, which could read the metadata, verify if the corresponding collection exists (with a direct database call), and create it if it doesn’t. Then, it can simply insert the item.
Reply to this answer
Step Functions are excellent for handling error retries! I’m curious about what you mean by struggling with retries. They have built-in mechanisms for that. Can you elaborate?
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions