Our pipelines mostly run on schedules and hope the upstream data has arrived. I'd like to trigger workflows from Kafka messages, S3 uploads, or payment webhooks without maintaining a custom event router. What tools work well alongside scheduled jobs?
What works for event-driven pipelines instead of cron and polling?
Answers
For Kubernetes, Argo Events plus Argo Workflows is worth a look. Argo Events can consume from SQS and start a workflow to process the event. One possible path for an S3 upload is S3 to SNS or EventBridge, then SQS, Argo Events, and Argo Workflows.
There are several pieces in that chain, so check the operational overhead against what you already run. Also plan for workflow concurrency and API request limits as volume grows.
Reply to this answer
Temporal is a stronger candidate when the workflow itself is complicated, with retries, failures, and dependencies to manage. It supports scheduled workflows too, though you'll still need to connect external events to workflow starts.
Make those starts idempotent. S3 notifications can be duplicated, so an object and version identifier can be part of the workflow key. For payments, use the payment event ID or another stable identifier. Moving the trigger from a clock to an event doesn't make duplicate processing disappear.
Reply to this answer
If the work is fairly simple and you're already on Kubernetes, look at KEDA with a queue such as SQS or RabbitMQ. It can scale workers based on queue activity. That solves a different problem from full workflow orchestration, but you might not need a workflow engine for every ingestion task.
Reply to this answer
Another search term is durable execution. Restate is an option in that space, with NATS as a possible way to communicate between processes. Scheduled jobs can invoke workflows too, so you don't have to replace every schedule on day one.
Reply to this answer
Before picking a tool, map out what happens when events arrive twice, out of order, or while a previous run is failing. For a small pipeline, a queue and a worker may be enough. Once you're coordinating dependencies and recovery across several steps, a workflow engine becomes more appealing.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions