What works for event-driven pipelines instead of cron and polling?

November 16, 2026 5 answers
Asked by 8k+chalkwinder_503

Our pipelines mostly run on schedules and hope the upstream data has arrived. I'd like to trigger workflows from Kafka messages, S3 uploads, or payment webhooks without maintaining a custom event router. What tools work well alongside scheduled jobs?

Answers

2v!lilaccrank_639

For Kubernetes, Argo Events plus Argo Workflows is worth a look. Argo Events can consume from SQS and start a workflow to process the event. One possible path for an S3 upload is S3 to SNS or EventBridge, then SQS, Argo Events, and Argo Workflows.

There are several pieces in that chain, so check the operational overhead against what you already run. Also plan for workflow concurrency and API request limits as volume grows.

Reply to this answer
8r+ticketspanner_315

Temporal is a stronger candidate when the workflow itself is complicated, with retries, failures, and dependencies to manage. It supports scheduled workflows too, though you'll still need to connect external events to workflow starts.

Make those starts idempotent. S3 notifications can be duplicated, so an object and version identifier can be part of the workflow key. For payments, use the payment event ID or another stable identifier. Moving the trigger from a clock to an event doesn't make duplicate processing disappear.

Reply to this answer
q.417+tramcrayon

If the work is fairly simple and you're already on Kubernetes, look at KEDA with a queue such as SQS or RabbitMQ. It can scale workers based on queue activity. That solves a different problem from full workflow orchestration, but you might not need a workflow engine for every ingestion task.

Reply to this answer
TinselLogic

Another search term is durable execution. Restate is an option in that space, with NATS as a possible way to communicate between processes. Scheduled jobs can invoke workflows too, so you don't have to replace every schedule on day one.

Reply to this answer
7e!modratchet_528

Before picking a tool, map out what happens when events arrive twice, out of order, or while a previous run is failing. For a small pipeline, a queue and a worker may be enough. Once you're coordinating dependencies and recovery across several steps, a workflow engine becomes more appealing.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions