Our pipeline platform is almost entirely schedule-based: workflows run at fixed times and assume upstream data has arrived. When it has not, we either retry blindly or wake someone up. I want workflows to start from real events instead—for example, a Kafka message triggering a transformation, an S3 object arrival starting ingestion, or a payment-provider webhook launching reconciliation. I have pieced together some of this with Lambda triggers and SQS, but the event-handling layer is becoming difficult to maintain. Is there a workflow orchestration tool that supports event-driven starts alongside scheduled workflows without requiring us to build and maintain a custom event router?
3 Answers
Temporal is worth a look. It supports starting workflows from application events as well as scheduled or cron-based runs, and it gives you durable execution and retries. Make every workflow start and activity idempotent, though—object-storage notifications can be delivered more than once for the same file.
If you are running on Kubernetes, Argo Events paired with Argo Workflows is a fairly direct fit. Argo Events can consume things like webhooks, queues, and cloud events, then trigger an Argo workflow. It keeps the event wiring and workflow execution separate without forcing everything through cron.
Restate is another option if you prefer defining workflows in application code instead of YAML-heavy platform resources. It handles durable execution and event-driven workflow starts, and you can use NATS or a similar messaging system for communication between services. KEDA can also be useful when the main requirement is scaling workers or launching Kubernetes jobs from queue depth, but it is more of an event-driven scaling mechanism than a complete workflow engine.
The useful search terms here are durable execution and workflow engine. There are plenty of choices, but the deployment and programming model matter as much as the trigger support.