Our pipeline platform is almost entirely schedule-based: workflows run at 2 a.m. and assume upstream data has arrived. When it has not, we either retry blindly or wake someone up. I want events to start workflows directly—for example, a Kafka message should trigger a transformation, an S3 object arrival should start ingestion, and a payment-provider webhook should launch reconciliation. I have pieced some of this together with Lambda-style triggers and queues, but the resulting event-routing logic is becoming difficult to maintain. Is there a workflow orchestration tool that supports event-driven starts as well as scheduled workflows without requiring a custom event router?
4 Answers
Restate is worth looking at if you want a code-first workflow engine rather than a Kubernetes-heavy setup. You define the workflow in normal application code, while the platform handles durable execution and recovery. NATS can also work well for communication between services, with scheduled jobs simply starting the same workflows when needed.
If you are running on Kubernetes, Argo Events combined with Argo Workflows is the most direct option. Argo Events can listen for things like Kafka messages, webhooks, and object-store notifications, then submit an Argo Workflow. It keeps the event triggers and workflow definitions separate without requiring you to build the whole routing layer yourself.
Temporal is a strong fit here. It supports starting workflows from application events as well as cron or scheduled runs, and it gives you durable execution, retries, and state management. Make every workflow start and activity idempotent, though—object-store notifications can be delivered more than once for the same file.
KEDA can scale consumers or jobs based on Kafka, queue, and other event sources, so it is useful for event-driven activation in Kubernetes. I would treat it as an event-aware autoscaling and job-triggering component, not a complete replacement for a durable workflow orchestrator. You may still need a workflow engine for retries, dependencies, and long-running state.
The useful search terms are durable execution and workflow engine. There are plenty of choices, but a code-first tool like Restate can be relatively simple to deploy and operate.