What’s the best way to test Logstash pipelines in CI?

If you’re able to reconsider the technology, Vector is worth evaluating because its tap functionality makes pipeline output easy to inspect. If Logstash is required, a simpler approach is to replace the production output with the stdout plugin and capture or parse that output during the test. Splitting production inputs and outputs away from the filter logic makes this much easier.

The logstash-filter-verifier project can still be useful even though it hasn’t had a recent release. The configuration format it targets hasn’t changed much, and we still run it in CI. We separated the pipeline so the input and output sections are in one file while the filter block is reusable on its own. Tests then use a generator or standard input and send results to the stdout rubydebug output instead of touching Kinesis or Datadog. From there, we feed in fixture events and assert against the emitted JSON without starting a full ELK stack.

We use integration tests with a minimal Logstash container. The tests provide sample events through a file input, wait for the pipeline to process them, and compare the resulting output with expected data. It adds some startup time, but it has been effective at catching filter regressions in CI.