I want to catch filter regressions in CI before pipeline changes are merged. The production inputs and outputs include services that are awkward to run locally. Is there a lighter way to feed in sample events and check the results without bringing up a full stack?
How do you automate tests for Logstash filters?
Asked by
after-2nd-intermission
Answers
Split the filter block into its own configuration file and keep the production inputs and outputs separate. In CI, combine the filters with a test input and a JSON stdout output. Feed in fixtures, parse the resulting events, and assert on the fields you care about with your usual test framework.
That keeps Kinesis and Datadog out of the filter tests. It won't prove those production integrations work, so treat those as a separate test concern.
Reply to this answer
Take another look at logstash-filter-verifier. It is designed for testing filters against sample events, and the older release history doesn't necessarily mean it can't work with your configuration. Try it against your Logstash version and a small fixture before committing to it.
Reply to this answer
A minimal Logstash container is another reasonable CI test. Give it a file input with representative events, capture the output, and compare the parsed fields with expected results. You still run Logstash, but you don't need Elasticsearch or the rest of the stack just to catch filter regressions.
Reply to this answer
If changing tools is on the table, Vector has a tap command for inspecting pipeline results. That is more of an alternative to investigate than a way to automate tests for an existing Logstash configuration, though.
Reply to this answer
Add your answer
Answers are reviewed before publication.
NO APPOINTMENT NEEDED:
A stdin or generator input can work for this too. A file input is handy when the fixtures are already stored as files.