I’m comparing Vector with Fluent Bit for a Kubernetes logging pipeline. With a Kubernetes logs source, a small transform, and a Kafka sink, Vector is using around 400 MiB versus about 120 MiB for Fluent Bit. Even a minimal setup with internal metrics enabled and no log traffic uses roughly 200 MiB. What should I check, and what kind of memory use is reasonable?
Why is Vector using so much memory as a DaemonSet?
Asked by
m3__Clenby
Answers
Check how many threads Vector is starting. Its memory use can scale with thread count, which makes the same configuration look much heavier on larger nodes. Try setting VECTOR_THREADS to a lower value, then compare memory under the same workload. That brought the comparison much closer in the case described here.
Reply to this answer
Before comparing numbers, make sure both collectors are measured the same way and under the same conditions. Compare idle with idle, then run the same log volume and transforms. Also check whether the reported figure is resident memory or a container-level metric, and whether the limits and workload are identical. A single idle snapshot can be misleading.
Reply to this answer
Keep an eye on bursts, not just steady-state usage. One concern raised about Vector is that memory can spike when log volume surges, especially if downstream delivery cannot keep up. Test a busy period and watch the sink, buffers, and memory together before settling on a limit.
Reply to this answer
If the goal is deciding which collector to run, benchmark your actual pipeline rather than relying on a generic comparison. The transforms, output behavior, traffic pattern, and node size all matter. Vector may still be worth the extra footprint if its transform language and testability simplify maintenance, but measure that tradeoff on your own workload.
Reply to this answer
Add your answer
Answers are reviewed before publication.
m3__Clenby:
That was the issue for me. Lowering VECTOR_THREADS brought memory use much closer to Fluent Bit.