I'm trying to find ways to minimize logging costs with Datadog, especially since my log volumes and usage patterns can change frequently. Although aggressive filtering and keeping indexed logs for a minimal duration seem like potential solutions, I've heard they might not be effective. I want to establish a good balance between getting enough useful data and avoiding excessive noise in my logs. Has anyone faced a similar challenge, and how did you manage to address it?
2 Answers
Datadog's quite valuable whether you're a small startup or a large enterprise. If you're managing medium to large operations, consider self-hosting your logs to save costs. You can negotiate favorable terms as your volume increases. Instead of relying solely on Datadog, you might look into developing a solid data pipeline with smart filters and retention strategies. Also, I’ve been thinking about AI log filtering, but it might end up being just as costly as paying more for Datadog.
Before deciding to stick with Datadog, think about how critical your logs are for your operations. Are they indispensable for resolving incidents or running analytics? If log correlation with other telemetry through Datadog is necessary, switching to a different tool might not solve the underlying issue of inconsistent log volumes. Even then, make sure you have a strategy to clear out unnecessary logs without losing crucial data.
Absolutely, I work in e-commerce, and my logs are crucial for quick incident resolution and analytics. While an alternative log management setup is possible, I really want to maintain the connection with Datadog for all the telemetry insights.