Hey there! I've been diving into part two of my pipeline series and came across a challenge with observability. Specifically, I'm trying to figure out how to pinpoint a particular error amid tons of log files without incurring high costs for ingestion or specialized storage solutions.
I ended up crafting a system using Azure Synapse Serverless SQL directly on my Data Lake (ADLS Gen2). It feels a bit like a cheat code since I'm able to query files as if they're tables, and the cost is low since I only pay for the queries I run.
I documented the details and code in a blog post: [Building Reliable Data Pipelines [Part 2]](https://medium.com/@yahiachames/building-reliable-data-pipelines-part-2-3e60c160a450).
Now I'm curious whether this method is sustainable long-term. It works well at the moment, but I'm concerned about potential issues like the 'small file problem' in the future. Is anyone using this setup in production, and do you think I should explore other options?
3 Answers
Even if you're considering a custom storage solution, databases aren't typically the best fit for log management. Have you looked into Elastic? It's specifically designed for logging and can effectively handle the fixed requirements that logs present.
Synapse Serverless does have its limitations, especially regarding size and the ability to join larger datasets. I suggest sticking to something that’s built for the task rather than creating a custom solution with tools that aren’t optimized for logging.
When considering the total cost of ownership (TCO) versus utilizing log analytics, there are a few things to keep in mind. With Log Analytics, for instance, annual costs can be around $40 per month for 90 days of retention, which translates to about $480 a year. The charges can escalate if you need to retain logs longer, and it can be challenging for support teams to work with KQL.
On the other hand, with Synapse Serverless, you’re looking at around $50 when combined with a storage account. It offers more flexibility, allowing you to tailor external tables based on your log structure, giving you greater control over access and retention. While cost savings may not be significant, the usability advantages of Synapse could be worth it, depending on your requirements.
Isn't there talk about phasing out Synapse in favor of Fabric in the long run?
You’re right that there’s been some chatter about Fabric potentially replacing Synapse. However, as an engineer following these developments, I haven't seen anything definitive indicating Synapse will be discontinued. It remains popular among large clients for data warehousing, and migrations to Fabric seem manageable, so it shouldn’t pose a significant risk at this point.
I appreciate your insights! I do see how ELK can be efficient for log management; however, we are currently using Log Analytics. We’re implementing data lineage through our processes which makes building external tables with Synapse quite effective for data discovery and ad-hoc queries, especially since we’re working with files that fit within the advised sizes.