I'm reaching out to see how others manage their data contracts in a microservices environment, particularly when it comes to ingesting events into a data warehouse. Recently, we faced an issue where a small schema change made by a producer caused our ingestion process to fail in decoding and validation downstream. This led to a significant data loss before anyone even noticed the problem. Since we're part of a large organization, I can't help but wonder if we're missing some crucial steps or best practices. Is this something that happens often in your experience? If you've found effective solutions, what guardrails or strategies have really helped you catch such issues quickly?
1 Answer
In my experience, we handle data ingestion using Avro schema encoding directly into a Kafka topic. This way, the downstream applications depend on that Avro schema as their serialization/deserialization method, which helps maintain consistency. If the producer fails to provide the correct schema, we get a hard failure. Based on our setup, we either throw an error and ignore the message or crash the app, alerting the on-call team if there’s an issue. The key takeaway is we get immediate feedback on failures, allowing us to react quickly.
That sounds like a solid method! We actually don’t use Kafka in our setup, so I’ll be looking into adapting your ideas to our infrastructure. Appreciate the insight!