I'm pulling pricing data from a REST API for a small side project. Most responses are JSON, but the structure is inconsistent: fields may be missing, arrays can be empty, and blocked requests sometimes return an HTML error page instead. I'm currently calling response.json() from requests and wrapping everything in try/except blocks, which feels fragile. Should I use Pydantic, rely on dict.get(), or structure the parsing another way? The budget is basically zero, so I'm looking for a practical approach I can maintain myself. The bigger issue may be distinguishing blocked requests from valid JSON that simply fails validation.
3 Answers
A small adapter or parser class can keep the request handling in one place. For every response, handle non-success status codes first, then parse JSON, normalize missing values, and validate the result. Whenever a new response shape breaks it, save that payload as a local test fixture and add a case for it. Over time, the messy try/except logic becomes a predictable set of tested edge cases.
Pydantic won’t solve an HTML block page because that failure happens before validation. Check the status and content type, and perhaps inspect the first character only as a fallback—HTML often starts with '<'. Then call response.json() and catch JSON decode errors separately from validation errors. I’d avoid bringing in pandas or another data tool unless you actually need tabular transformations; it’s unnecessary overhead for this problem.
Treat this as two separate problems: transport errors and data validation. First check the status code and Content-Type before trying to decode JSON. If the response is not JSON, keep a short snippet of the body for diagnostics instead of feeding it to the parser. Once you have a JSON payload, use Pydantic to validate the expected shape, making fields optional only when the API genuinely omits them. For optional nested values, .get() with sensible defaults is fine.
That makes sense. My original question blurred parsing with validation—the real problem is separating blocked requests from malformed or incomplete JSON.