I'm pulling pricing data from a REST API for a small side project. Most responses are JSON, but the API sometimes omits fields, returns empty arrays, or sends an HTML block/error page instead—especially when requests get rate-limited or blocked. I'm currently using requests.Response.json() with a growing pile of try/except blocks. Should I use Pydantic, dict.get(), or a different pattern? I have essentially no budget, so I'm looking for a practical approach I can maintain myself.
3 Answers
A small parser or adapter class can keep the calling code clean. Whenever a new response shape breaks it, save that response as a test fixture and add a case for it. Over time, the parser becomes predictable instead of being surrounded by scattered try/except blocks. Pydantic is useful for checking the final structure, but it cannot fix an HTML block page returned before JSON parsing.
Treat this as two separate problems: transport errors and payload validation. First check the HTTP status and Content-Type before attempting to decode JSON. If the response is not successful or is HTML, save a short capped body sample for debugging and handle it as a blocked or invalid response. Once you have JSON, validate its shape with Pydantic, making fields optional only when the API genuinely leaves them out. For simple optional values, dict.get(key) with a sensible default is fine.
At minimum, check for a successful status code and an appropriate Content-Type before calling response.json(). Catch JSON decoding errors separately from validation errors, and log the status, URL, and a limited response snippet rather than the entire body. Then use Pydantic or another schema model for the fields you actually depend on. Libraries like pandas or Polars are usually unnecessary unless you specifically need tabular analysis.
That makes sense. The main issue is probably separating blocked requests from legitimate JSON responses before I worry about validating the pricing fields.