I'm building a FastAPI backend that processes documents with AI, and I'm trying to find the right balance between reliable tests and meaningful provider coverage. The main CI pipeline currently runs fully offline with SQLite, a fake Redis service, and mocked HTTP calls, so pull-request checks stay fast and deterministic. A separate integration pipeline uses PostgreSQL and Redis to catch infrastructure-specific problems.
The open question is how to test the AI layer. Mocking the provider keeps the main suite stable, but it can delay discovery of provider regressions or changes in model behavior. I'm considering a layered setup with mocked unit tests, integration tests, recorded responses or contract checks, and a small number of real API calls against representative documents.
How do you handle this in production? Do you mock providers completely, replay recorded responses, run a separate suite of real requests, or use another strategy?
2 Answers
A layered testing strategy works well. Keep unit tests completely self-contained and mock or isolate external dependencies so they run quickly on every change. Add integration tests using services such as PostgreSQL and Redis in a local container setup, then run broader functional end-to-end tests after deployment or before releases.
For the AI portion, keep the provider behind a separate client or adapter. That lets the core document-processing and validation logic be tested without making model calls, while a smaller AI-focused suite can verify the integration independently.
Separate the tests by cost and purpose. Unit tests should never call an API. Regular integration tests can run on every pull request, while a slower AI integration suite can be enabled manually during development or triggered by a workflow label. Before releases, run a small set of real, happy-path AI workflows as end-to-end checks.
For model responses, avoid asserting exact prose or brittle snapshots. Have the model produce structured output and validate deterministic properties afterward, such as schema compliance, required fields, numeric reconciliation, and business invariants. This catches meaningful regressions without failing whenever the provider makes a harmless wording change.
I like the idea of enabling the AI suite separately instead of running it on every pull request. Contract-style validation also seems more durable than exact snapshots because the important checks are about the resulting data, not the model's precise wording.
That is close to the setup I'm moving toward. The AI extracts structured data, but a deterministic validation layer checks the schema, numeric consistency, and business rules before anything is persisted. Keeping the provider outside the core business logic has made the application much easier to test. I'm still deciding whether the slower integration suite should run on every merge or only before releases.