I'm working with FastAPI and have set up an endpoint to handle incoming messages that can contain text and multiple image files, which may reach up to 100MB each. After validating each message, I want to process these images and display them on the frontend once everything is done. The image processing involves sending GET requests to an external website to get download links, and then downloading the files from those links. This process is quite I/O-intensive, so using FastAPI's BackgroundTasks doesn't quite seem suitable. I also need the images to be persistent, which rules out in-memory solutions like asyncio queues. So, I'm considering task queues like Dramatiq, RQ, or Celery.
My current plan is:
1. The FastAPI endpoint receives and validates the message, then returns an OK response.
2. Enqueue the images for processing through a worker (Dramatiq, RQ, or Celery).
3. The FastAPI service will listen to a Redis pub/sub channel.
4. Once the worker completes the image downloading and processing, it publishes an event to Redis.
5. FastAPI receives the event and sends a reference to the processed images to the frontend.
Is this a sensible approach for handling such operations, or is there a more scalable alternative? I'm leaning towards Dramatiq because it supports async operations, which fits well with the I/O-heavy requirements of image processing.
1 Answer
You should definitely consider storing your image data externally, like in an S3 bucket. It keeps your queue lightweight too. Instead of putting the actual images in your task queue, just store links to them in the S3 bucket when you migrate to the cloud. It will make everything easier and faster in the long run!
Yeah, that makes sense! I’ll focus on that storage method when I move things over. Thanks for the tip!