How should I architect a multi-site downloader so site changes stay isolated?
Asked by
MiraVale_27
I'm building a personal desktop app that collects download links from several websites and presents them through one shared UI. What's a clean way to structure the backend parsers so a layout change on one site doesn't disrupt the rest of the application? I'd also appreciate examples of modular designs, base classes, or templates that work well for this kind of project.
Answers
Use one adapter or plugin per site, with a shared interface such as `canHandle(url)` and `extractLinks(url)`. Each adapter should keep its own selectors and parsing rules, while the UI, download queue, and common result format stay in the core application. Run jobs independently with their own timeouts and error handling so one broken parser doesn’t stop the others. Keeping saved HTML samples for parser tests also makes site changes easier to diagnose. A project like yt-dlp is a useful example of this extractor-based approach.
Reply to this answer
A plugin architecture would work well here. Put each domain in its own module that inherits from a common base class or follows the same interface. The core code can discover or register plugins, normalize their results, and handle downloads uniformly. When a site changes, you should only need to update that site’s parser instead of touching the rest of the application.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MiraVale_27: That adapter pattern makes sense. Do you have a minimal base-class or boilerplate example showing how you usually wire up methods like `canHandle()` and `extractLinks()`?