I'm building an HTTP downloader that can fetch files from direct URLs using tools such as urllib, requests, or httpx. I'd also like to support websites where the file URL is only revealed after clicking a download button. Since each site may generate different URLs, tokens, redirects, or request parameters, I'm looking for a way to automate this rather than handling every site manually. A low-level HTTP approach would be ideal, but I'm open to using higher-level tools if necessary.
3 Answers
Before automating the page, check whether the website exposes an API or a straightforward download endpoint. If one exists, calling that endpoint directly is usually faster and more reliable than controlling a browser.
Playwright is a good choice when the download requires JavaScript or an actual button click. It can launch a browser, interact with the page, and wait for the download event. Selenium can do something similar, although Playwright often has a simpler modern API for this use case.
It depends on how the button is implemented. If it’s just a link with a file URL, you can inspect the target and download it directly. But some buttons run JavaScript, submit a POST request, follow redirects, or create a one-time token. For those cases, inspect the browser’s developer tools to see the exact request and reproduce it with requests or httpx when possible.
Scripted downloads can turn into a bit of a cat-and-mouse problem because sites may hide the URL or generate temporary tokens. If reproducing the request is too complicated, browser automation may be the more practical option.