I'm building an HTTP downloader that currently fetches direct URLs with GET requests using libraries such as urllib, requests, or httpx. I'd also like it to work with websites where a user must click a download button first. Since different sites may generate different download URLs, I'm looking for a way to discover and follow the button's action automatically, preferably at the HTTP-request level, though I'm open to using a higher-level browser automation library if necessary. What approaches or tools should I consider?
3 Answers
For pages that rely heavily on JavaScript, Playwright or Selenium can open the page, click the button, and wait for the download event. Playwright is usually a convenient modern option, while direct HTTP requests are lighter when you can identify and reproduce the underlying request.
Check whether the website exposes a documented API or download endpoint first. An API may provide the file URL or accept the required parameters more reliably than simulating a button click, and it’s generally easier to maintain than scraping page behavior.
It depends on how the button is implemented. If it’s just a normal link pointing to a file, you can inspect the target URL and download it directly. But some buttons run JavaScript, send a POST request, follow redirects, or generate a temporary token. In those cases, you need to inspect the browser’s network activity and reproduce the relevant requests, including cookies, headers, form data, or tokens.
Exactly—it can become a moving target when the site creates one-time URLs or hides the request behind several redirects. Browser developer tools can show what the click actually triggers, and sometimes the final request can be reproduced with requests or httpx without automating the whole browser.