I've been trying to bulk download files from a list of URLs in a specific format using PowerShell. Each URL is separated by carriage returns, like this: `https://www.govinfo.gov/link/fr/78/2542?link-type=pdf`. When I open the URL in a browser, it downloads the document as expected. However, when I try using this PowerShell command: `Get-Content url-list.txt | ForEach-Object {Invoke-WebRequest $_ -OutFile (Split-Path $_ -leaf)}`, I encounter errors related to the URL formatting. It seems like the command can't handle the redirect to the actual file due to issues with the URL containing a question mark. I'm looking for a solution to download these files without errors.
3 Answers
It looks like the problem is with the question mark in your URL. That character isn't allowed in filenames. You could simplify your approach by replacing the `?` with something like an underscore in your download file names. That might resolve the issue you're facing! Let me know if that works for you.
Ah, I see! After I removed the question mark from my URLs using Excel, my downloads started working. However, I noticed it was downloading much slower than using a web browser. It could be that PowerShell has some throttling by default. Also, the downloaded files were missing the `.pdf` extension, but I can manage that later. Just a heads up!
I had a similar issue, and what worked for me was to replace the `Split-Path` command because it's intended for filesystem paths, not URLs. Instead, cast the URL like this: `[uri]'your-url-here'` to properly access the URL components. Also, check out this script for managing redirection. It outlines how to follow the redirection chain and download the actual file you'll need.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically