I'm working on a bash script to check if a streamer on Twitch is online. The script uses curl to make requests based on a list of usernames from my subscriptions file. Here's the current script I have:
```bash
#!/bin/bash
sub_list=$(while read -r line; do
sed -e '/#/d' -e '/^s*$/d' | tr -d '@'
done < "$HOME/subscriptions.txt")
while read -r line; do
user_check=$(curl -Ls https://twitch.tv/$line/ | grep -o "live_user" | head -1)
if [ "$user_check" = "live_user" ]; then
printf "$line is currently liven"
fi
done <<< "$sub_list"
```
However, I'm struggling to figure out how to run multiple curl instances in parallel to check all the URLs quickly. The current approach is too slow because it processes each link one at a time. Any suggestions on how I can improve this?
3 Answers
You could look into job control and use control characters in bash to help manage parallel processes. This way, you can create wait groups to control how many curl requests run at the same time and also handle their output effectively.
If you're open to using a little Java, you could write a small program that uses Java's native `HttpClient`, which has a `sendAsync` method to manage multiple requests simultaneously. It might simplify your life if you're familiar with Java!
The easiest solution would be to use GNU Parallel. Separate the logic for checking each user into a function:
```bash
function check_user() {
local user="$1"
if curl -Ls https://twitch.tv/$user/ | grep -q "live_user"; then
echo "$user is currently live"
fi
}
export -f check_user
```
Then pipe your cleaned list of usernames into parallel:
```bash
sed -e '/#/d; /^s*$/d; s/@//g' subscriptions.txt | parallel check_user
```
You can adjust the `-j` option in Parallel to set how many requests you want running at once. It's highly efficient compared to doing it all in bash.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically