How can I copy multiple date-named folders from S3 to a local directory?
Asked by
MellowCedar47
I have several folders in an S3 bucket named with dates, such as 2025-12-10/, 2025-12-11/, and 2025-12-12/. I want to automate copying selected folders to a local disk. I tried using a shell loop, but I'm unsure how to provide the list of dates correctly:
for i in $("2025-12-10"); do mkdir "$i" && aws s3 cp --recursive "s3:///$i" "./$i"; done
What is the correct way to write this?
Answers
Put the date values directly in the loop, separated by spaces. You can also use mkdir -p so the command succeeds when the destination already exists:
for date in 2025-12-10 2025-12-11 2025-12-12 2025-12-13; do
mkdir -p "$date"
aws s3 cp --recursive "s3:///$date" "./$date"
done
The $(...) in the original command is command substitution. It tries to execute 2025-12-10 as a command and then loops over that command’s output, which is not what you want.
Reply to this answer
Another option is aws s3 sync from the bucket root to the local directory, using --exclude and --include patterns to select the date folders. That can be convenient when copying a larger set, but for a small explicit list, a loop is clearer and easier to control.
Reply to this answer
If the dates follow a predictable range, generate them separately and feed them into the loop. For example, with GNU date:
start=$(date -d "2025-12-29" +%s)
for ((n=0; n<5; n++)); do
date=$(date -d "@$((start + n * 86400))" +%F)
mkdir -p "$date"
aws s3 cp --recursive "s3:///$date" "./$date"
done
This handles month and year boundaries instead of manually constructing potentially invalid dates.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowCedar47: That makes sense—I was treating the date like a value, but the shell was trying to run it as a command.