How can I copy multiple date-named S3 directories to a local disk with AWS CLI?
Asked by
QuietMaple47
I have several directories in an S3 bucket named by date, such as 2025-12-10/, 2025-12-11/, and so on. I want to automate copying selected directories to a local disk. I tried: `for i in $("2025-12-10"); do mkdir "$i" && aws s3 cp --recursive "s3:///$i" "./$i"; done` but it does not work. What is the correct shell syntax for handling these directory names?
Answers
If you need to generate a consecutive date range rather than typing every date, generate the dates with a date-aware tool or query the bucket contents first. That avoids creating invalid dates around month and year boundaries, and it also prevents unnecessary copy attempts for directories that do not exist.
Reply to this answer
The command substitution syntax is the problem. `$(...)` means “run this as a command and use its output,” so the shell tries to execute `2025-12-10`. Put the directory names directly in the `for` list instead, and quote the variables:
`for date in 2025-12-10 2025-12-11 2025-12-12 2025-12-13 2025-12-14 2025-12-15; do
mkdir -p "$date" || exit
aws s3 cp --recursive "s3:///$date/" "./$date/"
done`
There are no commas or parentheses around the list. `mkdir -p` also avoids failing if a destination directory already exists.
Reply to this answer
If the dates are part of a larger set of objects, `aws s3 sync` can be useful with include and exclude filters. For example, start by excluding everything and then include the desired prefixes:
`aws s3 sync "s3:///" ./
--exclude "*"
--include "2025-12-10/*"
--include "2025-12-11/*"`
For a short, explicit list of directories, looping over `aws s3 cp --recursive` is usually easier to read.
Reply to this answer
Add your answer
Answers are reviewed before publication.
QuietMaple47: Thanks—that explains it. I was unintentionally trying to execute the date as a command.