How do I loop over files in Bash and create matching output filenames?
Asked by
MellowPine42
Hi! This is my first time using Bash, and I'm trying to run a Python script on every file in /data/folder. I want each result to be written to another folder while keeping the input filename, but I'm not sure what to put after --out. I also want to make sure the loop syntax, quoting, and command-line options are correct.
#!/bin/bash
for i in /data/folder/*; do
/folder/munge_sumstats.py
--sumstats "$i"
--chunksize 500000
--snp rsID
--out "another/folder/..."
done
Answers
Check the paths carefully as well. The script path and every input/output path should be valid from the directory where you run the Bash script. Using absolute paths can avoid confusion, especially if the Python program resolves files relative to the current working directory.
Reply to this answer
Use regular straight quotes and write the options with two hyphens, such as --sumstats and --out. To keep the original filename while changing its directory, you can remove everything through the last slash with parameter expansion:
for i in /data/folder/*; do
/folder/munge_sumstats.py
--sumstats "$i"
--chunksize 500000
--snp rsID
--out "another/folder/${i##*/}"
done
The quotes around "$i" are important in case a filename contains spaces. Make sure the output directory already exists.
Reply to this answer
Another clear option is to use basename to get only the input file’s name:
for i in /data/folder/*; do
name=$(basename "$i")
/folder/munge_sumstats.py
--sumstats "$i"
--chunksize 500000
--snp rsID
--out "another/folder/$name"
done
If the input files have a suffix that should be replaced, you can strip it and add a new one, for example: name=$(basename "$i" .old); then use --out "another/folder/$name.new".
MellowPine42: That makes sense too—thank you!
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowPine42: I only used the odd spacing because I was typing this on my phone. Thanks, this explanation helped a lot!