How do I write a Bash loop that processes files and saves matching output names?
Asked by
MistyCedar42
Hi! I'm new to Bash and I'm trying to loop through every file in /data/folder, pass each one to munge_sumstats.py, and write the processed result to another folder. I'm unsure how to format the long command correctly, especially the option names and the value for --out. Ideally, each output should keep the original filename while being saved in the new directory. Here's the structure I'm aiming for:
#!/bin/bash
for i in /data/folder/*; do
/folder/munge_sumstats.py
--sumstats "$i"
--chunksize 500000
--snp rsID
--out "another/folder/..."
done
What should go after --out, and is there anything else I'm missing?
Answers
Make sure the input path passed to the Python script is absolute or otherwise correct for the directory from which the script is launched. A relative path may not resolve the way you expect, even if the shell loop itself is working.
Reply to this answer
Another readable approach is to use basename to extract the filename first. For example:
for i in /data/folder/*; do
name=$(basename "$i")
/folder/munge_sumstats.py
--sumstats "$i"
--chunksize 500000
--snp rsID
--out "another/folder/$name"
done
If the input has a suffix that should change, remove the old suffix and append the new one when constructing name.
Reply to this answer
Use two hyphens for long options, such as --sumstats and --out. To keep the same filename while changing the directory, you can strip off the path from "$i":
for i in /data/folder/*; do
/folder/munge_sumstats.py
--sumstats "$i"
--chunksize 500000
--snp rsID
--out "another/folder/${i##*/}"
done
The ${i##*/} part removes everything through the final slash, leaving only the input filename.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MistyCedar42: That makes sense—thanks! I was typing the extra spaces because I was working from my phone, but the filename expansion is exactly what I needed.