I'm trying to understand Bash's parsing rules in detail. How does it interpret spaces, pipes, redirections, quoting, and other special characters when turning a command line into commands? Is there a formal grammar or reference that explains the complete process?
4 Answers
For Bash’s actual grammar, the best reference is the `parse.y` file in the Bash source code. It uses a Bison-style grammar and shows how Bash recognizes pipelines, lists, redirections, compound commands, and other shell syntax. The Bash reference manual is also useful for understanding the behavior around that grammar.
A useful complementary resource is a technical presentation about parsing POSIX shell scripts, particularly the challenges of validating package-maintainer scripts. It explains why shell parsing has several distinct stages and why syntax, expansions, and execution behavior need to be considered separately.
The built-in Bash documentation is a good place to start. Run `info bash` or read the Bash Reference Manual, especially the sections on shell syntax, quoting, expansions, redirections, pipelines, and compound commands. The grammar alone won’t explain every later processing step.
There’s more involved than splitting commands at spaces and pipes. Bash first recognizes operators and reserved words, then handles quoting and escaping, performs expansions such as parameter expansion and command substitution, expands filename patterns, sets up redirections, and finally runs the resulting command. Characters such as quotes, parentheses, semicolons, ampersands, less-than and greater-than signs, brackets, and wildcard characters can all have special meanings depending on context.
That helps clarify why simply describing the process as whitespace-based tokenization is incomplete. I’ll look at the grammar alongside the expansion and execution stages.