Silicon Soul

The Bash shell has many special characters. These characters are special because they are used to access specific shell features.

Some of these characters, (/, ?, [, ], {, }, $, =, !) can be used literally on the command line. The rest will need to be escaped with a backslash (\) or surrounded with single quotes ('').

Note: If you are not familiar with the GNU/Linux command line interface, review the Conventions page before proceeding.

Input/Output

Redirection

A process on a GNU/Linux system can expect to have three handles (streams) that are used to access a file or other input/output (I/O) resource. These handles are referred to as file descriptors (fd).

The three standard streams are:

Each device node for a stream (e.g., /dev/stdin) is symbolically linked to its associated file in the procfs file system (e.g., /proc/self/fd/0). The associated procfs files themselves are symbolically linked to pseudo terminal slave (pts) devices (e.g., /dev/pts/0).

Redirection is used to redirect I/O streams to or from a device or file. Usually, your standard input source is a keyboard and your standard output source is the terminal you are using. With redirection, you can change that.

Redirecting Standard Input

You can redirect the standard input like so:

ex_command 0< ex_file

For the standard input and standard output, you can leave off the fd number, as it is implied:

ex_command < ex_file

Usually, commands take files as arguments, but some commands cannot. This is one good example of where redirecting the standard input becomes useful:

cpio -t < 'archive.cpio'

Above, the cpio -t command displays a list of files in a cpio archive called archive.cpio.

Redirecting Standard Output

ex_command > ex_file

If you want to display the first ten lines of a file and save it to a new file, you can do:

head 'errors.log' > 'log_start.log'

Above, the head command grabs the first ten lines of errors.log, and instead of displaying that information on the terminal, saves it to log_start.log.

The shell does redirection before it runs head 'errors.log', i.e., if log_start.log needs to be created, it is created first, or if log_start.log already exists, its content is overwritten first. Also, head obtains its output before writing it to log_start.log.

Most commands are unaware of whether their output is being redirected or not, and give the same output either way. Other commands behave differently.

For example, when outputting to the standard output, ls will try to fit as many object names on a line as possible, and possibly color code the output. When ls outputs to a file using redirection, only a single object name is listed per line, and the output is not colored.

You can use redirection to append information to a file, as well. This is how you would redirect the standard output and append data:

ex_command >> ex_file

In the example below, the cat command is used to output the contents of new_names.txt, but instead of sending the contents to the standard output, it is appended to names.txt using redirection:

cat 'new_names.txt' >> 'names.txt'

Redirecting Standard Error

ex_command 2> ex_file

As with the standard output, you can also append standard error content:

ex_command 2>> ex_file

Redirecting Standard Output and Standard Error

You can simultaneously redirect the standard output and standard error:

ex_command > ex_file 2> ex_file_2

If you want the standard output and standard error redirected to the same location, you can do:

ex_program > ex_file 2>&1

Above, 2>&1 (which can reduced to >&) redirects the standard error to the standard output. Order matters here, i.e., the redirection of the standard error must come after the redirection of the standard output. Otherwise, the standard error will be sent to the default standard output location, which is usually the terminal.

Keep in mind, the 2>&1 (>&) syntax simply redirects one I/O stream to another one. For example, you can redirect the standard output to the standard error by using 1>&2 or >&2, instead. Often, this is used in scripts to define error messages:

if [[ ${#} -ne 1 ]]; then
    echo >&2 "Usage: ${0} only accepts one parameter."
    exit 1
fi

The above example is a code snippet from a Bash script. The if conditional performs a test ([[ ${#} -ne 1 ]]) that checks whether the script was sent only one argument (${#} -ne 1). If not, it redirects the standard output to the standard error and outputs an error message with the echo command (echo >&2 "Usage: ${0} only accepts one parameter"; ${0} is a special shell variable that will be replaced by the name of the Bash script). Finally, the script is exited with an error code (exit 1).

Here Document

The << syntax denotes a here document, i.e., a file literal or input stream literal. A here document allows you to tell a program what the ending text of a document will be.

When that delimiter is seen by the program, the program will read all of the content that you have provided to it up until the delimiter, and perform a task upon it. For example, a multi-line string is provided to the wc command below:

$ wc << EOF
> uno dos tres
> quatro cinco
> EOF
 2  5 26

When the specified delimiter, EOF (End of File) is entered, the input is read by the wc command, which then displays the newline, word, and byte count of the string.

Here String

The <<< syntax denotes a here string. A here string allows you to provide a pre-made string to the standard input of a program. For example, wc <<< '???' is comparable to echo '???' | wc.

Unlike here documents, here strings do not use delimiters. They are especially useful for commands that take short input. Below, the bc (basic calculator) program outputs the product of 8 and 4

$ bc <<< 8*4
32

Pipelines

If you want to send content from one command to another command, you can use the pipe (|) to create a pipeline.

ex_command_1 | ex_command_2

The command above is an example of a pipeline, i.e., a chain of commands where the output of one command serves as the input for the next command. Creating custom pipelines are a substantive part of what makes the GNU/Linux command line a powerful and flexible tool.

Expansion

Filename Expansion

Special shell characters can be used to denote a search pattern. Search patterns are expanded by the shell to match object names that actually exist as pathnames on the file system, i.e., filename expansion.

?

Matches any single character (except /) that must exist (1 repetition).

*

Matches any string of characters (except /) that are optional (0 to many repetitions).

For example, f?d will match for the string fad, but will not match for the string food. However, f*d will match for both fad and food, as well as a string like fd.

By default, the * character will not match hidden objects. If you want hidden objects to be included in your search pattern results, you will need to explicitly specify that.

$ ls -dl .c*
drwx------  7 amnesia amnesia 140 Mar 25 23:36 .cache
drwx------ 11 amnesia amnesia 280 Mar 25 23:33 .config

The above example requests the long listing for hidden directories whose name begins with a c (hidden objects start with a ., the -d option tells ls to provide listings for directories themselves instead of their contents, and the -l option tells ls to provide a long listing format).

The expansion of search patterns is the responsibility of the shell. Commands that you execute usually know nothing about search patterns. All commands see is a list of pathnames, but not where those pathnames come from.

If the shell cannot find an object whose name matches a search pattern, the search pattern itself is passed onto the command in question. What the command does with the search pattern depends on the command. Typically, the search pattern is interpreted as a filename, but the file is not found and an error is shown.

Filename expansion has the potential to act on multiple file system objects. If you are not sure how the shell is going to interpret your search pattern, you can test how the shell will react by using your search pattern with the echo command, e.g., echo f*d.

Character Sets

A character set is denoted by square brackets ([]) and matches any single character specified in the set of characters. If you want to match any single character not specified in the character set, you can do so by adding a ! to the beginning of your character set ([!]).

Like with search patterns, character sets can be used to match for object names that already exist as pathnames on the file system.

This example requests the long listing for any directory whose name begins with either a M or a V:

$ ls -dl [MV]*
drwx------ 2 amnesia amnesia 40 Mar 25 23:33 Music
drwx------ 2 amnesia amnesia 40 Mar 25 23:33 Videos

The next example requests the long listing for any directory whose name begins with anything except a M or V.

$ ls -dl [!MV]*
drwx------ 2 amnesia amnesia 80 Mar 25 23:32  Desktop
drwx------ 2 amnesia amnesia 40 Mar 25 23:33  Documents
drwx------ 2 amnesia amnesia 40 Mar 25 23:33  Downloads
drwx------ 2 amnesia amnesia 40 Mar 25 23:33  Pictures
drwx------ 2 amnesia amnesia 40 Mar 25 23:33  Public
drwx------ 2 amnesia amnesia 40 Mar 25 23:33  Templates
drwx------ 2 amnesia amnesia 40 Mar 25 23:33 'Tor Browser'

You can also specify a range within a character set. For example, the following will present the long listing for objects that begin with file_ and end with a number from 1 through 5:

$ ls -l file_[1-5]
-rw-r--r-- 1 amnesia amnesia 0 Mar 26 00:09 file_1
-rw-r--r-- 1 amnesia amnesia 0 Mar 26 00:09 file_2
-rw-r--r-- 1 amnesia amnesia 0 Mar 26 00:09 file_3
-rw-r--r-- 1 amnesia amnesia 0 Mar 26 00:09 file_4
-rw-r--r-- 1 amnesia amnesia 0 Mar 26 00:09 file_5

Brace Expansion

Brace expansion is used to generate strings on the command line from a pattern that you specify. For example, if you want to simultaneously create multiple text files, and you want each file to have the name of a different color, you could use the touch command and brace expansion:

$ touch {red,yellow,blue}.txt
$ ls
blue.txt  red.txt  yellow.txt

In general, a word on the command line that contains several comma-separated pieces of text within braces is replaced by as many words as there are pieces of text between the braces. This replacement is purely based on the command line text and is completely independent of the existence or non-existence of any file system files or directories. This is unlike search patterns and character sets, which always produce only those names that actually exist as pathnames on the file system.

Here is a more complicated example:

$ echo {a,b,c}{1,2,3}.dat
a1.dat a2.dat a3.dat b1.dat b2.dat b3.dat c1.dat c2.dat c3.dat

Above, the cartesian product of the two brace expressions is created.

You can specify ranges with brace expansion by using .., i.e., extended brace expansion. For example, if you want to simultaneously create multiple files with the numbers 1 through 5 appended to their filenames, you can do:

$ touch file_{1..5}.txt
$ ls
file_1.txt  file_2.txt  file_3.txt  file_4.txt  file_5.txt

Tilde Expansion

The tilde (~) is a special character that is used to represent the current user's home directory. For example, if you were logged into a system as the amnesia user, ~ would be expanded to /home/amnesia via tilde expansion.

$ echo ~
/home/amnesia

If you want to specify a specific user's home directory, you can do so by placing their username directly after the ~, e.g., ~guest will be expanded to /home/guest.

Substitution

Variable Substitution

To use a variable on the command line and have its name substituted for its value, you can surround the variable name with curly braces ({}) and prefix it with a dollar sign ($).

echo ${ex_variable}

This is referred to as variable substitution. As shown above, the echo command is used with variable substitution to display the contents of a variable.

If the variable name is not going to be immediately followed by another character, you can leave off the curly braces.

$ number=4
$ echo "$number"
4

Variable substitution can be inhibited in two ways:

1. Put a backslash (\) in front of the dollar sign.
2. Put single quotes '' around the variable reference.

Command Substitution

If you want to feed the output of a command as an argument to another command, you can use command substitution. The command you run with command substitution is run in a subshell.

The following example uses the whoami command to incorporate the current user's username into a message that it is echoed back to the terminal:

$ echo "Hi there, $(whoami)."
Hi there, amnesia.

Command substitution supports nesting, as well.

Variable Assignment

The equals sign (=) is used for variable assignment (e.g., ex_variable=ex_value). White space is not allowed on either side of the = character.

Command History

The bang (!) is also used for command history (you can view your user's command history with the history command).

For example, if you want to run the command you ran three commands ago, you can do !-3. Here is an illustration:

$ ls
Desktop  Documents  Downloads Music  Pictures  Public  Templates  Videos
$ ls -l
total 32
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Desktop
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Documents
drwxr-xr-x 3 amnesia amnesia 4096 Apr  3 15:31 Downloads
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Music
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Pictures
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Public
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Templates
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Videos
$ history | tail -n 3
  263  ls
  264  ls -l
  265  history | tail -n 3
$ !-3
ls
Desktop  Documents  Downloads  Music  Pictures  Public  Templates  Videos

Three commands are run:

1. ls
2. ls -l
3. history | tail -n 3

Then, !-3 is used to run the command that was run three commands ago, i.e., ls.

To execute the previous command, you can enter !!. If you know the command history number of a command, you can run the command again by referring to this number (e.g., !ex_command_history_number). If you know the first few characters of a command, you can run the most recent command that starts with that string by doing !ex_string.

The previous command's last argument can be referenced by using !$. This value is also stored in the ${_} shell special parameter.

When a user starts an interactive Bash shell, their user's ${HOME}/.bash_history file is opened. Normally, the history stack is not written to the ${HOME}/.bash_history file until the user exits the interactive shell. If a user is executing a subshell to the interactive shell, the subshell reads the parent shell's history file and writes to it upon exit.

Quoting

Quotes are used to denote strings on the command line.

Strong Quotes

Single quotes ('') are used for strict quoting and are used to specify literal strings. Inside of single quotes, expansion, substitution, and word splitting do not occur.

If you want to use backslash escape sequences in a literal string, precede it with a dollar sign (e.g., $'This sentence ends with a newline.\n'). Run man 1 bash for more information.

Weak Quotes

Double quotes ("") are less strict. Inside of double quotes, filename expansion, brace expansion, tilde expansion, and word splitting do not occur, but variable substitution and command substitution do occur. Also, newline characters are preserved.

No Quotes

Without quotes, two things happen at the command line:

1. The result of a variable or command substitution is split into words wherever it contains white space. Specifically, the result of the expansion is split at each character that appears in the value of the Internal Field Separator (IFS) variable.

   If a sequence of separator characters contains white space (e.g., space, tab, or newline), the white space is counted as a single character.

2. Each field that results from splitting is interpreted as a search pattern (i.e., a glob) if it contains one of the following characters \[*?. If that pattern matches one or more file system objects, the pattern is replaced by the list of matching object names, i.e., filename expansion occurs.

An unquoted variable substitution, like ${ex_variable}, is known as the split+glob operator, where split refers to word splitting and glob refers to filename expansion (i.e., ${ex_variable} is replaced with the variable's value through variable substitution, and that value is then subject to word splitting and filename expansion).

In contrast, "${ex_variable}" is just replaced with the variable's value through variable substitution. No word splitting or filename expansion follows.

The same goes for command substitution. $(ex_command) is a command substitution followed by a split+glob, while "$(ex_command)" is just a command substitution.

Logic and Math

Logical AND

Two ampersands (&&) represent the logical AND.

ex_command_1 && ex_command_2

Above, ex_command_2 only runs if ex_command_1 successfully completes.

Logical OR

Two pipes (||) represent the logical OR.

ex_command_1 || ex_command_2

Above, ex_command_2 only runs if ex_command_1 does not successfully run.

In the order of operations, the logical AND has a higher precedence than the logical OR.

Logical NOT

As previously discussed, the bang (!) represents the logical NOT.

$ ls -dl [!P]*
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Desktop
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Documents
drwxr-xr-x 3 amnesia amnesia 4096 Apr  3 15:31 Downloads
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Music
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Templates
drwxr-xr-x 2 amnesia amnesia 4096 Feb 27 09:39 Videos

In the above command, the ! is used to create a negated character set, which is used to specify any object name that does not begin with a P.

Test Conditions

Double square brackets ([[ ]]) are a shorthand notation for the test command.

if [[ -e ex_file ]]; then
    ex_code
fi

In the example above, if ex_file exists, ex_code would run.

Word splitting and filename expansion do not occur inside of [[ ]].

Run help test for more information on the test command.

Integer Math

Double parentheses ((( ))) are used for integer math operations.

(( ex_integer + ex_integer ))

When using your integer math operation within a larger expression, you should prefix the parentheses with a dollar sign ($):

$ echo "$(( 7 + 7 ))"
14

This ensures that the expression is replaced with the result of the arithmetic evaluation.

You can use shell variables in your integer expressions, as well. When doing so, you can leave off the $ when referring to the variable, and should not quote the variable name itself:

$ number=7
$ echo "$(( number + 7 ))"
14

The shell will always round down the results of your integer math operations.

In regard to the order of operations in math expressions, BEDMAS is observed:

1. Brackets/Braces/Parentheses
2. Exponents
3. Division
4. Multiplication
5. Addition
6. Subtraction

Comments

The sharp (#) is used to denote comments that extend to the end of the line. They are not processed by the shell.

Code Blocks

The curly braces ({}) and parentheses (()) can be used to denote code blocks. The blocks in the examples below echo several strings, which are then redirected into a text file called lines.txt.

$ {
    echo 'Line 1'
    echo 'Line 2'
    echo 'Line 3'
} > 'lines.txt'

The curly braces are considered reserved words and must be separated from the command list by spaces or other special shell characters. This approach causes the commands to be executed in the current shell context, and a semicolon (;) or newline must follow the command list.

$ (
    echo 'Line 1'
    echo 'Line 2'
    echo 'Line 3'
) > 'lines.txt'

The parentheses are considered operators and do not need to be separated from the command list by spaces, although use of whitespace can help improve legibility. This approach causes a subshell environment to be created, where each of the commands in the command list are executed in that subshell.

Placeholders

Curly braces ({}) can also be used as a placeholder when used with certain commands, like find.

For example, find . -size +2G -exec mv '{}' "${HOME}/large_files/" \; looks for files larger than 2 gigabytes (-size +2G) in the current directory (.), and passes them to the mv command (-exec mv), which moves them to the ${HOME}/large_files/ directory. {} is used as a placeholder for every large file that is sent from find to mv.

Background a Process

To run a command, place its associated process(es) into the background, and return to the command line, you can use an ampersand (&).

ex_command &

When you load a process into the background, the process is assigned a job ID number.

Separating Commands

The semicolon (;) is used to separate commands.

ex_command_1 ; ex_command_2

This enables you to simultaneously specify more than one command on the same command line.

Shell Feature Precedence

The precedence for the shell features discussed is:

1. Brace Expansion
2. Tilde Expansion
3. Variable Substitution
4. Arithmetic Substitution
5. Command Substitution (from left to right)
6. Word Splitting
7. Filename Expansion

Further Reading

If you would like to learn more about the shell features covered in this post, please refer to the following references:

• Bash Reference Manual
• Learning the bash Shell, 3rd Edition
• The Bash Hackers Wiki
• The Linux Documentation Project