Page Body

Regular Expressions and the grep Command

Regular expressions are a powerful way to create customized patterns to match specific strings of text. The learning curve for regular expressions is steep, but taking the time to understand regular expression (regex) fundamentals will make you a more productive GNU/Linux user.

Note: If you are not familiar with the GNU/Linux command line interface, review the Conventions page before proceeding.

Regular Expressions

Often, regular expressions are recursively constructed from primitives that are themselves regular expressions (i.e., a regular expression can contain regular expressions). The simplest regular expressions are letters, digits, and many other typical characters, which usually stand for themselves.

While shell search patterns must always match beginning at the start of a filename, in programs selecting lines based on regular expressions, it is usually sufficient if the regular expression matches anywhere in a line.

Like with filename expansion, regular expressions use specific characters to denote special meanings. By default, regular expressions are greedy, i.e., they try to match as much of the input string as possible. You can append the ? character to your regex pattern to make it non-greedy.

Anchors

Anchors are meta-characters that match the empty strings at the beginning or ending of a line or word, i.e., they force matches to be anchored.

^
Matches the beginning of the input (e.g., ^C.* matches Cat, but not LUCK).
$
Matches the end of the input (e.g., .*C$ matches ELLIPTIC, but not COOL).
\<
Matches the beginning of a word (a place where a non-letter precedes a letter).
\>
Matches the end of a word (where a letter is followed by a non-letter).

Word brackets (i.e., \< and \>) are a GNU grep command specialty and help prevent returning strings that contain the actual string you are searching for (e.g., super in supergirl).

Character Classes

A character class is a list of characters enclosed by square brackets ([ and ]). It matches any single character in that list.

[]
Character class. Matches any single character contained within the square brackets. Order does not matter and special characters lose their meaning. A hyphen can be used to denote a range (e.g. [0-9]).
[^]
Negated character class. Matches any single character not contained between the square brackets (e.g., [^0-9]).

If you ever create a regex and want to match literals for regex special characters, you can either escape them with a backslash (\) or enclose them in a character class ([]).

Shorthand Character Classes

Shorthand character classes are named classes of characters that are predefined within bracket expressions.

[[:alnum:]]
Matches alphanumeric characters ([A-Za-z0-9]).
[[:alpha:]]
Matches alphabetic characters ([A-Za-z]).
[[:blank:]]
Matches a space or tab, including a line break ([ \t]).
[[:cntrl:]]
Matches control characters ([\x00-\x1F\x7F]).
[[:digit:]]
Matches decimal digits ([0-9], \d).
[[:graph:]]
Matches visible characters ([\x21-\x7E]).
[[:lower:]]
Matches lowercase alphabetic characters [a-z].
[[:print:]]
Matches visible characters or a space ([\x20-\x7E]).
[[:space:]]
Matches space characters, including a line break ([ \t\r\n\v\f], \s).
[[:upper:]]
Matches uppercase alphabetic characters ([A-Z]).
[\w]
Matches any word character ([A-Za-z0-9_]).
[[:xdigit:]]
Matches hexadecimal digits ([0-9A-Fa-f]).

Also, there are special expressions that have a specific meaning in terms of the characters they represent:

.
Matches any single character (except a newline) and enforces that the character must exist (this is similar to how the ? character is used in filename expansion).
\b
Matches the empty string at the edge of a word.
\B
Matches the empty string provided it is not at the edge of a word.
\s
Matches whitespace, it is a synonym for [[:space:]].
\S
Matches non-whitespace, it is a synonym for [^[:space:]].
\w
Matches word character, it is a synonym for [_[:alnum:]].
\W
Matches non-word character, it is a synonym for [^_[:alnum:]].

Keep in mind, common character encodings for letters are not contiguous. For example, [A-Za-z] matches for _ too. Run the man ascii command for a visual demonstration.

Character Groups

A character group is denoted by parentheses (( and )) and matches the contained characters in their exact order (e.g., (xyz) matches xyz, but not zxy).

Concatenation

Two regular expressions can be concatenated. The resulting regex matches any string formed by concatenating two substrings that respectively match the concatenated regexes.

Alternation

Two regular expressions can be joined by the infix operator (|), e.g., super (man | woman | girl). As the target string is scanned, expressions separated by | are tried from left to right (e.g., superman, superwoman, supergirl). When one expression completely matches, that branch is accepted.

Repetitions

Repetition operators follow a regular expression and describe how many times the matching string may occur.

?
Preceding item is matched from 0 to 1 repetitions (?? for non-greedy form), e.g., Jon? matches both both Jo and Jon, but not Jonn.
*
Preceding item is matched from 0 to many repetitions (*? for non-greedy form), e.g., Jon* matches Jo, Jon, and Jonn.
+
Preceding item is matched from 1 to many repetitions (+? for non-greedy form), e.g., Jon+ matches both Jon and Jonn, but not Jo.
{n}
Preceding item is matched n times (e.g., a{3}).
{n,}
Preceding item is matched n or more times (e.g., a{3,}).
{,m}
Preceding item is matched 0 to m times (e.g., a{,3}). This is specific to GNU grep.
{n,m}
Preceding item is matched from n to m times (e.g., a{1,3}).

As previously mentioned, regexes are greedy by default. For example, ^a.*a applied to the input string abacada matches the entire abacada string, not just aba or abaca. Non-greedy versions try to match as little of the input as possible, e.g., ^a.*?a would match aba.

Precedence

The precedence for regular expression concatenation, alternation, and repetition is as follows:

  1. Repetition
  2. Concatenation
  3. Alternation

For example, ab* is a single a followed by arbitrarily many bs (including none at all), not an arbitrary number of repetitions of ab.

Lookarounds

Lookarounds look around your regular expression matches, i.e., they look at the elements before or after your regex match.

(?=) Positive Lookahead
Matches the expression preceding the lookahead expression (e.g., q(?=u) matches a q followed by a u).
(?!) Negative Lookahead
Matches the expression preceding the lookahead expression that is not followed by the lookahead expression (e.g., q(?!u) matches a q not followed by a u).
(?<=) Positive Lookbehind
Get all matches preceded by a specific pattern (e.g., (?<=a)b matches a b that is preceded by an a).
(?<!) Negative Lookbehind
Get all matches that are not preceded by a specific pattern (e.g., (?<!a)b matches a b that is not preceded by an a).

Back References

A back reference \n, where n is a single digit, matches the substring previously matched by the nth parenthesized subexpression of the regular expression (e.g., (ab)\1 matches abab; (ab*a)x\1 matches abbaxabba), i.e., the regular expression query (denoted by use of parentheses) is repeated n times. When using the grep command and multiple regular expressions are given with the -e option or from a file (-f ex_file), back-references are local to each expression.

Flags

Flags are modifiers that redefine regular expression behavior.

i
Case insensitive matching.
g
Global search.
m
Multiline (i.e., anchor metacharacters work on each line).

Flags are placed at the end of a regular expression, e.g., /.+/g.

grep

The grep (Global Regular Expression Print) command prints lines that match patterns (i.e., regular expressions). The basic syntax for grep is:

grep 'ex_pattern' ex_file...

grep understand three different versions of regular expression syntax:

  1. Basic (BRE, Basic Regular Expression)
  2. Extended (ERE, Extended Regular Expression)
  3. Perl (PCRE, Perl-compatible Regular Expression)

By default, grep uses BREs. When using BREs, the operators +, ?, {, |, (, and ) lose their special meaning and must be escaped with a \.

Multiple files can be supplied to grep as arguments. If no file is given as an argument, grep uses the standard input for its input.

For grep, an exit status of 0 means a line was selected (i.e., there was a hit), an an exit status of 1 means no lines were selected (i.e., there were no hits), and an exit status of 2 means an error occurred.

To highlight matches in grep results, set a color in the GREP_COLORS (e.g., GREP_COLORS='0;31') variable and use the --color=always option with the grep command.

Options

Helpful grep options include:

-A ex_number, --after-context=ex_number
Print ex_number lines of trailing context after matching lines.
-B ex_number, --before-context=ex_number
Print ex_number lines of leading context before matching lines.
-c, --count
Output just the number of matching lines.
-C ex_number, --context=ex_number
Output context (i.e., additional lines surrounding the line with the hit.
-e ex_expression, --regexp=ex_expression
Introduce a regular expression to be searched. Can be used multiple times with the same grep command (e.g., grep -e ex_expression_1 -e ex_expression_2).
-E, --extended-regexp
Allow use of extended regular expressions.
-f ex_file, --file=ex_file
Read in search patterns (one per line) from a file. Searches are performed simultaneously.
-H, --with-filename
Print the filename for each match alongside each line match. This is the default behavior for grep when there is more than one file to search.
-i, --ignore-case
Perform case insensitive search.
-I
Process a binary file as if it did not contain matching data. This is equivalent to the --binary-files=without-match option.
-l, --files-with-matches
List just the names of matching files, not the actual line matches.
-L, --files-without-match
List just the filenames of non-matching files, not the actual line matches.
--line-buffered
Causes grep to write its output line by line, instead of buffering larger amounts of output, as it usually does, to make writing more efficient. This can cause a performance penalty.
-m ex_number, --max-count=ex_number
Stop reading a file after ex_number matching lines.
-n, --line-number
Include the line number of matching lines in the output.
-r, --recursive
Search recursively.
If a directory is not specified as an argument, grep will use the current directory as its argument and search recursively. Can only search contents of text files.
-v, --invert-match
Output lines that do not match the regular expression.
-x, --line-regexp
Output only exact whole line matches.
-w, --word-regexp
Output only exact whole word matches.

Documentation

You can learn more about regular expressions via the GNU Grep manual and the Python Software Foundation documentation. For more on grep, run man 1 grep or check out its man page online.

The best way to learn regular expressions is to start looking at examples and using them. Create a list of different text strings that you want to match and see what kinds of regular expressions you can create to target them.

For testing these expressions, an online tool like regular expressions 101 is invaluable. There, you can enter a test string and a regular expression. Then, you get a step-by-step breakdown of what each part of the regex is doing and what it matched (if anything) in your test string.

Avatar

Enjoyed this post?

Subscribe to the feed for the latest updates.