This final part of the article series covers some other important behaviour of bash, particularly that relating to text expansions and substitutions.

The famed text-processing abilities of the Bourne shell (sh) has been vastly expanded in the Bourne-Again shell (bash). These abilities are efficient to a fault and the code can be even more cryptic. This makes the bash command interpreter and programming language very powerful but also a minefield of errors.

Before going further, let us review what we learned in the last article:

• sh and bash are not the same
• if statements check for exit codes (0 for true and non-zero for false), not boolean values
• [, true and false are programs, not keywords
• Presence/absence of space in variable comparisons and assignments make quite a difference
• [[ is not the fail-safe version of [
• Arithmetic operations are not straight-forward as in other languages
• Array operations use cryptic operators
• By default, bash will not stop for errors

In this article, we will focus on how bash performs text expansions and substitutions. I will only cover what I think are the most important text-processing features. For comprehensive information, you will have to study the Bash Reference Manual. Bash and several commands such as sed and grep also use regular expressions to perform text processing. Regular expressions is a separate topic on its own and I will not cover them either.

History expansion character (!)

This feature is available when typing commands at the shell prompt. It is used to access commands stored in bash history file.

!n	Execute nth command in bash history
!!	Execute last command (Equivalent to !-1)
!leword	Execute last command beginning with ‘leword’
!?leword?	Execute last command containing ‘leword’
^search^replace	Execute last command after replacing first occurrence of ‘search’ with ‘replace’

You can modify the history search using certain word designators, preceded by a colon (:).

!?leword?:0	Execute with 0th word (usually the command executable) in last command containing ‘leword’
!?leword?:2	Execute with second word of last command containing ‘leword’
!?leword?:$	Execute with last word in last command containing ‘leword’
!?leword?:2-6	Execute with second word to sixth word in last command containing ‘leword’
!?leword?:-6	Execute with all words up to 6th word in last command containing ‘leword’ (Equivalent to !?leword?:0-6)
!?leword?:*	Execute with all words of last command (but not the 0th word) containing ‘leword’ (Equivalent to !?leword?:1-$)
!?leword?:2*	Execute with the second word to the last word in last command (but not the 0th word) containing ‘leword’ (Equivalent to !?leword?:2-$)
!?leword?:2-	Execute with all words from the 2nd position to last but-one word and not the 0th word in the command containing ‘leword’

Remember that bash will execute whatever you have retrieved from the history with whatever you have already typed at the prompt.

You can also use any number modifiers, each preceded by a colon (:).

!?leword?:p	Display (but not execute) last command containing ‘leword’
!?leword?:t	Execute with last command containing ‘leword’ after removing all pathname of last argument (i.e., leave the tail containing the file name)
!?leword?:r	Execute with last command containing ‘leword’ after removing the file extension from the last argument
!?leword?:e	Execute with last command containing ‘leword’ after removing pathname and file name from the last argument (leaving just the extension)
!?leword?:s/search/replace	Execute last command containing ‘leword’ after replacing the first instance of ‘search’ with ‘replace’
!?leword?:as/search/replace	Execute last command containing ‘leword’ after replacing all instances of ‘search’ with ‘replace’

If you omit the search text (‘leword’) and use the history expansion character with the word designators and the modifiers, bash will search the last command.

Until you become proficient in using the history expansion character, use the modifier :p to display the command before you actually execute it.

Text expansions and substitutions

These features are available at the shell prompt and in shell scripts.

• Tilde (~): In your commands, bash will expand instances of ~ with the value of the environmental variable $HOME, that is, your home directory.
• ? and *: These are metacharacters. In file descriptors, ? matches in for any one character while * matches any number of any characters. If they do not match any file names, bash will use their literal values.
• Brace expansion: You can use comma-separated text strings within curly brackets to generate combination of strings with their suffixes and/or prefixes. When I start a new book, I create its folders like this.

  mkdir -p NewBook/{ebook/images,html/images,image-sources,isbn,pub,ref}
  

  This command creates folders like this.

  NewBook
  NewBook/ref
  NewBook/pub
  NewBook/isbn
  NewBook/image-sources
  NewBook/html
  NewBook/html/images
  NewBook/ebook
  NewBook/ebook/images
  

• Parameter expansion: When bash executes a script, it creates these special variables for the script.
  

  Shell variable	Use
  $0	Name of the shell script
  $1, $2,…	Positional parameters or arguments passed to the script
  $#	Total count of arguments passed to the script
  $?	Exit status of last command
  $*	All arguments (double-quoted)
  $@	All arguments (individually double-quoted)
  $$	Process ID of current shell/script

  At the terminal, $0 will usually expand to the shell program (/bin/bash)

  On a terminal, you can use the set command to specify parameters to the current shell ipso facto.

  # Displays 0
  echo $#
  
  # Displays an empty string and causes a new line
  echo $*
  
  # Sets hello and world as parameters to current shell
  set -- hello world
  
  # Displays 2 (the number of parameters)
  echo $#
  
  # Displays hello world
  echo $*
  
  # Remove parameters to current shell
  set --
  
  # Displays 0 (as earlier)
  echo $#
  

  The option -- (two hyphens) represents the end of options and implies that whatever following it must be command parameters.

• Command substitution: Instead of backquotes, you can use the form $(commands) to capture the output of those commands for use in some other commands or variables. It makes quoting and escaping much more easier.
• Variable substitution: You can use these substitutions with command parameters (created by bash for a shell script) or with variables that you have created.
  

  Substitution	Effect
  ${var1:-var2}	If var1 is null or does not exist, var2 is used
  ${var1:=var2}	If var1 is null or does not exist, value of var2 is used and set to var1
  ${var1:?msg}	If var1 is null or does not exist, msg is displayed as error
  ${var1:+var2}	If var1 exists, var2 is used but not set to var1
  ${var:offset}	Everything of var after offset number of characters
  ${var:offset:length}	length number of characters of var after offset number of characters
  ${!prefix*} ${!prefix@}	All variables names beginning with prefix
  ${!var[@]} ${!var[*]}	All indexes of array variable var
  ${#var}	Length of value of var
  ${var#drop}	Value of var without prefix matching Regex pattern drop
  ${var##drop}	Empty string if prefix matches Regex pattern drop
  ${var%drop}	Value of var without suffix matching Regex pattern drop
  ${var%%drop}	Empty string if suffix matches Regex pattern drop
  ${var^letter}	Changes first letter of var to uppercase if it matches letter (any alphabet, * or ?) If letter is not specified, all first letter(s) of var will be changed to uppercase
  ${var^^letter}	Changes any letter of var to uppercase if it matches letter (any alphabet, * or ?) If letter is not specified, all letter(s) of var will be changed to uppercase
  ${var,letter}	Changes first letter of var to lowercase if it matches letter (any alphabet, * or ?) If letter is not specified, all first letter(s) of var will be changed to lowercase
  ${var,,letter}	Changes any letter of var to lowercase if it matches letter (any alphabet, * or ?) If letter is not specified, all letter(s) of var will be changed to lowercase
  ${var/find/replace}	Value of var with instances of find replaced with replace. If find begins with ‘#’, then a match is made at the beginning. A ‘%’ makes it match at the end.

Escaping

You can escape

• special characters using the backslash (\). To escape the backslash character, use double backslashes (\\).
• literal text strings by wrapping them in single quotation marks (' '). Bash will not perform any expansions or substitutions. The single-quoted string should not have any more single-quotation marks. Bash will not perform any backslash-escaping either.
• literal text strings by wrapping them in double-quotation marks (” “) but allowing for
  • $-prefixed variables, expansions and substitutions
  • backslash-escaped characters
  • backquoted (` `) command strings
  • history-expansion character

# Displays Hello World
a=World; echo "Hello $a"

# Displays Hello $a
a=World; echo 'Hello $a'

# Displays Hello 'World'
a=World; echo "Hello '$a'"

Printer’s error

The Bash Reference Manual or even this article may use wrong characters for the quotation marks. The apostrophe or u+0027 used in single-quoted strings may be replaced with the right single quotation mark or u+2019. The grave accent or u+0060 used in backquoted strings may be replaced with left single quotation mark or u+2018. The quotation mark or u+0022 used in double-quoted strings may also be replaced with left and right double quotation marks. They look similar but will result in an error if used in a shell script or in the command-line. I write my books and articles in CommonMark (MarkDown) and output them as HTML, ODT, EPUB and PDF documents. These documents will not have such errors. When someone edits the document (before it goes to print) in a rich-text editor such as LibreOffice or Microsoft Office or Adobe Indesign, that software’s autocorrect feature may change ordinary quotation marks and backquotes with inverted quotation marks. Just be aware that this can happen. To avoid mistakes, just type the commands by hand. Do not copy-paste them.

Summary

I am sure you will also conclude that bash code can be very cryptic. A lot of production code (industrial-strength shell scripts) are hundreds of lines long. If bash was not so succinct and powerful, it would take forever to write the lines. If you are doing any kind of serious shell scripting, then it is best if you know all about bash‘s myriad secrets. I think I have covered enough of them to kindle your interest. You are on your own now.

---

• This article was originally published in the Open Source For You magazine in 2022. I have re-posted it on CodeProject in 2023.
• This article has been sourced from my book Linux Command-Line Tips & Tricks. It is available for free on many ebook stores.

Last year, I wrote a book on Linux command-line tips and tricks (see note below) and made several updates to it. Annoyingly, I continue to discover something new and important about the bash shell program almost every week. I do not want this happening after I had ordered my author copy. The discoveries make me wonder what I have been doing all these years without knowing these bash secrets.

sh and bash are not the same

The Bourne shell (sh) program began its life in the 70s with the Unix operating system. In Ubuntu Linux, they continue to have this old shell alongside its new avatar, the Bourne-Again shell (bash). Try bash -version in the command line, it will display its version number. Try sh -version, you get an error. The two are different. While sh remains an ancient relic, bash continues to be developed and has a lot more features.

It was my practice (in the late 90s) to run my shell scripts in SCO Unix with the sh command. I continued this in Ubuntu and found that a lot of online script examples did not work with it. (As a security measure, I never give the extension .sh or the +x permission to my scripts. My scripts remain anonymous with an innocuous .txt extension.)

Aware of this problem, a lot of script authors place a comment #!/bin/bash on the first line. This comment ensures that the script will be run with bash even if it is invoked with sh.

Some fanatics use the comment #!/usr/bin/env bash instead as a more failsafe measure. They say that bash may not always be at /bin so it is better make env to find it. By this, they assume that env will always be found at /usr/bin. Seems overkill to me. If you are on Ubuntu as most people are, then #!/bin/bash should do fine.

if statements are not what they seem to be

The shell’s if statement is very unusual.

if test-expression; then

   statements;

else

   statements;

fi

The test-expression needs to return 0 (zero) to be true and any non-zero value to be false. In most languages, 1 (one) evaluates as true and 0 (zero) evaluates as false. Why does bash behave differently?

This is because shell scripts often need to determine how other programs have performed. They do this by reading the exit value of those programs. By convention, when a program exits without an error, it returns control to the invoking program (the shell program) with an exit code of 0 (zero). If it needs to exit after encountering an error, it returns with a non-zero exit code. To help in troubleshooting, program authors publish special meaning to each non-zero exit code.

Thus, in the if statement, the test-expression could be a program. If the program executed successfully and returned 0 to the shell, then the if statement behaves as if it evaluated to true. If the program exited with a non-zero value, then the if statement behaves as if evaluated to false.

What you need to remember is that the if statement is not looking for the boolean values true or false.

Does this mean that if true; then will evaluate to false because it is not 0 (zero)? No!

That brings us to another strange feature of shell. true is actually a program! It is not part of the shell language. In Ubuntu, it resides in /usr/bin/true and it exits with a return code of 0. There is also a false program residing at /usr/bin/false and it exits with a return code of 1.

[ is a program

To test whether a file exits, you can use if [ -f the-file.ext ]; then. Here, the single bracket [ is not part of the language. It is a program at /usr/bin/[ and its arguments are: -f, the-file.ext and ]

To ensure that the [ commands are executed properly, there has to a space after the opening bracket and before the closing bracket. If you omit the first, you are not invoking the correct program. If you omit the latter, you failed to terminate the command with the correct closing argument.

Beware of space in string comparisons

When you assign a value to a string variable, DO NOT leave any space before and after the = sign. If you do, it seems to the shell that the variable is a command and the = and the attempted value for the variable are its arguments.

When you test whether two strings are equal, DO leave a space before and after the = sign. If you do not, the [ program will think you are trying to make an assignment. This assignment statement has an exit value of 0 (zero). That means the if statement will always be forced to evaluate to true!

# Causes an error because 'sTest' looks like a command 
# and '=' and '"hello"' become its arguments
sTest = "hello"

# Assigns string variable correctly
sTest="hello"

# Temporary assignment evaluates to true whatever the value
if [ "$sTest"="hellooooooooo" ]; then
  echo "Yep"
else
  echo "Nope"
fi

# String comparison evaluates to true
if [ "$sTest" = "hello" ]; then
  echo "Yep"
else
  echo "Nope"
fi

[[ is not the fail-safe version of [

Unlike [, which is a program, the [[ construct is a part of the shell language. Some misguided fellows on the Internet recommend that you replace all your [ evaluations with [[ ones. Do not follow this advice.

[[ are used for a more literal evaluation of text strings. You do not have to quote everything.

• Words and file names are not expanded. However, other forms of expansion such as expansions and substitutions are performed.
• The = operator behaves like the way = or == operators do with [.
• The != and == operators compare the text expression on the left with a pattern on the right.
  • A pattern is a text string containing at least one wildcard character (* or ?) or a bracket expression [..]. A bracket expression encloses a set of characters or a range of characters (separated by a hyphen (-)) between the square brackets ([ and ]).
• A new =~ operator is available. (It cannot be used with [.) It compares the text on the left with a regular expression on the right. (It will exit with a return value of 2 if the regular expression is invalid.)

  The =~ operator is great for matching substrings.

  # Matches substring ell
  $ if [[ "Hello?" =~ ell ]]; then echo "Yes"; else echo "No"; fi
  Yes
  
  # Matches substring Hell at beginning
  $ if [[ "Hello?" =~ ^Hell ]]; then echo "Yes"; else echo "No"; fi
  Yes
  
  # Does not match substring ? (a regex special character) at the end
  $ if [[ "Hello?" =~ ?$ ]]; then echo "Yes"; else echo "No"; fi
  No
  
  # Matches substring ? at the end when quoted
  $ if [[ "Hello?" =~ "?"$ ]]; then echo "Yes"; else echo "No"; fi
  Yes
  

Evaluations with [ and [[ have their legitimate use-cases. Do not use one for the other.

Operator use	Result
[ -f "$file" ]	Does it exist as a file?
[ -d "$file" ]	Does it exist as a directory?
[ -h "$file" ]	Does it exist as a soft link?
[ -r "$file" ]	Is the file readable?
[ -w "$file" ]	Is the file writeable?
[ -x "$file" ]	Is the file executable?
[ -z "$string" ]	Is the string empty?
[ -n "$string" ]	Is the string not empty?
[ "$string1" = "$string2" ]	Are the strings same? = is same as ==
[ "$string1" != "$string2" ]	Are the strings different?
[ "$string1" < "$string2" ]	Does first string sort ahead of second?
[ "$string1" > "$string2" ]	Does first string sort after second?
[ n1 -eq n2 ]	Are the numbers same?
[ n1 -ne n2 ]	Are the numbers different?
[ n1 -le n2 ]	Is n1 less than or equal to n2?
[ n1 -ge n2 ]	Is n1 greater than or equal to n2?
[ n1 -lt n2 ]	Is n1 less than n2?
[ n1 -gt n2 ]	Is n1 greater than n2?
[ ! e ]	Is the expression false
[ e1 ] && [ e2 ]	Are both expressions true?
[ e1 ] || [ e2 ]	Is one of the expressions true?

Do not use -a and -o logical operators. You will make mistakes reading and writing them. They are the sh way of doing things. Square brackets and operators && and || are so bash.

Arithmetic operations are not straight-forward

If you set a=1 and then try a=a+1, does $a echo as 2 or 11? The answer is a+1. Until a few years back, I did not know how to perform arithmetic operations in bash. I never had to so I never learned it. I just assumed that it must be the same as in other languages but it was not to be. To add one plus one, you can use:

let a=a+1

# or

a=$(( a+1 ))

Array operations can be cryptic

Does every language out there need to have a totally different method to create and use arrays? Who so evil? Why?

# Creates an array
var=(hello world how are you)

# Displays hello
echo $var

# Displays how
echo ${var[2]} 

# Changes hello to howdy
var[0]=howdy

# Displays howdy
echo ${var[0]}

# Displays values — howdy world how are you
echo ${var[@]}

# Displays values — howdy world how are you
echo ${var[*]}

# Displays indexes or keys — 0 1 2 3 4
echo ${!var[@]}

# Displays indexes or keys — 0 1 2 3 4
echo ${!var[*]}

# Displays dy
echo ${var:3:2}

# Displays rld
echo ${var[1]:2:3}

# Displays 5, the number of variables in the array
echo ${#var}

bash is a minefield of careless errors

A shell script will execute like no tomorrow irrespective of any errors it encounters. If a statement encounters an error and exits with a non-zero exit code, bash is happy to display any error message it wants but will nonchalantly continue to execute the subsequent statements.

If you try to use an undefined variable, bash will not treat it as an error. bash will substitute an empty string and proceed. If you try sudo rm -rf $non-existent-variable/, the command evaluates to sudo rm -rf /. I have not tried it yet so I cannot tell what protections Linux has.

These shell behaviours are extremely dangerous. To fail early, place the following statement at the top of your scripts.

set -eu

That is, after the #!/bin/bash comment. If you are using if-else constructs, just use set -u. The option -e stops execution of the script when an error is encountered so your code will never get a chance to evaluate the error code of the previous statement.

It is not possible to cover all bash secrets in one article. In the next article, I will cover how bash performs text expansions, substitutions and removals.

---

Notes:

• This article was originally published in the Open Source For You magazine in 2022. I have re-posted it on CodeProject in 2023.
• I have made the ebook version of Linux Command-Line Tips & Tricks free on many ebook stores. This article has been sourced from it.
• Warning to Richard MacCutchan and allies: Do not get overzealous again and ban this article and my account like you did before. Read the previous warning by Sean Ewington. The same applies for this article too. I have written 29 mostly non-fiction books. I do not have to copy anything from anyone.