PBS 143 of X: Shell Script Basics (Bash)

In the previous instalment we described the project structure we’ll be using for the port of the Crypt::HSXKPasswd Perl module to JavaScript. One of the folders in that structure is /buildScripts/, which we described as containing the various scripts that will be used to build the project. These scripts will be shell scripts, which are new to us, so it makes sense to spend a little time learning some shell scripting basics at this stage. This won’t be an exhaustive look at shell scripting, far from it, we’ll just be focusing on what we need for the build scripts. Our focus will be on script portability, variables, conditional statements, loops, and error handling. In other words, the most basic programming fundamentals.

Throughout this instalment we’ll be calling out specific chapters in the Taming the Terminal series which you can access through several means. You can access the entire series via a web browser at ttt.bartificer.net/… and you can listen to the companion podcast Allison and I created along with the full text. You can also download Taming the Terminal in book form through the Apple Books app. You can find other forms of the book at podfeet.com/tttbook.

Matching Podcast Episode

Listen along to this instalment on episode 757 of the Chit Chat Across the Pond Podcast.

You can also Download the MP3

Read an unedited, auto-generated transcript: CCATP_2023_01_07

The Absolute Fundamentals — Hello Shell World

Just to be clear, when we use the term shell in this series we’re using it as an abbreviation for the technically correct term command shell. As described in detail in the first instalment of the Taming the Terminal series, a command shell is an environment for executing terminal commands. A terminal window is simply an interactive command shell. So, each time we’ve used the terminal in this series we’ve been using our OS’s default command shell. Shell scripts simply automate the process of entering terminal commands — the first line of a shell script tells the OS which shell to use to execute the remaining lines as commands.

Why Bash?

The shell scripts we’ll be using in the XKPasswd project will use the Bash shell. Bash is an extended and improved version of the original Unix shell (sh), and it was itself extended and improved to form the Z Shell (zsh). Why use Bash and not the more modern Zsh? Simple — maximal compatibility, even Windows 11 can run Bash scripts with installation of some optional extra packages!

You may have heard that recent versions of macOS use Zsh as their default shell, does that not mean that Bash is going away on the Mac? Nope! There’s a big difference between an OS’s default shell, the one used to power interactive terminal windows, and the set of shells the OS provides for scripting. Not only do the very latest versions of macOS support Bash scripts, they still support sh scripts!

Script Structure

As mentioned in the introduction, the first line of a shell script tells the OS what shell with which to execute the remaining lines of the script. For reasons that are still very much debated, this is called a shebang line. We’ll look at the best shebang line for our scripts in a moment. Each line after the shebang line is treated as a separate command, though blank lines are ignored.

Comments can be added by using the octothorp character at the start of a line, or at the start of a word to ignore the remainder of a line, e.g.:

# a full-line bash comment
echo 'hello world' # an end-of-line bash comment

By default each line is treated as a separate command, so long commands need to go onto long single lines. You can avoid this by using a backslash (\) as the last character on the line, this effectively escapes the end-of-line character.

echo 'first argument to echo' \
  'second argument to the echo command started above'

A Portable Shebang Line

The simplest form of shebang line is an octothorp (#) followed by an exclamation, followed by the full path to the shell. If Bash was reliably stored in the same location on every OS, we could use a shebang like the one below reliably:

#!/bin/bash

Unfortunately, different OSes and even different Linux distros store Bash in different places, so we need to add a little indirection into our shebang line to account for that.

Thankfully, the standard env terminal command can be used to find bash on the system and then execute it, so that’s what we’ll be doing. Our shebang line will be:

#!/usr/bin/env bash

Our First Shell Script

As is traditional, we’ll start our shell script journey by writing a script to print the phrase Hello World. Using your favourite plain text editor, create a file named helloWorld.sh with the contents below, save it to a folder of your choice, open a terminal, and navigate to your chosen folder.

#!/usr/bin/env bash

echo 'Hello World!'

The first line is our shebang line, and the only other line is the command to print the phrase, so in Bash, the print command is echo.

Before we can run our script we need to make it executable. chmod changes the permissions of a file and adding +x makes the file executable:

chmod +x helloWorld.sh

Then, you can execute it with:

./helloWorld.sh

This should print “Hello World!” in your terminal session.

Shell Variables

You’ll find more details in instalment 12 of the Taming the Terminal series, but the basics of shell variables is that their default scope is the script in which they are defined. They’re created by assigning them a value with the = operator, and they’re accessed by prefixing their name with the $ operator.

The most important thing to remember is that variable definitions must be cuddled, that is to say, there cannot be a space on either side of the = symbol.

Note that an interactive terminal, i.e. a terminal window, behaves like a shell script you’re writing in real time, so you can experiment with variables without the need to write scripts.

As an initial example, we can create a variable named desert with the value waffles with the following command:

desert=waffles

We can then use this variable in shell commands by prefixing it with a $ symbol, e.g.:

# print the value of the desert variable
echo $desert

# use the wc command to count the letters
echo -n $desert | wc -c

Note the use of command pipelining to use the output of the echo command as the input to the wc command. You can learn all about this kind of terminal plumbing (as I call it) in instalments 15 & 16 of Taming the Terminal. As a bonus tidbit, note the use of the -n flag to tell echo not to add a trailing newline character to its output (without it the count would be out by one).

Everything is a String

Bash is very primitive when it comes to data types. As far as it’s concerned it’s all just strings that get passed about — if terminal commands want to treat a value as a number, it’s up to them to do the needed conversions themselves.

Because all values are strings, you don’t need to quote them unless doing so makes things easier for you. If your string contains no spaces or special characters you can just use it bare. If you think about it, you know this already because most of the time, we don’t quote our arguments to terminal commands!

The following three commands have the same effect:

echo pancakes
echo 'pancakes'
echo "pancakes"

Things change when your desired value contains spaces and/or special characters. When this happens you have three choices:

1. Escape every special character.
2. Use an uninterpolated string by wrapping the value in single quotes (can’t handle values which contain single quotes)
3. Use an interpolated string by wrapping the value in double quotes and escaping any " & $ characters with a backslash

The following three commands also have the same effect:

echo I\ like\ \$5\ pancakes
echo 'I like $5 pancakes'
echo "I like \$5 pancakes"

Note that not every string can be expressed in each of the three ways, an interpolated string can’t end in an exclamation mark!

# both work fine
echo I\ like\ \$5\ pancakes!
echo 'I like $5 pancakes!'

# DOES NOT WORK - causes error
echo "I like \$5 pancakes!"

How can we avoid this problem? The answer is string concatenation, which Bash allows you to do by literally putting two strings back-to-back, so the following two do work:

# interpolated string concatenated with bare string
echo "I like \$5 pancakes"!

# interpolated string concatenated with uninterpolated string
echo "I like \$5 pancakes"'!'

Injecting Values with Interpolated Strings

So we’ve seen that double-quoted strings behave differently to single-quoted strings, and we’ve referred to them as interpolated, what does that mean? It means we can use the $ operator to inject values into interpolated strings!

There are two syntaxes, and a shortcut you can sometimes get away with. Let’s start with the two syntaxes:

1. Insert the value of a variable into a string with ${varName} where varName is the name of a variable.
2. Insert the result of executing a command into the string with $(command) where command is a terminal command which can have arguments.

Finally, the shortcut is that when there is no ambiguity, we can omit the curly braces when inserting the value of a variable, so the following two echo commands are equivalent:

desert=waffles
echo "I like ${desert}"!
echo "I like $desert"!

We’ve now seen variable injection, so now let’s look at command injection.

Before we use it in an interpolated string, let’s build out a command that will return the number of letters in a variable’s value. First, lets update our desert variable:

desert=pancakes

Now let’s try count the letters:

echo -n $desert  |  wc -c

If you run this command you’ll initially think we’ve succeeded, but actually, there’s a subtle problem, the number is prefixed with spaces. How do we get rid of those? There are lots of tricks for doing this, but a common one is to pipe the string that needs its leading or trailing spaces removed through the xargs command, so the following works as desired:

echo -n $desert  |  wc -c  |  xargs

Now we can put it all together and create an interpolated string that includes variables and the result of executing a command:

echo "your favourite desert is $desert, it has $(echo -n $desert  |  wc -c  |  xargs) letters"

Final Thoughts

We’re well on our way to being competent shell scripters — we can write and execute scripts, and we can create and use variables in our scripts. In the next instalments we’ll move on to dealing with inputs to and outputs from scripts, including passing arguments to scripts, before moving on to look at basic flow controls like conditionals and loops.