PBS 148 of X: Potpourri (Subshells, Relative Paths & More) (Bash)

Sometimes the simplest of tasks expose the biggest of knowledge gaps! When attempting to solve the bonus extra part of the challenge set at the end of the previous instalment I fell into a 3-hour hole of frustration that exposed a shortcoming in my understanding of Bash that I’d somehow managed to avoid being bitten by for years. This serves as a great example of how even seasoned coders never ever stop learning and always find new rakes to step on! The next time you hit one of those coding bugs where it feels the entire universe has stopped making sense, remember it happens to each and every coder, no matter how well-seasoned they may be!

This instalment is an unplanned pause for one very important concept I’d overlooked (subshells) and some other small things that didn’t fit in elsewhere.

Matching Podcast Episode

Listen along to this instalment on episode 763 of the Chit Chat Across the Pond Podcast.

You can also Download the MP3

Read an unedited, auto-generated transcript: CCATP_2023_03_25

Episode Resources

• The instalment ZIP file — pbs148.zip

Introducing the Programming By Stealth Student Organization in GitHub

We now have a place on GitHub to share our homework. Read about it and join at git

PBS 147 Challenge Solution

The challenge was to store a breakfast menu in an array, use a select loop to allow users to choose items from the menu, capture the user’s choices, and then print out their order when they indicate that they’re done choosing items. For bonus extra credit, the challenge was to replace the hard-coded menu array with an array loaded from a text file.

Let’s start with my sample solution to the base challenge. (which you’ll find in the instalment ZIP as pbs147-challengeSolution.sh):

#!/usr/bin/env bash

# define the menu
menu=(pancakes waffles porridge sausages bacon eggs spam tea coffee 'orange juice')

# create an empty array to hold the order
declare -a order

# present the menu, with a done option
echo 'Choose your breakfast'
select item in done "${menu[@]}"
do
    # skip invalid selections ($item is empty)
    [[ -z $item ]] && continue

    # exit if done
    [[ $item == done ]] && break

    # store and print the item
    order+=("$item")
    echo "Added $item to your order"
done

# print the order
echo -e "\nYou ordered the following ${#order[@]} items:"
for item in "${order[@]}"
do
    echo "* $item"
done

For the most part, this is very by the book. I create an array of strings, being sure to quote the item with a space in it. I then create an empty array that will hold the user’s order with the declare keyword before using a select loop to offer the user a choice from the menu.

We want to offer the user the choice of the entire menu, and the option to say they’re finished. Remember, select expects to be passed the options to display as a list of arguments after the special argument in. I chose to add the option to end the loop first, so the first item in my list is done. Next, I explode the menu array into a list of arguments with the ${array[@]} syntax, being sure to quote the expansion so orange juice remains a single item.

Within the loop, I use the && shortcut in conjunction with the [[ conditional syntax and the break command to exit the select loop when the user chooses option 1. Still within the select loop, I use the += operator to append the user’s chosen items to the order array. (I know I’ve skipped a line, put a pin in it for later 🙂)

Finally, I use a for in loop to print out the user’s order, and proceeded with an example of using the ${#array[@]} syntax for accessing the length of an array for completeness.

Ignoring Invalid Selections in select loops

Something we’ve glossed over up to this point is what happens in a select loop if the user enters a number that’s outside the range of displayed options, or if they type some other random text. What does Bash do? Throw some kind of error? Ask for another selection? Nope — it runs the loop with an empty string as the value for the looping variable (item in my sample solution).

So, it’s up to your scripts to choose how they react to invalid values, and you detect them by checking if the looping variable is empty, hence the following lines in my sample solution:

# skip invalid selections ($item is empty)
[[ -z $item ]] && continue

PBS 147 Challenge Solution with Bonus Extra

It literally took me 5 minutes to write the sample solution above, and I was expecting it to take me just 5 more minutes to write the solution for the extra challenge to read the menu from a file, but that’s not what happened! It took me 3 hours to successfully load the lines from a text file into an array while ignoring blank lines and comment lines. You’ll find my full sample solution in the file pbs147-challengeSolution-bonusCredit.sh, but it’s identical to my original sample solution except that instead of declaring the menu array, I load the menu from a text file. To test that I was correctly ignoring blank lines and comment lines, I used the following file (menu.txt) as the source for my menu:

# sweet stuff
pancakes
waffles

# healthy stuff
porridge
muesli

# fried
sausages
bacon
eggs
spam

# drinks
tea
coffee
orange juice

I use the following snippet to load the menu:

# read the menu
declare -a menu
while read -r menuLine
do
    # skip comment lines
    echo "$menuLine" | egrep -q '^[ ]*#' && continue

    # skip empty lines
    echo "$menuLine" | egrep -q '^[ ]*$' && continue

    # store the menu item
    menu+=("$menuLine")
done <<<"$(cat $(dirname "$BASH_SOURCE")/menu.txt)"

There is a lot going on here, so much we’ll be dedicating the remainder of this instalment to understanding these few lines of code!

Bash Subshells

We’ve seen that you can use syntax of the form $(command) to capture the result of running a Bash command within a Bash script. What is happening here is actually two-fold — the ( and ) are creating a subshell, and the $ is capturing its output.

A subshell is a Bash shell within a Bash shell, and the key point is that subshells get their own scope! When you create a subshell it inherits a copy of its parent scope (usually your script’s scope), so you can access the values in all the variables you’ve created before, but if you change any variables, those changes are not seen in the parent scope!

You can see this yourself by executing the following commands in an interactive Bash shell (terminal):

bash-3.2$ desert=waffles
bash-3.2$ (echo "$desert")
waffles
bash-3.2$ (echo "$desert"; desert=pancakes; echo "$desert")
waffles
pancakes
bash-3.2$ echo "$desert"
waffles
bash-3.2$

I create a variable named desert in the terminal’s scope, I then access that variable in a subshell and it prints the expected value because the echo command in the subshell sees a copy of the parent scope’s desert variable.

Next, I echo the variable, change it, and echo it again all within a subshell. It behaves as expected, printing the original value and the new value.

Finally, not in a subshell, I echo the original variable again — it has not changed!

I knew that when I intentionally create subshells they get their own scope. I had planned to mention that subtlety in the final wrap-up episode because 99.9% of the time it’s not important to know, and I didn’t want to confuse you all too early in the series.

What I didn’t know was that Bash doesn’t only create subshells when you explicitly ask it to, it also does it implicitly in some circumstances!

Beware Stealthy Subshells

When you chain commands together in a pipeline, each command is execute in its own subshell!

This is what caused me 3 hours of frustrated confusion. We have seen that the following works:

path=/etc/hosts
cat "$path" | while read -r hostsFileLine
do
	echo "$path: $hostsFileLine"
done

This prints each line in /etc/hosts pre-fixed with the path to the file.

This lead me to assume the following would work:

declare -a menu
cat ./menu.txt | while read -r menuLine
do
	menu+=("$menuLine")
done
echo "${menu[@]}"

What did I discover? The menu array remained stubbornly empty! Adding an echo statement inside the loop showed the loop was successfully executing over each line of my text file, but my array remained empty? 😖🤬😕

The root of my problem was that pipe passing the output from cat to while. The entire while loop executes in a subshell! So, inside my loop, there was a copy of the empty menu array, and I was adding items to it, but the moment the loop ended, that copy of the array vanished!

It took a lot of very confused Googling to learn that pipes create implicit subshells. The moment I saw that sentence in a stack overflow answer, the clouds parted and it all made sense, but I used a lot of choice words before I got that far 😉

OK, so piping from cat won’t work, what will? The answer is a Bash feature I’ve been trying to find an excuse to include in the series somewhere, multi-line strings!

Here Strings

For reasons I don’t understand, the official Bash documentation refers to multi-line strings as here strings. The syntax uses three chevrons followed by a single string that can contain newline characters, and can be an un-interpolated string, e.g.:

cat <<<'un-interpolated
two line string'

Or an interpolated string:

breakfast=porridge
cat <<<"interpolated two line 
string with some $breakfast"

How does this help with our loop?

Well, notice that here strings get used as standard input for the command on their left (cat in our examples). And, critically, unlike pipelined commands, here strings do not create implicit subshells!

We still need to use a subshell to read the contents of our menu file, but we can have our while loop be outside the subshell!

We can re-write our broken code at the start of this section like so:

declare -a menu
while read -r menuLine
do
	menu+=("$menuLine")
done <<<"$(cat ./menu.txt)"
echo "${menu[@]}"

The key point to note is that the subshell is now entirely contained within the here string, and the while loop and all its contents remain in our script’s scope, so, when we append to the menu array, we’re appending to the real menu array!

Loading Content Relative to the Script’s Path

If you look at my sample solution you’ll notice I don’t read from ./menu.txt, but from $(dirname "$BASH_SOURCE")/menu.txt, what’s going on here?

When you execute a script, the present working directory is the path your terminal is at when you run the script. If you always run your scripts from the folders they’re saved in, then that’s a distinction without a difference, but if you save your script in one folder and run it from another, your relative paths break!

The Special $BASH_SOURCE Variable

The first component to solving this problem is the special variable $BASH_SOURCE which Bash creates when it executes code from a script file. This variable points to the path to the script that is executing. Note that it points to the full path, including the file name.

The dirname & basename Terminal Commands

To access a file next to your script, you need to strip the file name from $BASH_SOURCE. You could do that with regular expressions, but that would be a brittle solution, and someday, a special character in a file path will break one of your scripts in a very confusing way.

Thankfully, there are a pair of terminal commands for splitting file paths into the folder and file parts, dirname gives you the folder, and basename the filename, e.g.:

bash-3.2$ echo "$(dirname /etc/hosts)"
/etc
bash-3.2$ echo "$(basename /etc/hosts)"
hosts
bash-3.2$

Final Thoughts

If any of you ever feel infuriated that you can’t get even a simple piece of code to work, that the whole universe just isn’t making sense anymore, and that this clearly broken code absolutely should work, remember it happens to everyone who codes! There will be a technical explanation, and you will find it, but it’s OK to be derailed for a few hours. It’s not a sign of inexperience or incompetence, it’s just one of those things that happens to absolutely everyone!

Next time we’ll pick up our planned narrative again, looking at how to pass named arguments to scripts, and how to make your scripts compatible with command pipelines.

An Optional Extra Bonus Challenge

In the meantime, if you want some more Bash practice, update your solution to the challenge to accept an optional argument limiting the number of items a user can order from the breakfast menu.