PBS 153 of X: Functions & Scope (Bash)
We’re very near the end of our exploration of Bash, but we’re missing one vital tool from our toolbox — easy code reuse through functions. As things stand, if we want to use the same lines of code multiple times, we need to copy and paste them, and in software engineering jargon that’s one heck of a bad smell!
In this instalment we’ll learn how to use POSIX functions to reuse Bash code efficiently, and in the process, we’ll be forced to grapple with the unusual way in which shell scripts handle scope. Most languages, including JavaScript and PHP, use lexical scope, but POSIX-compliant shell scripting languages like Bash use a completely different approach, dynamic scope.
Lexical scoping (also referred to as static scoping) refers to when the location of a function’s definition determines which variables you have access to. Dynamic scoping uses the location of the function’s invocation to determine which variables are available.
There is no right or wrong here, the key point is that shell scripts deal with scope differently to most other languages you’re likely to meet. You need to keep that in mind as you script, or you’ll be bamboozled by spooky action at a distance errors that will drive you around the twist!
Matching Podcast Episodes
Listen along to this instalment on episode 774 of the Chit Chat Across the Pond Podcast.
You can also Download the MP3
Read an unedited, auto-generated transcript: CCATP_2023_07_29
Episode Resources
- The instalment ZIP file — pbs153.zip
PBS 152 Challenge Solution
The challenge set at the end of the previous instalment was to update your solution to the previous challenge (to print multiplication tables) to make use of arithmetic expressions and/or xargs to improve your code as appropriate.
You’ll find my full sample solution in the instalment ZIP as pbs152-challengeSolution.sh, but it’s largely unchanged because I was already using xargs. All that remained for me to do was convert calls to the basic calculator (bc) to arithmetic expressions. For the most part, those changes were very mundane, e.g.:
capLen=$(echo "8+$nLen+$maxMLen+$maxPLen" | bc)
Became:
capLen=$(( 8 + nLen + maxMLen + maxPLen ))
However, this is one change that’s worth a deeper look.
Cleaner getopts with an Arithmetic Expression
We know that when we’re done processing optional arguments with getopts we need to use the shift command to clean the appropriate amount of entries from the front of the arguments array. We need to shift by one less than the value of $OPTIND, so we have been doing that by using the bc command like so:
shift $(echo "$OPTIND-1" | bc)
We can now make this line easier to read and understand with an arithmetic expression:
shift $(( OPTIND - 1 ))
Introducing Functions
As we’ve mentioned before, Bash is a super-set of the POSIX standard. That means it supports multiple different syntaxes for defining functions. In this series we’re only going to use one of those syntaxes — the one defined in the POSIX standard. The advantage of learning to write POSIX functions rather than Bash functions is that our syntax will work in other shell scripting languages too.
Think of Functions as Custom Terminal Commands
When we were learning JavaScript we described functions as black boxes that take input and produce output. If you squint at POSIX functions just right, they do fit that model, but for your sanity, you should probably not think of POSIX functions that way. Instead, think of POSIX functions as custom terminal commands you get to define.
POSIX functions can accept arguments, and they can alter variables, but, like other terminal commands, their primary IO is via the standard POSIX streams ( STDIN, STDOUT, and STDERR). We know that each time we execute a terminal command or a shell script, that command or script gets its own streams, which default to inheriting from the calling process, but can be changed using the stream redirection operators like | , > etc. Also, like regular terminal commands and scripts, the return value from a POSIX function is an exit code, specifically, the exit code of the last command executed within the function.
The POSIX Function Syntax
A POSIX function is simply a named sequence of terminal commands, and the syntax is simply:
FUNCTION_NAME () {
# TERMINAL COMMANDS
}
Beware — never put anything into those round brackets! Those of us who have used C-style programming languages like JavaScript are probably expecting those parentheses to be some kind of placeholder where we define the function’s arguments, but no, that’s not how POSIX works! The only purpose those parens serve is to mark the start of a function definition — think of them as a single symbol that means “this is a function”.
Once a function has been defined, you use it as if it were a standard terminal command.
As a simple example, let’s look at pbs153a-fn.sh in the instalment ZIP:
#!/usr/bin/env bash
# store the number of times to call the function
fnCounter=1
echo "$1" | egrep '^\d+$' -q && fnCounter=$1
# define a POSIX Hello World fuction
# Arguments: NONE
hellow () {
echo 'Hello World!'
echo '(from inside a POSIX function)'
}
# call simple greeting function the appropriate number of times
while (( fnCounter > 0 )) # NOTE the arithmetic expression
do
# call the function
hellow
# decrement the counter with an arithmetic expression
(( fnCounter-- ))
# print an empty line unless done
(( fnCounter > 0 )) && echo ''
done
This script defines a POSIX function named hellow, then calls it once by default, or, the number of times specified in the first argument to the script, e.g.:
$ ./pbs153a-fn.sh
Hello World!
(from inside a POSIX function)
$ ./pbs153a-fn.sh 3
Hello World!
(from inside a POSIX function)
Hello World!
(from inside a POSIX function)
Hello World!
(from inside a POSIX function)
$
The key point to note is that we define the function hellow like so:
hellow () {
echo 'Hello World!'
echo '(from inside a POSIX function)'
}
And we call with simply:
hellow
As a quick aside, note the script’s use of arithmetic expressions for making numeric comparisons and decrementing the counter.
As we learned in the previous instalment, arithmetic expressions without a leading $ evaluate to an exit code, and inside arithmetic expressions, variable names do not get prefixed with $, so to print a new line only when the counter is greater than zero we can simply use:
(( fnCounter > 0 )) && echo ''
Similarly, I used an arithmetic expression to decrement the counter:
(( fnCounter-- ))
Functions and Streams
Like every script and terminal command gets its own versions of each of the three standard streams (STDIN, STDOUT & STDERR) each time they’re executed, so do functions. This means commands in your functions can use the three streams just like any other terminal command can, and your functions can have their inputs and outputs redirected just like any other terminal command. To show this in action, let’s have a look at pbs153b-fnStreams.sh in the instalment ZIP:
#!/usr/bin/env bash
# define a POSIX function to mirror a string into a palindrome
# Arguments : NONE
# STDIN : string
# STDOUT : string
pal () {
# slurp STDIN into a variable
str=$(cat)
# write the original string back out to STDOUT (without trailing newline)
echo -n "$str"
# write the reverse of the string to STDOUT
echo "$str" | rev
}
# call the palindrome function in a pipeline
echo 'Your palindromic username: '$(echo "$USER" | pal)
# write a palindrome of the hostname in all upper case to a variable
palHost=$(hostname | pal | tr a-z A-Z)
# now print the variable in all upper case
echo "Upper Case Palindrome Hostname: $palHost"
Note that this example makes use of the rev command which writes a reversed version of STDIN to STDOUT (for more, see the man page).
Note that with the pal function we use the cat command to read from the function’s STDIN and save the contents to a variable named str, and we use the echo command to write to the function’s STDOUT.
When it comes to making use of our function, we use the standard terminal plumbing to redirect text into and out of the function. Notice that there is no difference in how we use our pal function compared to how we use standard terminal commands like rev and tr.
Functions and Exit Codes
Each time you call a function it returns an exit code. It will always be the exit code of the last command that executed inside the function. Sometimes that’s all you need, but sometimes it can be helpful to be more explicit and to clearly return an error code of your choosing. This is where the return keyword comes in. Like in other languages, return ends the function’s execution. Beware — in Bash return a function with an exit code, unlike in other languages, it cannot be used to return arbitrary values, you should use STDOUT for that (like in our pal example above).
As an example, pbs153c-fnReturn.sh in the instalment ZIP defines a function named is_int that returns with a successful code if STDIN contains an integer:
#!/usr/bin/env bash
# define a POSIX function to test if a value is an integer
# Arguments : NONE
# STDIN : value to test
# STDOUT : NOTHING
# Return Codes:
# 0 - the value is an integer
# 1 - the value is not an integer
is_int () {
cat | egrep -q '^[-]?\d+$' && return 0
return 1
}
# check some values
testVals=(42 4.5 -1 waffles)
for val in "${testVals[@]}"
do
if echo "$val" | is_int
then
echo "'$val' is an integer"
else
echo "'$val' is NOT an integer"
fi
done
As is often the case with Bash code, there is a lot happening in that two-line function. The first line uses cat to redirect the function’s STDIN to the egrep command which checks it against the pattern for an integer (an optional leading - followed by one or more digits). The pattern matches, egrep will return a successful exit code, so the command on the other side of the && (lazy and operator) will also get executed, and the function will exit with the successful exit code 0, otherwise, the function will keep executing, causing it to exit with the exit code 1 instead.
Function Arguments
In the above example, taking the text value from STDIN is cumbersome. The code would be much cleaner if the function accepted the test value as an argument instead.
Similar to how functions get their own streams, functions get their own argument variables, i.e they get their own $#, $@, $1, $2, etc.
So, we can improve our is_int function so it only reads STDIN if it’s passed no arguments, otherwise it loops over all arguments and exits with success if all arguments are integers. You’ll find this function in pbs153d-fnArgs.sh in the instalment ZIP:
#!/usr/bin/env bash
# define a POSIX function to test if all passed values are integers
# Arguments : 1...n, the values to test
# STDIN : value to test (only read if no args passed)
# STDOUT : NOTHING
# Return Codes:
# 0 - the value is an integer
# 1 - the value is not an integer
is_int () {
# save the RE used to recognise integers
intRE='^[-]?\d+$'
# check for args and process them or STDIN as appropriate
if [[ $# -gt 0 ]]
then
# loop over args
for val in "$@"
do
# return an error if not a match
echo "$val" | egrep -q "$intRE" || return 1
done
# if we got here, all were matches, so return success
return 0
else
# process STDIN
cat | egrep -q "$intRE" && return 0
return 1
fi
}
# check values via STDIN
testVals=(42 4.5 -1 waffles)
for val in "${testVals[@]}"
do
echo -n "Testing '$val' via STDIN: "
if echo "$val" | is_int
then
echo 'YES'
else
echo 'NO'
fi
done
# check single values via args
for val in "${testVals[@]}"
do
echo -n "Testing '$val' via Single Arg: "
if is_int "$val"
then
echo 'YES'
else
echo 'NO'
fi
done
# many values via args
echo -n "Testing '42' & '-1' via Args: "
if is_int 42 -1
then
echo 'ALL integers'
else
echo 'NOT all integers'
fi
echo -n "Testing '11' & 'waffles' via Args: "
if is_int 11 waffles
then
echo 'ALL integers'
else
echo 'NOT all integers'
fi
Firstly, notice that we can now use the function in three ways:
# via STDIN as before
if echo "$val" | is_int
# …
# via a single arg
if is_int "$val"
# …
# via multiple args
if is_int 42 -1
# …
To achieve this we used exactly the same code we would do if we were writing a script to achieve the same goals. We used the $# (number of arguments) variable to decide whether to read from STDIN, or whether to process args:
if [[ $# -gt 0 ]]
then
# loop over args
# …
else
# process STDIN
# …
fi
The code in the else block is unchanged from the previous example, so the new code of note is in the then block:
for val in "$@"
do
echo "$val" | egrep -q "$intRE" || return 1
done
return 0
What you’ll notice is that the test looks very similar to the one used for processing values from STDIN, but the logic has been reversed by using an || (or) instead of an && (and), and returning an error instead of success. This means that the loop only stops if a value is not an integer, at which point the function immediately returns, and then if the entire loop finishes without returning, it returns success.
Variable Scope
So far, this all looks quite simple, but I have been picking and choosing my examples very carefully to avoid the kind of spooky action at a distance Bash’s dynamic variable scope will produce if you don’t accommodate it in your code.
Beware Spooky Action at a Distance!
To illustrate how things break, let’s update our is_int function so it accepts an optional -a flag to indicate that it should return success if any of the passed values are integers. Since getopts processes the standard argument variables, and since functions get their own argument variables, you would expect getopts to work just fine within functions, and a first glance, it does. We can prove this with pbs153e-fnOptArgsBAD.sh in the instalment ZIP. This script defines the updated version of our is_int function described above and then calls it a few times with and without the -a flag:
#!/usr/bin/env bash
# define a POSIX function to test if all passed values are integers
# Arguments : 1...n, the values to test
# Flags :
# STDIN : value to test (only read of no args passed)
# STDOUT : NOTHING
# Flags:
# -a Return success if any tested value is an integer
# Return Codes:
# 0 - the value is an integer
# 1 - the value is not an integer
is_int () {
# save the RE used to recognise integers
intRE='^[-]?\d+$'
# process the optional arguments
anyOK=''
while getopts ':a' opt
do
case $opt in
a)
# save the fact that any int is OK
anyOK=1
;;
?)
# render a sane error, then return
echo 'Function Usage: is_int [-a] [VALUES ...]'
return 1
;;
esac
done
# remove the optional args from the argument list
shift $(( OPTIND - 1 ))
# check for remaining args and process them or STDIN as appropriate
if [[ $# -gt 0 ]]
then
if [[ -n $anyOK ]]
then
# any matching int is success
for val in "$@"
do
echo "$val" | egrep -q "$intRE" && return 0
done
return 1
else
# all must match
for val in "$@"
do
echo "$val" | egrep -q "$intRE" || return 1
done
return 0
fi
else
# process STDIN
cat | egrep -q "$intRE" && return 0
return 1
fi
}
# print test array
testVals=(42 -1 4.5 waffles)
echo "Test Values:"
for val in "${testVals[@]}"
do
echo "- $val"
done
echo ''
# trigger spooky action at a distance!
echo -n "Result of first call WITHOUT -a (should be negative): "
if is_int "${testVals[@]}"; then echo "all ints"; else echo "NOT all ints"; fi
echo -n "Result of first call WITH -a (should be positve): "
if is_int -a "${testVals[@]}"; then echo "at least one int"; else echo "NO ints"; fi
echo -n "Result of second call WITHOUT -a (should be negative): "
if is_int "${testVals[@]}"; then echo "all ints"; else echo "NOT all ints"; fi
The function itself still contains the same logic for dealing with STDIN or arguments, but the code for handling optional arguments has been added above the if statement to detect whether or not there are args, a variable has been added to track whether or not we are OK with there being at least one integer, and an extra if statement has been added to add the logic for considering at least one integer to be success.
Starting with the optional arguments:
# process the optional arguments
anyOK=''
while getopts ':a' opt
do
case $opt in
a)
# save the fact that any int is OK
anyOK=1
;;
?)
# render a sane error, then return
echo 'Function Usage: is_int [-a] [VALUES ...]'
return 1
;;
esac
done
The only difference to our previous uses of getopts is the use of return rather than exit when dealing with invalid options. The $anyOK variable is being used to track whether or not we need every value to be an integer, or just one.
Finally, let’s look at the logic for dealing with any -v- all:
if [[ -n $anyOK ]]
then
# any matching int is success
for val in "$@"
do
echo "$val" | egrep -q "$intRE" && return 0
done
return 1
else
# all must match
for val in "$@"
do
echo "$val" | egrep -q "$intRE" || return 1
done
return 0
fi
The code for handling the case where all values must be integers is unchanged from the previous version, and the code for handling the case when any integer is OK has yet again flipped the logic — as soon as a single integer is matched, success is returned. If the loop finishes without returning success, then none matched, so return an error.
You might imagine this function would work, and at first glance, it will. If you call it just once all will be well, but if you call it multiple times, weirdness will ensue.
To that end, the sample file calls the function three times on the same set of sample data which contains a mix of values that are and are not integers (42, -1, 4.5 & waffles):
echo -n "Result of first call WITHOUT -a (should be negative): "
if is_int "${testVals[@]}"; then echo "all ints"; else echo "NOT all ints"; fi
echo -n "Result of first call WITH -a (should be positve): "
if is_int -a "${testVals[@]}"; then echo "at least one int"; else echo "NO ints"; fi
echo -n "Result of second call WITHOUT -a (should be negative): "
if is_int "${testVals[@]}"; then echo "all ints"; else echo "NOT all ints"; fi
We would expect the output of the script to be:
Result of first call WITHOUT -a (should be negative): NOT all ints
Result of first call WITH -a (should be positve): at least one int
Result of second call WITHOUT -a (should be negative): NOT all ints
But what we get is a little different:
Test Values:
- 42
- -1
- -4.5
- waffles
Result of first call WITHOUT -a (should be negative): NOT all ints
Result of first call WITH -a (should be positve): at least one int
Result of second call WITHOUT -a (should be negative): Function Usage: is_int [-a] [VALUES ...]
NOT all ints
Where did that error come from?
The problem is that all bash variables are global by default. This means that all the variables we use inside the function retain their values the next time the function executes, and that includes the variables getopts uses to do its work, i.e. $OPTIND and $OPTARG. So, when optarg tries to process the args, it’s not starting from zero each time, it’s starting from the last location it found an optional argument previously! That means the first option it sees on the third pass is -1, which is not a valid option!
Dynamic Scope in Bash
As we’ve just demonstrated, by default, all variables in Bash are global. That means that by default, if a function edits the value of a variable, that value is changed everywhere, even in the main body of the script, in other functions, and in any commands called anywhere in the script. This is why editing the value of the input field separator variable ($IFS) causes so much spooky action at a distance, and why I have been so strongly advising against it!
I keep saying that by default all variables are global, so that implies we can change this default, which is where the local keyword comes in.
Using local to Manage Scope
Inside a function, you can use the local keyword to mark one or more variables as being in the function’s scope. Because Bash uses dynamic scope rather than lexical scope, that does not mean the variable only exists within the function. A local variable exists within the function, and within all commands and other functions called within the function. Basically, they are global to a sub-set of the universe! If the same variable is declared local again in a command or function called directly or indirectly by your function, another nested local scope will be created. You can think of each local scope as being global within their own little universes, each just a part of a bigger universe. Another analogy you’ll often see is shadows — the local keyword blocks out the variable’s global value to the function and all the code it calls like the function casts a shadow over the code it executes. You can then have shadows within shadows.
The bottom line is that you must make a deliberate decision about the scope you want every variable you use in a function to have, and, you should get into the habit of starting every function with a call to local listing all the variables you want to localise within your function.
Note that the argument variables ($#, $@, $1 etc.) are localised by design, so you don’t have to localise those yourself.
So, let’s update our is_int one last time, and localise all the variables we need to make local by adding the following line to the top of the function:
# localise the appropriate variables
local intRE anyOK opt OPTIND OPTARG val
Note that like when assigning a value to a variable, or telling a Bash builtin like for what variable name to use, we do not pre-fix the variable names with $ symbols when calling local.
You’ll find a fully working example in pbs153f-fnOpArgs.sh, and when you run it you now get the expected output:
Test Values:
- 42
- -1
- -4.5
- waffles
Result of first call WITHOUT -a (should be negative): NOT all ints
Result of first call WITH -a (should be positve): at least one int
Result of second call WITHOUT -a (should be negative): NOT all ints
But literally the only change in the code from the previous example is the addition of the call to local, so I won’t include the full code in the article.
Reusing Functions with source
If you write functions that are very task-specific, it makes sense to just save them in your actual script, but you may find yourself building up a library of functions you use regularly. Could you save those in an external file and import them as needed? Yes you can — you can use the source command to import all variables and functions from another script into your script. Simply give the path to the file as the first argument.
As a final example, you’ll find the final copy of our is_int function in the file utils.sh in the instalment ZIP, and you’ll find one final example script (pbs153g-source.sh) that imports that file, and hence the function using source:
source "$(dirname "$0")/utils.sh"
Note the use of the dirname command in conjunction with the path to the executing script ($0) to make the path relative to the script’s location rather than the user’s present working directory. Unless you’re using absolute paths, maybe to a special folder in your home dir where you keep your reusable functions, you definitely want to do this in real-world scripts.
Final Thoughts
Now that we understand the power of functions and source, we can start writing really reusable Bash code. And, now that we understand variable scope, we have the power to avoid spooky action at a distance and we can use that new-found power to use special variables like IFS safely.
We’ve now come to the end of our exploration of Bash. Well, almost. There will be one final instalment to pull it all together, to put some structure on things, and, to act as a kind of quick-reference guide to help you in your future Bash-scripting endeavors. Because Bash is such a dense language, it’s unlikely anyone but a full-time sysadmin will ever become familiar enough with all the nuance not to need a cheat sheet of some kind!
An Optional Challenge
Finally, if you’d like to put your new-found function writing skills to the test, update your multiplication table script to replace duplicated code blocks with functions.