PBS 125 of X: on Multiple Computers (Chezmoi)

We’ve finally arrived at the finale of our exploration of Chezmoi — the point where we expand our horizon to start syncing our dot files across multiple computers, but in such a way that we can handle the inevitable small differences between computers elegantly. While we might well want 90% or more of our settings to be the same on all computers, that 10% difference can be oh so important, and it’s how Chezmoi deals with that 10% that made me fall in love with it. 🙂

Matching Podcast Episode

Listen along to this instalment on episode 599 of the Chit Chat Across the Pond Podcast.

You can also Download the MP3

Chezmoi Git Recap

Chezmoi’s cross-computer syncing is entirely built upon its use of Git as its underlying content management system. In instalment 123 we learned how to create an empty remote Git repo and push our dot files out to it. That remote repo is the key to syncing across computers.

As we learned in instalment 122, we can choose to configure Chezmoi to automatically commit our changes to Git and push them. This is obviously convenient, but it means you lose control over when the commits happen, and the commit messages are all auto-generated. I don’t find that a good experience, so I manage my Git repository myself, choosing to only commit and push when changes have been tested and debugged, not on each edit.

You have four choices for interacting with the Chezmoi Git repo:

1. Use a GUI client (my preferred approach on desktop computers).
2. Use the chezmoi git -- command (my preferred approach on headless computers/VMs).
3. Use the chezmoi cd command to start a new shell session inside the Chezmoi source directory and use the git command from there.
4. Change into the Chezmoi source directory in your current shell session with cd $(chezmoi source-path) and use the git command from there.

Some Additional Useful Templating Functions

Before we go any further, there are a few additional text/template and sprig functions it would be useful to know about.

Quoting Strings

When using templates to add contents to a data file, it’s important that the appropriate special characters get escaped. If you don’t properly escape your data, you could end up corrupting your dot files, which is obviously not what we want to do. 🙂

These functions are all available for use in pipelines. The following are the most relevant to us:

1. quote — wrap the input in quotation marks and escape any quotation marks within the string.
2. squote — same as quote but uses single quotes, also known as apostrophes.
3. js — convert the passed value to a valid JavaScript variable definition.
4. toJson — convert the passed value to a valid JSON string.
5. toPrettyJason — convert the passed value to nicely formatted JSON string.

A Ternary Operator of Sorts (a Function actually)

We’ve already seen two sprig functions that are very useful when defining values in a template — we’ve seen how the default function can be used to define a single fall-back value, and how the coalesce function searches for the best value from multiple sources. Well, sprig provides a third related function — ternary. This is not quite the same as the ternary operator in JavaScript, but it is similar.

Sprig’s ternary function expects three arguments, the last of which is a boolean value. The first argument is the value to return if the last argument is true, and the second the value to return if the last argument is false. The ordering might seem a little strange at first, but remember, with pipelining the output of the previous function is passed as the last argument to the next function, so this arrangement allows the output of some kind of test to be piped to the ternary function.

For example, the following reads the SHELL environment variable, checks it’s Zsh, and then outputs an appropriate value:

When it comes to shells, I'm {{ env "SHELL" | eq "/bin/zsh" | ternary "hip" "a laggard" }}!

(Seriously, running Zsh is hip, I promise 😉)

Chezmoi’s Sync Model

The basic model is very simple — initialise Chezmoi on your first computer, add some dot files, commit them locally, create an empty remote repo, push your dot files to it, then on all your other computers, initialise Chezmoi by cloning the remote repo.

Once all the computers are initialised, changes are managed by pushing and pulling to and from the remote repo, and running chezmoi apply after each pull to push the synced changes live.

Initialising Chezmoi from a Remote Git Repo

When working on a single computer we initialised Chezmoi with the simple command chezmoi init. Initialising from a remote Git repo is almost as simple — simply pass the repo’s URL as an argument!

However, don’t do it just yet, we can ensure we have a better experience by doing a little more preparation first.

Handling Differences Across Machines

There is an entire section of Chezmoi’s How To page dedicated to the many different options open to you for dealing with differences across computers. As with everything else in this series, we’re not going to look at every possibility here, we’re simply going to look at a few of the more commonly used options. If you need something more complex, check out the documentation 🙂

Define a Chezmoi Config Template

One of Chezmoi’s most powerful features is its ability to generate its own config file from a template. If a file exists at the root of the repo named .chezmoi.json.tmpl, then when you do a chezmoi init, that template will be used to build the config.

This gives you two very important capabilities — firstly, and most obviously, it allows you to have Chezmoi behave differently on different computers, but secondly, because you can define data in the config file, it allows you to customise data fields that change from computer to computer.

To add just a little more power, Chezmoi provides special templating functions that only work when processing a config template which allow you to prompt the user for input, specifically; promptString, promptInt, and promptBool. All three functions require a single-word variable name as their only argument.

Initialising a Config Template

If you already have a Chezmoi config, then you can use that as the basis for your template by copying it to the appropriate location with:

cp ~/.config/chezmoi/chezmoi.json $(chezmoi source-path)/.chezmoi.json.tmpl

Note that $() is way to use the output of one command as an argument to another in Bash or Zsh.

If you don’t already have a config simply create a new file using your favourite text editor and save it in the root of your Chezmoi source directory as .chezmoi.json.tmpl.

Testing Your Config Template

It’s easy to make mistakes when writing a template, so you can test your config template with the following useful command:

chezmoi execute-template < $(chezmoi source-path)/.chezmoi.json.tmpl

If you make use of any of the prompt functions you need to pass values for each of the named variables using the --init flag followed by the --promptString, --promptInt, and/or --promptBool flags. Each of these flags takes name-value pairs as values, e.g.:

chezmoi execute-template --init --promptString email=something@somewhe.re < $(chezmoi source-path)/.chezmoi.json.tmpl

The --init flag is important, it tells the execute-template sub-command to make the special init-only functions available.

Rebuilding Your Config File from the Template

Assuming your repo has a config template, then it will be used when Chezmoi is first initialised on a computer. But, you can rebuild it any time by simply running chezmoi init again (without any arguments).

Use the .chezmoiignore File

The most coarse-grained difference you can have from computer to computer is that some dot files may only be needed on some of your computers. By default, when you run chezmoi apply all of the targets in your source get outputted to your destination.

If you don’t use Git on a computer, do you really want your Git config and Git ignore files in your home dir? Sometimes having an unneeded dot file is just clutter, but some files have an effect on an app or service’s behaviour by their mere existence, so there may be situations where it’s not just nice not to skip some targets, but where it’s necessary!

Thankfully Chezmoi has a mechanism for filtering targets as the source gets applied to the destination — .chezmoiignore files!

Simply create a file named .chezmoiignore in the root of your Chezmoi source directory.

This file is similar to a .gitignore file, but rather than tell Git what not to track, it tells Chezmoi which targets to skip when applying the source state to the destination. The syntax for listing targets is basically the same as that for the Git ignore files, though strictly speaking it’s the syntax understood by the Match function in the doublestar Go package (documented here).

There are two very important things to bear in mind though:

1. The paths in Chezmoi ignore files are interpreted as paths in the destination folder.
2. Chezmoi ignore files are processed as templates

This last point is vital — this allows us to ignore different files on different computers. As an example, the following ignores the appropriate shell configuration files when you’re not running Bash or Zsh:

{{- if ( env "SHELL" | ne "/bin/bash" ) }}
# not running Bash, so ignore Bash profile
.bash_profile
{{- else }}
# using Bash, so not ignoring Bash profile
{{- end }}

{{- if ( env "SHELL" | ne "/bin/zsh" ) }}
# not running Zsh, so ignore Zsh environment
.zshenv
{{- else }}
# using Zsh, so not ignoring Zsh environment
{{- end }}

Note that I’m using the env templating function to access the SHELL environment variable to figure out my shell. Also note the use of the ne comparison function for non-equality.

You can see what your ignore file evaluates to with the chezmoi execute-template command:

chezmoi execute-template < $(chezmoi source-path)/.chezmoiignore

For the above example I get the following output (my SHELL environment variable has the value /bin/zsh):

# not running Bash, so ignore Bash profile
.bash_profile
# using Zsh, so not ignoring Zsh environment

Reminder — you can see the value of an environment variable by prefixing its name with a $ symbol and passing it as an argument to the echo command, e.g.:

bart-imac2018:~ bart% echo $SHELL
/bin/zsh
bart-imac2018:~ bart% 

Note that you can have multiple .chezmoiignore files, by adding them into subdirectories in your source state. When you add multiple files, the scope of the files is limited to the folder the file is in, and its sub-folders. Unless you have a very large repository, I would advise against this approach, it can result in what seems to you like spooky action at a distance as some of your targets unexpectedly don’t appear in your destination.

Some Tips

While ignoring entire files on some machines is great, the vast majority of your computer-to-computer differences will be managed using templates. That’s why we’ve been focusing on them so heavily over the past few instalments.

Based on my use of Chezmoi I have a few suggestions for how to make your templates work better for you.

1. Minimise your use of the prompt functions in your config template — you don’t want to be overwhelmed each time you update the template and have to re-init!
2. Distinguish between specific computers by hostname
3. Use the config template to define useful boolean data fields, this will make the if actions in your templates much easier to read

A Real-world Sample

Because everyone’s computer is so different, a realistic example didn’t seem practical, so instead, I’m going to share parts of my actual real-world Chezmoi setup.

Firstly, this is my Chezmoi config template:

{
  {{- /* prompt for an editor choice */}}
  {{- $editorName := promptString "editorName" }}
  "edit": {
  	{{- if eq $editorName "smultron" }}
    "command": "/usr/bin/open",
    "args": ["-a", "/Applications/Smultron.app", "-F", "-n", "-W"]
    {{- else if eq $editorName "textedit" }}
    "command": "/usr/bin/open",
    "args": ["-t", "-F", "-n", "-W"]
    {{- else }}
    "command": "/usr/bin/vi"
    {{- end }}
  },
  "git": {
    "autoAdd": true
  },
  "data": {
  	{{- /* define useful booleans */}}
    "is_mac": {{ eq .chezmoi.os "darwin" | js }},
    "needs_git": {{ any (eq .chezmoi.hostname "bart-imac2018") (eq .chezmoi.hostname "cc-dsk-2ss") | js }},
    
    {{- /* define personal info */}}
    "name": "Bart Busschots",
  	{{- if eq .chezmoi.hostname "cc-dsk-2ss" }}
    "organisation": "Maynooth University",
    "email": "bart.busschots@mu.ie"
  	{{- else }}
    "organisation": "Bartificer Creations",
    "email": "opensource@bartificer.net"
  	{{- end }}
  }
}

Notice my use of the promptString command to allow me easily specify my preferred editor when I initialise Chezmoi. Also notice that in the data section I’m using the hostname to determine whether or not I’m on my work laptop, and, whether or not I need my Git config.

I can test this config template like so:

bart-imac2018:~ bart% chezmoi execute-template --init --promptString editorName=smultron < $(chezmoi source-path)/.chezmoi.json.tmpl
{
  "edit": {
    "command": "/usr/bin/open",
    "args": ["-a", "/Applications/Smultron.app", "-F", "-n", "-W"]
  },
  "git": {
    "autoAdd": true
  },
  "data": {
    "is_mac": true,
    "needs_git": true,
    "name": "Bart Busschots",
    "organisation": "Bartificer Creations",
    "email": "opensource@bartificer.net"
  }
}
bart-imac2018:~ bart%

Notice the use of the --promptString flag to pass a value for editorName. Also notice that because I used templating comments, the output is perfect JSON, but I still have comments when I need to make edits.

Next, let’s look at how I use the .needs_git boolean in my .chezmoiignore file to only include my Git configs when needed:

{{- if eq .needs_git false }}
# ignore Git dot files on this computer
.gitconfig
.gitignore_global
{{- else }}
# Git files not ignored on this computer
{{- end }}

On my Mac this results of testing the ignore file:

bart-imac2018:~ bart% chezmoi execute-template < $(chezmoi source-path)/.chezmoiignore

# Git files not ignored on this computer
bart-imac2018:~ bart% 

Next let’s look at my Git config file:

[user]
	email = {{ .email }}
	name = {{ .name }}
[core]
	excludesfile = /Users/bart/.gitignore_global
[push]
	followTags = true
[fetch]
	prune = true

Note the use of two template variables.

This renders like so on my Mac:

bart-imac2018:~ bart% chezmoi cat .gitconfig 
[user]
	email = opensource@bartificer.net
	name = Bart Busschots
[core]
	excludesfile = /Users/bart/.gitignore_global
[push]
	followTags = true
[fetch]
	prune = true
bart-imac2018:~ bart%

Finally, let’s look at how my SSH config (~/.ssh/config) file makes use of the is_mac boolean to only configure KeyChain integration on Macs:

{{- /* only add Keychain support on Macs */}}
{{- if .is_mac }}
# enable integration between Keychain and SSH Agent
UseKeychain yes
AddKeysToAgent yes
{{- end }}

# force services VM onto non-standard port
Host services.so-4pt.net
	Port 2222

This renders like so on my Mac:

bart-imac2018:~ bart% chezmoi cat .ssh/config

# enable integration between Keychain and SSH Agent
UseKeychain yes
AddKeysToAgent yes

# force services VM onto non-standard port
Host services.so-4pt.net
	Port 2222
bart-imac2018:~ bart%

You’re Ready to Rock and Roll

Once you have your config template, your Chezmoi ignore file and all your templates in place you’re ready to start using Chezmoi on other computers. Make sure you’ve committed and pushed all your changes to your remote repo, then simply run (replacing URL with the clone URL of your repo, if you use GitHub, use the Code button to get the SSH URL):

chezmoi init URL

Final Thoughts

This is where we’ll draw a line under our exploration of Chezmoi. There should be enough here for most people to effectively manage their dot files, either with versioning and backup on just one computer, or across many computers with good support for computer-to-computer differences. There are of course more advanced usages available, and if you run into a problem you can’t solve with just what we’ve covered here, I highly recommend having a look at Chezmoi’s How To page for inspiration.

We’ve now come to the end of the first phase of this series. We’ve covered the three main technologies that power front-end web development in great detail, that is to say we’ve learned how to use HTML to define the contents of a page, CSS to define its appearance, and JavaScript to bring it to life. We’ve also made a start on building out our developers toolbox, learning to use open source libraries and frameworks like jQuery, Bootstrap, and Momment.js to speed up development and avoid reinventing the wheel. Because of our use of third-party libraries I hope you’ve also become comfortable reading documentation, that’s an absolutely vital skill! Finally, we’ve made friends with one of the most important tools in any developer’s toolbox, version control, in our case, with Git.

Phase 2 will be quite different to phase 1. We’ll be using a real-world project to drive our learning, and in the process do the following:

1. Expand our developers toolkit further to include code linters and documentation generators.
2. Apply our understanding of programming to a new language, specifically PHP.
3. Learn about web servers, and how they interact with web clients via the Common Gateway Interface, or CGI.
4. Put our understanding of Git into practice to collaborate on an open project.