PBS 121 of X: Managing Dot Files and an Introduction (Chezmoi)
From the first moment I introduced the concept of version control I emphasised the fact that it’s for managing all kinds of files, not just code. I’ve already mentioned that these very show notes are managed in Git, and some listeners/readers have already made use of that fact to contribute typo fixes via pull requests.
The deeper you get into the coding world the more likely you are to start using both the command line and developer tools of all sorts. If you stay in the open source community you’re likely to be using a POSIX shell like Bash or Zsh, and most of the open source developer tools have their origins in the Unix/Linux world, so they tend to do things the Unix-way, regardless of which OS you run them on, even on Windows! A big part of the Unix-way is plain-text human-readable configuration files. A very common design pattern is having multiple tiers of config file, with system-wide defaults overridden by user-level settings perhaps overridden at even smaller scales within a user’s account. The user-level configurations tend to be store in files or folders who’s names start with the dot or period character, so, they’re often known as dot files.
Managing these so-called dot files is likely to become ever more important the deeper you get down the programming rabbit-hole, so you’re likely to feel a desire for a tool to help you manage them sooner or later. Lots of such tools exist, and many of them are powered by Git. I use one such tool, and I’d like to introduce you to it, and more importantly, to the problems it might solve for you.
Matching Podcast Episode
Listen along to this instalment on episode 693 of the Chit Chat Across the Pond Podcast.
You can also Download the MP3
Getting to Know Your Dot Files
System-level config files tend to be stored in the /etc (Editable Text Configuration) folder and named for the app they control, but user-level config files are usually stored in a user’s home directory.
In POSIX operating systems (Unix/Linux/macOS), file names beginning with a . are hidden from file listings by default. This is true both on the command line with the ls command, and in the OS’s file explorer GUIs.
While it’s great to be able to view and edit your config files when you want to, you don’t really want them cluttering up your view of your home directory all the time. This is why the convention developed to name these config files or the folders that contain them with a leading . so as to hide them from view when you’re not explicitly looking for them. On the command line you can include hidden files in your file listings by adding the -a (for all) flag to the ls command.
As mentioned in the previous instalment, POSIX shells use the ~ character to represent a user’s home directory, so you’ll often see the paths to dot files written as something like ~/.gitconfig, which simply means “the file named .gitconfig in the current user’s home dir”.
What About on Windows?
Many open source tools retain the same basic design when ported to Windows, but, rather than using a simple ~ to represent the your home directory, you have to use a pair of environment variables together — %HOMEDRIVE% (usually C:) & %HOMEPATH% (usually \Users\ followed by your username).
For example, on Windows, your Git config file will be at %HOMEDRIVE%%HOMEPATH%\.gitconfig.
Note that as with the rest of this series, the examples for the remainder of this instalment will be for POSIX OSes, i.e. Unix, Linux, Mac, and the Linux Subsystem for Windows.
Exploring Your Dot Files
At this stage in the series you should have a ~/.gitconfig file. Every time you use the git config --global command you’re reading values from, or writing values to, this file.
You can see your user-level Git config with:
cat ~/.gitconfig
You may well see that some of the Git GUIs you use also add settings to this file. Here’s mine on the Mac I podcast from:
bart-imac2018:~ bart% cat ~/.gitconfig
[user]
email = opensource@bartificer.net
name = Bart Busschots
[core]
editor = nano
excludesfile = /Users/bart/.gitignore_global
[difftool "sourcetree"]
cmd = opendiff \"$LOCAL\" \"$REMOTE\"
path =
[mergetool "sourcetree"]
cmd = /Applications/Sourcetree.app/Contents/Resources/opendiff-w.sh \"$LOCAL\" \"$REMOTE\" -ancestor \"$BASE\" -merge \"$MERGED\"
trustExitCode = true
[push]
followTags = true
[fetch]
prune = true
bart-imac2018:~ bart%
There’s a good change you’ve got many more dot files though.
You can combine the -a for all and -1 for single-file-per-line flags for the ls command with the text-filtering command egrep (see TTT XXX for more) to list all files and folders in your home directory with names starting with a period:
bart-imac2018:~ bart% ls -1a ~ | egrep '^[.]'
./
../
.CFUserTextEncoding
.DS_Store
.Trash/
.bash_history
.bash_sessions/
.bundle/
.config/
.cpan/
.cpanm/
.cups/
.dropbox/
.gem/
.gitconfig
.gitflow_export
.gitignore_global
.gitkraken/
.hgignore_global
.lesshst
.local/
.node_repl_history
.npm/
.npmrc
.rbenv/
.ssh/
.subversion/
.viminfo
.vscode/
.wget-hsts
.zshrc
bart-imac2018:~ bart%
The Problems to be Solved
The more work you put into customising your shell, your SSH config, your Git config, and so on, the more valuable these dot files will become to you, and more you’ll yearn for a good tool for managing them. I see three major problems to be solved:
- Versioning — when I mess up, I would like to be able to go back in time to before I messed things up!
- Backup/Restore — if something happens my computer, I’d like to be able to get back to where I was quickly and easily. Also, if I buy a new computer or simply do a fresh install of an OS I’d like to get all my customisations back as quickly and easily as possible.
- Sync — I use many computers, when I put the effort into getting my Git config just so, I’d like the results of that work to follow me around from computer to computer!
I use the open source dot file manager Chezmoi to address all three of those problems. It’s by no means the only option, but it’s the one I have direct experience with, so it makes sense to share this one with you.
Introducing Chezmoi
Chezmoi is open source (hosted on GitHub), cross platform (Windows, Linux & macOS), and built on top of Git. It’s also focused and opinionated, which is probably why I’m so drawn to it 🙂 It’s entirely dedicated to managing user-level files, and explicitly not designed to manage system-level configs. Chezmoi is designed to live and work within your home directory, which means you don’t need admin-level permissions to install or use it.
With Chezmoi there is no secret sauce — the reference manual explains how it does everything it does, and in great detail to boot. But, you get to choose how much or how little you want to know. The Chezmoi command allows you to treat it as a black box, a crystal-clear box, or anything in between. Personally, I use Chezmoi in black-box mode for almost everything I do, with one exception — when it comes to managing the interactions between my local Chezmoi install and my GitHub-hosted repo for syncing between machines I pull the curtain right back and take direct control.
Chezmoi’s World View
The first thing to note is that Chezmoi will utterly ignore everything you don’t explicitly tell it to manage.
The second thing to note is that Chezmoi stores canonical copies of all the files it manages in the folder ~/.local/share/chezmoi — this is the folder you sync between computers with Git. In Chezmoi jargon this is your source directory.
Finally, Chezmoi applies its own well defined naming scheme to the files in the source directory. There is a one-to-one bi-directional mapping between the names, paths, and properties of the files being managed, and their canonical copies in the source directory. For example, the file ~/.zshenv will be stored in the source directory as dot_zshenv.
For the docs for the chezmoi command to make any sense you need to understand a few Chezmoi-specific terms, i.e., some Chezmoi jargon.
Source and Target
Chezmoi is built around a simple idea — you define a source state which it converts into a machine-specific target state that it then applies to a destination directory.
The source state is what gets stored in your source directory, and it consists of definitions for targets which are files, folders, or symlinks.
The destination directory is simply your home folder.
So source state in from your source directory, target state out to your destination directory.
Target Attributes
Each target in your source state has a number of attributes. These attributes tell Chezmoi how to process the target when translating it from its source state to its target state.
The most important attribute is whether the target should be treated as a literal file to be copied as-is, or processed as a template to generate the target state.
Installing Chezmoi
You can install Chezmoi from source if you really want to get your hands dirty, but you don’t have to, it’s available in all the common package managers. You’ll find a full list of all the installation options here, but on the Mac I like to use the Homebrew package manager:
brew install chezmoi
Getting Started With Chezmoi
Once you have the chezmoi command installed, regardless of where you got it from, you’re ready to get stared.
Before we look at specific chezmoi commands, let’s take a moment to look at the basic structure of all chezmoi commands — like git, chezmoi uses the command-sub-command design pattern so commands will take the form:
chezmoi [] []
Where [] are optional flags that are supported on all sub-commands like --verbose, `` is a required subcommand to execute, and that is then followed by optional flags and arguments for the subcommand.
If that all seems a little abstract, hopefully some practical examples will clarify.
The first thing you’ll need to do is initialise a source directory to hold your source state. We do this with the init subcommand:
bart-imac2018:~ bart% chezmoi init
Initialized empty Git repository in /Users/bart/.local/share/chezmoi/.git/
bart-imac2018:~ bart%
Notice that we did not pass any global flags, nor did we specify any arguments or flags for the subcommand.
We now have an entirely empty source state. Chezmoi is not managing any targets for us. We can prove this to ourselves with the managed subcommand which simply lists all the targets in the source state:
bart-imac2018:~ bart% chezmoi managed
bart-imac2018:~ bart%
We can also use the unmanaged subcommand to see all possible targets in our home dir that are not yet managed:
bart-imac2018:~ bart% chezmoi unmanaged
.CFUserTextEncoding
.DS_Store
.Trash
.bash_history
.bash_sessions
.bundle
.config
.cpan
.cpanm
.cups
.dropbox
.gem
.gitconfig
…
Pictures
Public
perl5
bart-imac2018:~ bart%
Note that I’ve truncated the output for brevity and privacy.
Final Thoughts
To a large extent this instalment has been a bit of a tease — we know why we want to manage our dot files, that Chezmoi is a good tool for doing that job, we’ve examined Chezmoi’s world view, and we’ve even initialised Chezmoi, but we haven’t actually managed any files yet!
That’s where we’ll start next time — using Chezmoi to manage dot files on a single computer. Next we’ll take things up a notch and add a remote Git repo and a second computer into the mix, and then we’ll finish up by looking at how we can use templates to generate different target states for each computer from the same source state.