PBS 122 of X: Managing Dot Files (Chezmoi)

In the previous instalment we described what so-called dot files are, why we might like to manage them, and why I like using Chezmoi to do that management.

In this instalment we’ll learn the Chezmoi basics — how to add a file for management, edit it, manage our changes in Git, configure Chezmoi, and finally, remove a file from management. That covers the most basic but common Chezmoi use-case: a single computer and no templates. Many Chezmoi users never take things any further than this.

Matching Podcast Episode

Listen along to this instalment on episode 695 of the Chit Chat Across the Pond Podcast.

You can also Download the MP3

Reminder — Chezmoi Terminology

Chezmoi manages targets (files, folders and symlinks), saving their desired definitions in the source state and converting them to a destination state which gets applied to a destination directory (usually your home directory).

The source state is stored in a directory (~/.local/share/chezmoi by default) which is referred to as the source directory.

Note that when using the chezmoi command, targets are always specified by their path in the destination directory, e.g. ~/.gitconfig, not by their path in the source directory.

We need to understand these terms to be able to understand the Chezmoi Reference Manual, but until we learn about more advanced features like templating, we can think of Chezmoi as simply a tool for syncing canonical versions of our dot files from a Git repo at ~/.local/share/chezmoi to our home directory.

Finally, remember that Chezmoi only alters files it has been explicitly instructed to manage. All other files in the destination directory (your home directory) are considered unmanaged.

Managing Files (Targets)

If you have existing dot files that you’d like Chezmoi to start managing, simply add them to your source state with the chezmoi add command. As an example, let’s add our Git config and global ignore files:

chezmoi add ~/.gitconfig ~/.gitignore_global

We can now see that Chezmoi is managing these two files:

bart-imac2018:~ bart% chezmoi managed
.gitconfig
.gitignore_global
bart-imac2018:~ bart%

Peeking Under the Hood — the Chezmoi Source State

As we’re already reminded ourselves, Chezmoi stores its source state in the source directory, and there’s a bi-directional re-naming rule applied to all managed targets. We can use the chezmoi source-path command to see the path within the source directory for any managed target.

For example, we can see the path to the file in the source state that stores the canonical version of the target ~/.gitconfig with:

bart-imac2018:~ bart% chezmoi source-path ~/.gitconfig 
/Users/bart/.local/share/chezmoi/dot_gitconfig
bart-imac2018:~ bart% 

We can also use this command to reveal the path to the source folder itself by running it without any arguments:

bart-imac2018:~ bart% chezmoi source-path             
/Users/bart/.local/share/chezmoi
bart-imac2018:~ bart% 

We can use the Bash/Zsh shell’s back-tick operator to use the output of the above command as the argument to the cd command to actually change into the Chezmoi source directory (the pwd command shows the current path):

bart-imac2018:~ bart% pwd
/Users/bart
bart-imac2018:~ bart% cd `chezmoi source-path`
bart-imac2018:chezmoi bart% pwd
/Users/bart/.local/share/chezmoi
bart-imac2018:chezmoi bart%

Now that we’re in here we can have a proper look around with the ls command:

bart-imac2018:chezmoi bart% ls -la
total 16
drwxr-xr-x  5 bart  staff  160  5 Aug 23:59 ./
drwxr-xr-x  3 bart  staff   96 23 Jul 10:03 ../
drwxr-xr-x  9 bart  staff  288  4 Aug 23:50 .git/
-rw-r--r--  1 bart  staff  431  5 Aug 23:59 dot_gitconfig
-rw-r--r--  1 bart  staff   10  5 Aug 23:59 dot_gitignore_global
bart-imac2018:chezmoi bart% 

Notice that we have our two re-named managed files, and a hidden .git folder, because Chezmoi initialises a git repo for us.

We chose to change into the Chezmoi folder using the cd command in conjunction with the chezmoi source-path command. There is a more direct route to almost the same outcome, but with one subtle difference.

The chezmoi cd command opens a new shell and changes to the source directory inside that shell:

chezmoi cd

At first glance nothing has changed, but you are now in a shell within a shell. The most obvious difference is that you have lost your history, hitting the up arrow will not show chezmoi cd like you would expect. You can also prove you are in a new shell by leaving it with the exit command. If you exit in your terminal windows’s primary shell you’ll be told the process has completed and be left with an unusable window, but if you’re in a shell within a shell you’ll simply return to the outer shell.

You can also prove you are in a new shell by showing the current process ID before and after the chezmoi cd and exit commands. Bash and Zsh use the environment variable named $ to store the process ID. You access an environment variable by pre-fixing its name with the $ symbol, so you access the process ID with $$. The easiest way to see the value of an environment variable is with the echo command, so to see your current process ID you can use echo $$. The following sequence proves chezmoi cd starts an entire new shell rather than just changing the directory of the current shell:

bart-imac2018:~ bart% echo $$   
61420
bart-imac2018:~ bart% chezmoi cd
bart-imac2018:chezmoi bart% echo $$
66669
bart-imac2018:chezmoi bart% exit
bart-imac2018:~ bart% echo $$   
61420
bart-imac2018:~ bart%

I’m not really sure why Chezmoi’s developer chose to do it this way, but he did, so I think it’s important to explain why your terminal might behave differently to how you expect after using the chezmoi cd command.

Managing the Chezmoi Git Repo

While Chezmoi sets you up with a Git repository, it doesn’t take responsibility for managing it, that’s your job. You decide when to commit, what way to describe your commits, when to branch, and what remotes to configure, if any.

When it comes to managing your Chezmoi repo you have three choices:

1. Plain Git — you can change into the source directory either with the cd command or with chezmoi cd, and then you can use the git cli just like you would in any other repo.
2. Your Favourite Git GUI — if you have a Git GUI you like using, simply open the Chezmoi source directory in that GUI and work away. (FWIW, this is my preferred approach.)

3. The chezmoi git Command — for convenience, chezmoi can act as a proxy to the git command. The big advantage of this approach is that you can run the command from any folder, but that convenience does come at a price. Both the chezmoi and git commands accept command line flags, so how does chezmoi know which flags it should act on, and which it should pass on to Git? Simple — all flags before the special -- flag will be assumed to be for chezmoi, and all flags after will be passed to git. All arguments get passed to git, regardless of whether they are before or after the -- flag.

   So, to simply see the status of the Chezmoi Git repo the command is very straight forward: chezmoi git status. But, to commit all changes with a commit message the command gets a little more complicated: chezmoi git commit -- -am 'Some Git Commit Message'.

Editing a Managed File

Ideally, you should get into the habit of editing the source version of managed files rather than editing the target version, but, assuming you’re not using templates, Chezmoi makes it easy to push changes from the target back to the source by simply re-adding the file with chezmoi add.

So, if you want to start off building good habits that will stand as you start to use more advanced features, get into the habit of the following workflow:

1. Edit the file in the source state
2. Apply the changes
3. Commit the changes to Git

Editing a Target in the Source State

If you don’t want to look under the hood, the correct way to edit a target in the source state is with the chezmoi edit command. This will open the target’s matching file in the source directory for editing in an editor. By default Chezmoi first checks if there is a $VISUAL environment variable defined, and if there is, it uses its value as the editor, if not, it checks for a $EDITOR environment variable, and failing that, it uses vi. On my Mac Chezmoi defaulted to vi, which works fine for me, but confuses most regular humans 😉 You could try to learn vi (it’s an optional extra at the end of instalment 11 of Taming the Terminal), but you probably want to get an editor you like.

If you’re happy to use a terminal-based editor, but want something a little more human-friendly than vi you can use nano (or its proverbial daddy pico). To do that, simply export an environment variable named EDITOR with the appropriate value in your terminal before running chezmoi edit, e.g. export EDITOR=nano.

If you’re on a Mac you can also have Chezmoi open files with the built in TextEdit GUI editor by exporting an environment variable named VISUAL with the value 'open -e', i.e. export VISUAL='open -e'.

On the Mac you can use a similar approach to use any GUI editor you like by using the open command’s -a flag to specify the desired app, e.g. to use Smultron the command would be:

export VISUAL='open -a /Applications/Smultron.app'

Note that if you’re using Linux you should be able to easily install nano from your distribution’s standard package manager, and you can set the path directly to your visual editor of choice.

Note that the $EDITOR and $VISUAL environment variables are not Chezmoi-specific, they are used by other command line tools too, including Git.

Having to export your editor each time is a pain, but don’t worry, there is a better way — we can configure our shell to export the variable for us. Let’s do that as a worked example.

The dot file you should use to specify the environment variable export depends on your choice of shell. You can tell which shell you’re using from the value of the SHELL environment variable, e.g.:

bart-imac2018:~ bart% echo $SHELL
/bin/zsh
bart-imac2018:~ bart%

As you can see, I’m using Zsh, but if I were using Bash the value would be /bin/bash.

If you’re using Zsh you should export your environment variables in ~/.zshenv, and if you’re using Bash you should export them in ~/.bash_profile. I’m going to use Zsh in the example below, but if you’re using Bash, substitution ~/.bash_profile for ~/.zshenv in all the commands.

Note that Chezmoi can only manage files that exist, so we need to be sure there is a ~/.zshenv (or ~/.bash_profile) before we can add it, but there is a possibility you already have a copy of this file with important settings in it, so, we need to ensure it exists in a non-destructive way. The simplest approach is to append a comment to the end of the file with the echo command, e.g.:

echo '#boogers' >> ~/.zshenv

Once we’re sure the file exists, we need to tell Chezmoi to manage it with the chezmoi add command:

chezmoi add ~/.zshenv

We can now edit it, being sure to export our favourite editor first (I went with export VISUAL='open -e' to use TextEdit):

chezmoi edit ~/.zshenv

I replaced the silly boogers comment with the following and then saved the file:

export VISUAL='open -e'

Note if you use TextEdit, be sure to add an empty line at the end of the file or you’ll get very odd behaviour.

At this point we have a difference between our source state, and our destination state, we can see that with the chezmoi status command:

bart-imac2018:~ bart% chezmoi status
 M .zshenv
bart-imac2018:~ bart%

This shows I have one modified target, .zshenv. I can see the details of the change with the chezmoi diff command:

bart-imac2018:~ bart% chezmoi diff ~/.zshenv 
diff --git a/.zshenv b/.zshenv
index e9df22b36675f90ff80d7ecf030f4b6e5410243a..333c2f85295158b06b95af45cf4d723f1f667ae6 100644
--- a/.zshenv
+++ b/.zshenv
@@ -1 +1 @@
-# boogers
+export VISUAL='open -e'
bart-imac2018:~ bart%

Note that while Chezmoi used Git to calculate and display the diff, this is not showing the difference between the last commit and the current state of the target in the source directory, it’s showing the diff between the target in the source and destination directories.

The edit we have made is in Chezmoi’s source state, but it has not yet been applied in the destination directory. We can prove this to ourselves by outputting the current contents of the file in our home dir:

bart-imac2018:~ bart% cat ~/.zshenv 
# boogers
bart-imac2018:~ bart%

Applying the Updated Source State to the Destination Directory

We’re now ready to apply our updated source state to the destination directory. The command to do that is chezmoi apply:

bart-imac2018:~ bart% chezmoi apply                    
bart-imac2018:~ bart%

This doesn’t give any output if there are no errors, but our changes have been applied:

bart-imac2018:~ bart% cat ~/.zshenv                    
export VISUAL='open -e'
bart-imac2018:~ bart%

I started this section by noting that if you didn’t want to get under the hood then you should use the chezmoi edit command. If you don’t mind getting under the hood, simply open the Chezmoi source folder in your favourite IDE and edit any file you’d like that way. This is my preferred approach for doing any kind of significant work, but I do still use chezmoi edit for quick edits.

Committing the Changes

At this point we’ve added three files to Chezmoi, edited one, but committed nothing to Git.

At this stage we have a standard Git repo with three new files and no commits. Let’s use plain Git to get our changes committed in an orderly fashion:

bart-imac2018:~ bart% chezmoi cd    
bart-imac2018:chezmoi bart% git status
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	dot_gitconfig
	dot_gitignore_global
	dot_zshenv

nothing added to commit but untracked files present (use "git add" to track)
bart-imac2018:chezmoi bart% git add dot_git*
bart-imac2018:chezmoi bart% git commit -am "Feat: added my Git config"     
[master (root-commit) fb2783f] Feat: added my Git config
 2 files changed, 17 insertions(+)
 create mode 100644 dot_gitconfig
 create mode 100644 dot_gitignore_global
bart-imac2018:chezmoi bart% git add dot_zshenv 
bart-imac2018:chezmoi bart% git commit -am "Feat: added environment variable for Chezmoi editor"
[master 122420e] Feat: added environment variable for Chezmoi editor
 1 file changed, 1 insertion(+)
 create mode 100644 dot_zshenv
bart-imac2018:chezmoi bart% git log --pretty
commit 122420e7f4bbd1ef9eeab137db36abd522f8bf13 (HEAD -> master)
Author: Bart Busschots <opensource@bartificer.net>
Date:   Sat Aug 7 15:12:11 2021 +0100

    Feat: added environment variable for Chezmoi editor

commit fb2783f503c81dfcf281bcfa7274fed5bf59c20f
Author: Bart Busschots <opensource@bartificer.net>
Date:   Sat Aug 7 15:11:29 2021 +0100

    Feat: added my Git config
bart-imac2018:chezmoi bart% git log --oneline
122420e (HEAD -> master) Feat: added environment variable for Chezmoi editor
fb2783f Feat: added my Git config
bart-imac2018:chezmoi bart% exit
bart-imac2018:~ bart% 

Note that I used chezmoi cd to open a new shell in the source directory, then used standard Git commands to add the files as two separate commits with two separate messages, then listed my commits to show the repo’s history, then exited out of the new shell to return to where I was before I started.

While I consider it a feature rather than a bug that I need to take care of all the Git stuff myself, it would be nice to have some of it automated, and, some people may well like it all automated. This is a good opportunity to explore Chezmoi’s own config file — yes, there is a dot file to manage the dot file manager 🙂

Configuring Chezmoi

Because Chezmoi makes use of the open source library viper for managing its config, it doesn’t impose a single format, but supports all formats supported by viper. The default is TOML, which is quite human-friendly, and will feel familiar to anyone who remembers Windows .ini files in the days before the registry, but that’s not what we’ll be using. While I generally lean towards using defaults in this series, I much prefer JSON over TOML, and we already have experience with JSON in this series.

Once we have a config file in place, we can quickly edit it using the editor we have configured for chezmoi edit with the command chezmoi edit-config, but unfortunately this command can’t be used to create an initial empty config.

Regardless of the format you use, your Chezmoi config file will be in the folder ~/.config/chezmoi/ and will have the base filename chezmoi, the extension will determine the format, so we want to create a JSON file with no data in it a ~/.config/chezmoi/chezmoi.json. The easiest way to do that is by piping the JSON syntax for an empty object to that file name:

echo '{}' > ~/.config/chezmoi/chezmoi.json

We can now edit the file in our editor of choice with chezmoi edit-config.

The config section of the Chezmoi reference describes all the possible config values. When writing configs in JSON the entire config is contained within a single object. The top-level variables get added directly to this object, and each supported section is a nested object within the top-level object. So, were we to want to change Chezmoi’s default output data format from JSON to YAML we’d give the top-level key format the value yaml:

{
  "format": "yaml"
}

There are actually no top-level variables I suggest changing, so the above is just an illustration.

The section we’re interested in is git, and within that section there are two variables I suggest you consider setting to true — autoAdd or autoCommit.

Setting autoAdd to true will cause Chezmoi to automatically do a git add each time you do a chezmoi add. This will automatically stage all new files, making Git commits quicker and easier, but it won’t actually do any commit, allowing you to choose when to commit and what message to use.

Setting autoCommit to true implies autoAdd, but takes things to the next level by automatically committing each time you use the chezmoi command to make a chance to any target’s source state. Every change will get committed immediately, and the message will be automatically generated.

Personally, I prefer to use autoAdd, but I’ll let you decide what works best for you.

The edit section of the config allows you to specify the editor Chezmoi uses without the need to export environment variables. You use the command setting to specify the full path to the app as a string, and the args setting to specify any additional arguments as an array of strings. On the Mac, the following will set the editor to nano:

{
  "edit": {
    "command": "/usr/bin/nano"
  }
}

And the following to use TextEdit:

{
  "edit": {
    "command": "/usr/bin/open"
    "args": ["-e"]
  }
}

You can only use autoCommit if the command to open your editor waits to exit until you save your changes. This is the default behaviour for terminal editors like nano, but not for the Mac’s open command. The closest you can get is to combine the -W, -n, and -F flags which open a fresh new instance of the app and wait for that instance of the app close before open exits (closing the window is not enough, you have to close the instance of the app).

I’ve chosen to use Smultron as my editor and to auto add but not auto commit. Even though I’m not using auto commit, I set my editor to be compatible with auto commit, so this is my ~/.config/chezmoi/chezmoi.json:

{
  "edit": {
    "command": "/usr/bin/open",
    "args": ["-a", "/Applications/Smultron.app", "-F", "-n", "-W"]
  },
  "git": {
    "autoAdd": true
  }
}

I tested autoAdd by telling Chezmoi to start managing my SSH config file:

bart-imac2018:~ bart% chezmoi git status
On branch master
nothing to commit, working tree clean
bart-imac2018:~ bart% chezmoi add ~/.ssh/config 
bart-imac2018:~ bart% chezmoi git status       
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   dot_ssh/config

bart-imac2018:~ bart% chezmoi git commit -- -am 'Feat: added SSH config'
[master 2b4ce26] Feat: added SSH config
 1 file changed, 7 insertions(+)
 create mode 100644 dot_ssh/config
bart-imac2018:~ bart%

As you can see, before I managed the file Git had a clean working tree, after I added it to be managed Git had a staged new file ready for committing, and I was then able to commit the file without needing to run chezmoi git add dot_ssh/config.

Un-managing and/or Deleting Targets

Since we’ve now added our Chezmoi editor to our Chezmoi config, we don’t need to export any environment variables. Assuming you did not already have the variables set before we started, let’s remove our now useless ~/.zshenv (or ~/.bash_profile) file.

When it comes to removing a target we have two choices:

1. We can use chezmoi unmanage to remove the target from the source, but leave it in the destination.
2. We can use chezmoi remove (or its alias chezmoi rm) to remove the target from both.

Again, assuming you did not already have a ~/.zshenv or ~/.bash_profile, you can delete the file completely with:

bart-imac2018:~ bart% chezmoi rm ~/.zshenv 
Remove /Users/bart/.zshenv and /Users/bart/.local/share/chezmoi/dot_zshenv [yes,
no,all,quit]? yes
bart-imac2018:~ bart%

Notice I was asked for confirmation. I could have avoided that with the -f or --force flag.

If I had used autoCommit this change would have been automatically committed to Git, but because I only used autoAdd, the deletion was staged automatically, but not committed:

bart-imac2018:~ bart% chezmoi git status  
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	deleted:    dot_zshenv

bart-imac2018:~ bart% chezmoi git commit -- -am "Feat: removed environment variable export"
[master 4cef320] Feat: removed environment variable export
 1 file changed, 1 deletion(-)
 delete mode 100644 dot_zshenv
bart-imac2018:~ bart% 

Final Thoughts

We’ve now seen how to use Chezmoi to manage config files on one computer without applying templates. We have two more steps to take on our Chezmoi journey — using templates, and using Chezmoi across multiple computers.