PBS 132 of X: Managing Documentation (JSDoc)

We started our short journey into the world of documentation in instalment 130 by exploring the more philosophical side of documentation — why it’s a good idea in general, why we need it for our project, who our audiences are, and what documentation we need. I also explained why I think JSDoc is a good fit for us, and set the big-picture scene. We will use JSDoc comments to embed our API documentation into our source code, we’ll add some additional static content, and there will be two versions of our documentation, one for developers using the API, and one for contributors helping to write the API itself.

With that foundation laid, we spent the previous instalment learning about those JSDoc comments we’ll be embedding in our source code. Now, in this instalment, we’ll look at the practicalities of managing our documentation within the project, how we can add a lot of very useful automation, and we’ll look at how we can customise the look of the generated documentation with a JSDoc theme.

Matching Podcast Episode

Listen along to this instalment on episode 710 of the Chit Chat Across the Pond Podcast.

You can also Download the MP3

Episode Resources

• The instalment ZIP file — pbs132.zip.

Configuring JSDoc (with a Config File)

In the previous instalment we got a glimpse of how to use a JSDoc configuration file, but it deserves a deeper look. We’ve named our JSDoc config jsdoc.conf.json.

JSDoc’s configuration is ultimately a simple JavaScript object (dictionary), but there are two ways to define it — directly as as JSON file, or indirectly as a CommonJS (not ES6) JavaScript module that exports it. Writing code to generate the config is overkill for most people most of the time, and since we’ve not learned about CommonJS modules (opting for the standard ES6 format instead), we’ll be using a JSON file.

JSDoc’s config file support is asymmetric — anything you can do on the command line you can do via a config file, but there are things you can do via a config file that you can’t do on the command line.

This asymmetry is the reason we used a config file last week even though I’d have preferred not to. The config setting source.includePattern doesn’t have a command line equivalent, so we needed to use a config file to get JSDoc to process .mjs files at all, hence, for our purposes, the most basic possible config file is:

{
	"source": {
        "includePattern": ".+\\.(mjs|jsdoc)$"
    }
}

Specifying the Source

Up to this point we’ve been specifying the source files as arguments on the command line, but we can avoid that by specifying one or more folders to load from using source.include which should be an array of file paths. Let’s add the src folder:

{
	"source": {
		"include": ["./src/"],
        "includePattern": ".+\\.(mjs|jsdoc)$"
    }
}

By default JSDoc will recurse 10 levels deep into the included folders, but you can control it with the recurseDepth option, or disable it altogether by setting opts.recurse to false.

Replacing Command Line Flags (Readme & Encoding)

Speaking of the opts dictionary, that’s an import one — every command line flag can be controlled using it’s long name as a sub-key of opts, e.g. the -R flag’s long name is --readme, so we can include our README.md by setting opts.readme:

{
    "opts": {
        "readme": "README.md"
    },
    "source": {
        "include": ["./src/"],
        "includePattern": ".+\\.(mjs|jsdoc)$"
    }
}

What happens if the config file specifies one value for a flag, and the command line another? The command line wins, or, in the formal jargon — command line arguments take precedence over config file keys.

You might imagine we should set the destination folder in the config file too. The flag is --destination, so it could indeed be set with opts.destination, but there are two reason we don’t want to do this:

1. There is no one output dir for us, we will be generating two sets of documentation for our two audiences after all.
2. As part of our automation of the process we’ll need to empty this folder, so to avoid having to change the value in two places, we should specify it in the automation only.

However there are two more options we should set while we’re editing this section of the config. Firstly, it’s good practice to be specific about character encodings, so let’s use the --encoding flag to specify UTF8. Secondly, by default JSDoc quietly swallows a lot of errors and warnings which means we can end up with doc comments being silently dropped from the output. To avoid that we need to set the --pedantic flag:

{
    "opts": {
        "encoding": "utf8",
        "pedantic": true,
        "readme": "./README.md"
    },
    "source": {
        "include": ["./src/"],
        "includePattern": ".+\\.(mjs|jsdoc)$"
    }
}

You’ll find the full list of command line flags in the documentation.

Enabling Plugins

There are no command line flags for controlling plugins, so the only way to enable them is by adding them to the plugins array. The only plugin we need is the built-in Markdown one, so let’s add that:

{
    "opts": {
        "encoding": "utf8",
        "pedantic": true,
        "readme": "./README.md"
    },
    "source": {
        "include": ["./src/"],
        "includePattern": ".+\\.(mjs|jsdoc)$"
    }, 
    "plugins": [
        "plugins/markdown"
    ]
}

We’ll be using the Markdown plugin in its default configuration, but there are a few tweaks you can make, and they’re described on the Markdown page of the documentation.

Controlling Tags

By default JSDoc actually supports two sets of doc comment tags (the block and inline tags we met in the previous instalment like @param and {@link}). The JSDoc documentation refers to these sets of tags as tag dictionaries), and the two it supports are its own, and those for Google’s Closure Compiler. There’s a very significant overlap between the two, and by default, when ever there’s a conflict of meaning, JSDoc preferences its own meaning over Google’s. However, you can choose to allow only one or the other, or to reverse the order with the tags.dictionaries config setting. This should be an array of strings, each being the name of a tag dictionary, and the they should be listed in order of precedence, the highest precedence first. By default, tags.dictionaries has the value ["jsdoc", "closure"]. Because JSDoc supports plugins, it would also be possible for a third party to provide other tag dictionaries, but I’ve never seen any.

As well as specifying the tag dictionaries to use, the tags config setting also allows you to tell JSDoc how it should respond when it meets a tag it doesn’t recognise. By default it simply ignores it, but, if you set tags.allowUnknownTags to false, then JSDoc will print a warning when it meets an unknown tag. I always set JSDoc this way, because the unknown tags are almost always typos in my experience 🙂

We’ll be keeping the default tag dictionary order, but, we’ll be disallowing unknown tags:

{
    "opts": {
        "encoding": "utf8",
        "pedantic": true,
        "readme": "./README.md"
    },
    "source": {
        "include": ["./src/"],
        "includePattern": ".+\\.(mjs|jsdoc)$"
    }, 
    "plugins": [
        "plugins/markdown"
    ],
    "tags": {
        "allowUnknownTags": false,
        "dictionaries": ["jsdoc","closure"]
    }
}

Automating The Document Generation (NPM Scripts)

A JSDoc config file can simplify our jsdoc command significantly, but there’s still too much to remember, so we need to automate the process. We can use NPM’s scripts feature to do this for us.

NodeJS project files (package.json) support a key named scripts that’s a dictionary of terminal commands indexed by human-friendly names. You then run the named terminal commands with npm run.

The npm init command pre-populates scripts with a default placeholder command named test. We’ll add our commands above that.

Let’s start with a simple script to generate our docs:

"scripts": {
    "docs": "npx jsdoc -c ./jsdoc.conf.json --destination ./docs",
    "test": "echo \"Error: no test specified\" && exit 1"
  },

Once we save package.json, we could generate our docs with the command npm run docs.

However, this initial implementation is too naive. While JSDoc will create a folder if one does not already exist, it won’t empty one that does, so we need to take care of that. If we’re prepared to tolerate some error messages we could naively pre-fix our command with the rm command to delete the contents of our output folder:

"scripts": {
    "docs": "rm -rf ./docs/*; npx jsdoc -c ./jsdoc.conf.json --destination ./docs",
    "test": "echo \"Error: no test specified\" && exit 1"
  },

The first time we try to generate our documentation the folder won’t exist, so we’ll see an error from the rm command, but the docs will still generate. Many people would find this tolerable, but TBH, I don’t, so we can do better 🙂

The Posix test command has an alias named [[ which allows simple tests to be written as [[ CONDITION ]] && COMMAND where the command will only execute if the condition is met. The -e condition checks for existence, so [[ -e ./docs ]] && rm -rf ./docs/* will do nothing if there is no docs folder, and empty it if there is.

At this stage, the same string ./docs appears in three places in our command. How easy would it be to forget to update it in one of those three places if we were ever to want to change the name? Too easy!

The solution for that is a shell variable which I’ll named docDir. On a POSIX shell, a variable is defined with its name immediately followed by an = immediately followed by a value — no spaces! A variable can be included in a double-quoted string with its name pre-fixed with a dollar sign, so we can define docDir with the command docDir='./docs', and then use it in an rm command like so: rm -rf "$docDir/"*. Putting all that together with the test command we get the following raw Bash command:

docDir='./docs'; [[ -e "$docDir" ]] && rm -rf "$docDir/"*; npx jsdoc -c ./jsdoc.conf.json --destination "$docDir/"

Notice that the raw Bash command contains quotation marks, that means we need to escape those as \" when adding them into JSON. That command becomes the following snippet in package.json:

 "scripts": {
    "docs": "docDir='./docs'; [[ -e \"$docDir\" ]] && rm -rf \"$docDir/\"*; npx jsdoc -c ./jsdoc.conf.json --destination \"$docDir/\"",
    "test": "echo \"Error: no test specified\" && exit 1"
  },

Example — Automated JSDoc

In the instalment ZIP file you’ll find a folder named pbs132a, this is a NodeJS project folder containing our example code from last week, our updated jsdoc.conf.json, and a package.json file that defines our scripts.docs command.

Change into this folder in the terminal, then install the NodeJS dependencies with:

npm ci

Now generate the docs with:

npm run docs

Generating Different Documentation for Users and Contributors

As we laid out in instalment 130, we need two versions of our API documentation, one for users of the API we’re building, and one for those building the API. Both audiences are developers of course, but by convention I refer to the documentation for the API users as simply the docs, and the one for contributors to the API as the dev docs. For that reason we’ll be saving our regular documentation in ./docs, and our dev docs in ./docs-dev.

Marking Doc Comments as Private

The difference between the developer docs and the regular docs is that the developer docs include information on items not exported by the module. If we just add normal doc comments for these elements they’ll show up in both sets of documentation, but we can stop that happening by including the @private tag in any doc comment that shouldn’t be included in the regular docs. To that end I’ve added @private tags to the doc comments for the un-exported (or inner) variable and functions in ./src/Replicator.class.mjs. For example:

/**
 * Test if a given value is a valid replicator charge.
 * @param {*} val - The value to test.
 * @returns {boolean}
 * @see The {@link ChargeAmount} type definition.
 * @private
 */
function isCharge(val){
    return String(val).match(/^\d+$/) ? true : false;
}

If you open the documentation generated in the example and navigate to the page for the PBSReplicator module you’ll see the docs shows just the class, not the menu inner variable, or any of the inner functions likeisCharge().

To include private items in the output you need to use the --private flag (or set the config variable opts.private to true).

To generate our dev docs we need to add a new NPM script named docs-dev which does almost the same things as docs, but uses the target ./docs-dev, and specifies the --private flag:

"scripts": {
  "docs": "docDir='./docs'; [[ -e \"$docDir\" ]] && rm -rf \"$docDir/\"*; npx jsdoc -c ./jsdoc.conf.json --destination \"$docDir/\"",
  "docs-dev": "docDir='./docs-dev'; [[ -e \"$docDir\" ]] && rm -rf \"$docDir/\"*; npx jsdoc -c ./jsdoc.conf.json --destination \"$docDir/\" --private",
  "test": "echo \"Error: no test specified\" && exit 1"
}

We can now generate our developer docs with:

npm run docs-dev

This should create a new folder named docs-dev, within which you should find another entire documentation site. Again, open index.html in your favourite browser, and navigate to the page for the PBSReplicator module. Notice that in these docs the inner variable and functions are included.

Adding Additional Pages

JSDoc has support for additional pages in the documentation that are not created from doc comments, but from stand-alone files. It refers to these pages as tutorials, but they can contain anything.

The files can be in HTML or Markdown, and need to be saved in a single folder which is specified with the --tutorials flag. The files can have any of the standard file extensions for those file types, e.g. .md or .markdown for Markdown, and .htm or .html for HTML files. We’ll be using Markdown in this series.

Under the hood JSDoc references each tutorial by a so-called tutorial identifier, this is simply the filename with the file extension removed. These identifiers are important because they allow us to reference our pages within our documentation using the @tutorial block tag, or the {@tutorial} inline tag.

For example, the doc comment for the Replicator class has been updated to reference the Star Trek tutorial with a block tag:

/**
 * A virtual Star Trek-style replicator.
 * 
 * …
 * 
 * @tutorial startrek
 */
class Replicator{
	// …
}

The description for the module has been updated to use an in-line tag to reference the PBS tutorial:

/**
 * …
 * 
 * This module serves as an example in the {@tutorial pbs} series.
 * 
 * @module PBSReplicator
 * @requires is_js
 * @see {@link https://is.js.org}
 * @requires lodash-es
 * @see {@link https://lodash.com}
 */

By default tutorials are titled with their identifier, but a JSON file can be used to specify custom titles, and even nesting.

To define the titles, start by creating a JSON file in the tutorials folder. You can name the file anything you like, but I like to name it index.json. Within the JSON file, create a top-level object (dictionary) indexed by the identifiers (filenames without extensions) for each of your tutorials. The values for these top-level keys should each be objects too, indexed by title, and optionally, children. As its name suggests, the title should be a string, and children should be an array of identifiers for the tutorials that should be nested within the top-level tutorial (for an example, see the documentation for tutorials).

In our very simple project we have no need for nesting, so our index.json is very straight-forward:

{
    "startrek": {
        "title": "Star Trek"
    },
    "pbs": {
        "title": "Programming by Stealth"
    }
}

JSDoc Themes (Templates)

The most visible customisation possible with JSDoc is theming the generated documentation site. JSDoc’s documentation refers to templates, but the community describes them as themes, which seems like a more appropriate term to me.

The basic process for using a non-default theme is the same for all themes, but the configuration is different for each theme, being driven by the features the theme offers.

The first step is to install the theme as a dev dependency with npm install --save-dev. Once that’s done the theme can be selected using the --template flag or the opts.template configuration option. The value to use depends on the theme, and the theme’s home page will usually show it. The appropriate value to specify the default theme is templates/default.

Choosing a Theme

I don’t like the default theme. It’s very pretty, so superficially it’s nice, but it’s not actually very effective at communicating information. It’s difficult to see where entries start and end, and the right-side menu design just does not work for me.

So, I set out to find some alternatives, one of which we’ll use in this series. To narrow down the field I used the following criteria:

1. Must be actively maintained
2. Must be responsive
3. Must be clear
4. Should be aesthetically pleasing, and ideally quite visually plain so as not to distract from the actual documentation

That left just three options I was able to find:

1. Minami — the latest version of a theme I’ve used for many projects over the years.
2. Clean JSDoc Theme
3. Docdash

Two of the three have sample sites linked from their GitHub pages, but I don’t find those very useful, instead, let’s test each on our example code. All three have been specified as dev dependencies already, so they were installed when you ran the npm ci command to initialise the pbs132a folder as a NodeJS project. I also added commands to generate each of them to package.json, so you can generate the three example sites with:

npm run docs-dev-minami
npm run docs-dev-clean
npm run docs-dev-docdash

This will create three folders, docs-dev-minami, docs-dev-clean & docs-dev-docdash. Open index.html in each in your favourite browser to see what they look like.

Minami

Starting with Minami — I really love the look with the different coloured rounded boxes acting as icons for classes, functions, etc..

I’ve used this theme many times over the years, and I’ve never been happy with how it lays out the sidebar. The fact that the functions are nested under the classes and modules is just not clear. The separation between entries on the page for a class or a module is much better than you get with the default theme, but still quite subtle.

The theme offers very few configuration options, so for the most part, what you see is what you get.

Unfortunately development on this theme seems to have stalled. It’s been two years since it was updated. It’s not quite abandoned, but it certainly doesn’t feel like it’s under active development.

Clean JSDoc Theme

Moving on to Clean JSDoc Theme — the sidebar is better, showing the nesting well, and while the collapsible sections seem like overkill on a small demo project, they make a lot of sense for larger code bases.

The separation between entries is OK, but not great.

Where this theme really shines is in its configurability. Lots and lots of settings to tweak.

The theme is also being actively maintained with the most recent commits just days ago.

Docdash

Finally, Docdash — the sidebar is again better than Minami, the nesting is very clear, and the separation between entries within classes etc. is extremely good with those big solid banners.

The theme is also extremely configurable, so a lot of opportunities to tweak and even enhance the theme.

Development on this theme seems much more active — the GitHub page shows relatively recent fixes, and a large team of contributors.

My only complaint is that I’m not a fan of the purple!

A Final Choice?

The first draft of this instalment ended with my sitting on the fence between Docdash and Clean, but Allison hated Clean, so that was enough to break the tie, and we’ll be using Docdash.

Configuring Docdash

One of the reasons I chose Docdash is that it’s configurable. Docdash expects its configuration settings to be stored under the top-level key docdash. We won’t go into all the details here, but let’s do a final example where we tweak Docdash a little.

To do this I created a duplicate of the JSDoc config file and named it jsdoc.docdash.conf.json. It’s identical to the previous config, but with a docdash section added. I also added a matching NPM automation script in package.json so we can generate documentation with this config file with the command npm run docs-dev-docdash-tweaked, and the resulting docs will be saved in the folder docs-dev-docdash-tweaked.

The first thing I changed was the inclusion of a search box, and the inclusion of static items and typedefs in the sidebar:

"docdash": {
    "search": true,
    "static": true,
    "typedefs": true
}

Next I reordered the sections in the side bar (the Global section can’t be moved for some odd reason), and configured the sidebar to only show nested detail in the current section:

"docdash": {
    "collapse": true,
    "search": true,
    "sectionOrder": ["Modules", "Classes", "Tutorials"],
    "static": true,
    "typedefs": true
},

Finally, I added some custom menu items:

"docdash": {
    "collapse": true,
    "menu": {
        "Bartificer Creations": {
            "href":"https://bartificer.net/",
            "target": "_blank"
        },
        "Podfeet Podcasts": {
            "href":"https://podfeet.com/",
            "target": "_blank"
        }
    },
    "search": true,
    "sectionOrder": ["Modules", "Classes", "Tutorials"],
    "static": true,
    "typedefs": true
}

In the real project this will be most useful for including a link to the project’s GitHub page.

Another feature that looks interesting is the ability to include custom scripts or CSS files. It should be possible to use this to replace the purple I dislike so much with a nice blue 🙂

Putting it all together our final JSDoc configuration becomes:

{
    "docdash": {
        "collapse": true,
        "menu": {
            "Bartificer Creations": {
                "href":"https://bartificer.net/",
                "target": "_blank"
            },
            "Podfeet Podcasts": {
                "href":"https://podfeet.com/",
                "target": "_blank"
            }
        },
        "search": true,
        "sectionOrder": ["Modules", "Classes", "Tutorials"],
        "static": true,
        "typedefs": true
    },
    "opts": {
        "encoding": "utf8",
        "pedantic": true,
        "readme": "./README.md",
        "tutorials": "./pages/"
    },
    "source": {
        "include": ["./src/"],
        "includePattern": ".+\\.(mjs|jsdoc)$"
    }, 
    "plugins": [
        "plugins/markdown"
    ],
    "tags": {
        "allowUnknownTags": false,
        "dictionaries": ["jsdoc","closure"]
    }
}

Final Thoughts

We’ve now seen how we can configure JSDoc to take the friction out of documenting our code, and hence, to make accurate and up-to-date documentation a realistic aim for the XKPasswd re-write project.