PBS 137 of X — Bundling a Library with Webpack
Since starting phase 2 of this series we’ve been building out the developer toolkit we’ll need to deliver a JavaScript version of the Crypt::HSXKPasswd Perl module, and we’ve been making excellent progress:
- Introduce ES6 modules — done ✅
- Introduce Node & NPM — done ✅
- Choose a Linter and learn how to use it — ESLint (with thanks to guest teacher Helma van der Linden) ✅
- Choose a documentation generator and learn how to use it — JSDoc with the DocDash theme ✅
- Choose a Test Driven Development (TDD) platform and learn how to use it — Jest ✅
- Choose a bundler and learn how to use it — to do
It’s time to complete the checklist — after a lot of procrastination, I’ve chosen to use the bundler that has by far the most community mind-share, Webpack.
Webpack is an extremely feature-rich and powerful tool. It’s so feature rich and powerful that it’s quite difficult for beginners to find the sub-set of that functionality that solves their specific problem best. It would take many many instalments to cover even all of the commonly-used features offered by Webpack, so we’re not going to do that. Instead, we’re going to look at the slice of Webpack that solves our immediate problem: bundling a collection of JavaScript modules and all their dependencies into a single .js file that browsers can import with a <script> tag.
Basically — we want to use Webpack to make the new HSXKPasswd library as easy to import into a web page as jQuery is!
Matching Podcast Episode
Listen along to this instalment on episode 724 of the Chit Chat Across the Pond Podcast.
You can also Download the MP3
Episode Resources
- The instalment ZIP file — pbs137.zip.
Why do we need a Bundler?
When you use an open source library like jQuery or MomentJS in your code, you import a single file, usually from a CDN (content delivery network), and it injects a variable into the global scope that you use to access the library, $ in case of jQuery, and moment in the case of MomentJS. If you’ve ever opened one of those all-in-one files, you’ll notice the code looks utterly cryptic and like no human could understand or maintain it. That’s because it wasn’t written by a human; it was built by a bundler based on some (hopefully) well organised human-readable code, probably broken into lots of sub-modules and spread across multiple files. There’s also a good chance the code the humans wrote imported 3rd-party code from elsewhere, and the bundler injected all that code into the one file too.
So, when it comes to bundling a library, the input is a folder full of your own JavaScript files which can, in turn, import third-party modules, and the output is a single file that includes and encapsulates all your code and all the dependencies it imports. This single output file can be formatted in many different was, including as an ES 6 JavaScript module, or any of the older standard that pre-date the ES 6 spec like CommonJS. One of the most common output formats is UMD (Universal Module Definition), which is compatible with all the common pseudo-standards, and is still the format with the broadest browser support. This is why UMD is the format used by many major open source libraries like jQuery and MomentJS.
Why Encapsulate Dependencies?
The encapsulation of dependencies is one of the most important concepts to understand. Up until this point, when our code has made use of a third party library like jQuery or MomentJS we have relied on those dependencies being loaded separately form our code with their own script tags pointing at an appropriate CDN to fetch the dependencies.
I’ve avoided the jargon until now, but what we have been doing is specifying all our dependencies as peer dependencies. There are times when you want to do that, and bundlers like Webpack support that. When you specify a module as a peer dependency in your Webpack config it will be excluded from the bundle. A real-world example of this is that because jQuery is so big, Bootstrap 4 chose to make jQuery a peer dependency, hence we always had to use two script tags to get Bootstrap 4’s JavaScript functionality to work — jQuery from a CDN, then Bootstrap 4 from a CDN.
Most of the time though, we don’t want to burden the users of our library like that — we want them to be able to add a single script tag into their HTML, and have the library we published just work. This is the experience we get with MomentJS.
User convenience is definitely a major reason to encapsulate dependencies, but there are other reasons too, including reliability, and security.
With peer dependencies, your code is at the mercy of the user’s ability to reliably import the correct versions of those dependencies. This makes your code unreliable in two ways — firstly, they can import a slightly different version to the one you tested on, opening up the possibility of difficult-to-reproduce bugs, and secondly, if they use a CDN, your code is now dependent on the reliability of a CDN over which you have no control.
This brings us nicely to security. Encapsulating dependencies gives you control over your library’s security, but it also gives you responsibility for your library’s security! There are two scenarios I want to highlight.
Firstly — if a vulnerability is found and patched in a peer dependency, your users all have to be sure they update their code to include a non-vulnerable version. If you encapsulate the dependency you can simply update the version you bundle and publish an update to your library that’s secure.
Secondly — if a dependency author goes rogue and sabotages their library (as has happened repeatedly in the spring of 2022), your users won’t get that bad update if you don’t update your bundle. This is why maintainers of bundled projects have a responsibility to update their dependencies in a controlled way, and not in an entirely automatic way. This caught quite a few open source project offguard in the spring of 2022, because they had entirely automated processes that blindly accepted all dependency updates and built fresh bundles nightly or even hourly, hence they inadvertently spread malware!
Introducing Webpack
Webpack is an open source project primarily developed to bundle all the static assets for a website together, and later extended to support bundling libraries. The reason I spent so much time procrastinating between Webpack and the second contender, Rollup, is that Rollup is the opposite. It was primarily designed to bundle libraries, then expanded to support bundling websites. I finally decided on Webpack for two reasons:
- We can use it both to bundle the library itself initially, and later, the new XKPasswd website.
- It has the biggest mindshare in the community by a long shot, so we’ll be learning a very transferrable skill, rather than learning something more niche.
So, with all that said, let’s meet Webpack! To be more specific, let’s meet Webpack 5, the major version we’ll be using.
Webpack is a Node Package
First and foremost, Webpack is written from the ground up to be used within NodeJS. If you’re not using NodeJS to build your library, web app, or website, you can’t use Webpack.
Webpack is a big ecosystem with lots of officially-supported and third-party extensions, so it’s not actually one package, but many. At a very minimum you’ll need the core Webpack library (webpack), and the Webpack CLI (webpack-cli). All the Webpack dependencies should be installed into your Node project as dev dependencies, i.e. with the npm install --save-dev command.
Core Concepts
Webpack’s own documentation opens with a page describing the core concepts, and I think we should follow their lead.
Everything starts with an entry point, this is the thing you’re trying to publish — it could be the primary file in a Javascript library, the main file for a JavaScript app, or the home page of a website.
To both reduce the workload on the developer and avoid bundling more than is needed, Webpack tries to figure out what to bundle and what not to bundle by loading the entry point and then following all the imports to build a dependency graph. You can use Webpack’s config file to tweak things, but it tries to automate as much of the process as it can.
At the opposite end we have the output, this will be a folder with a name of your choosing into which Webpack will write the file or files it creates based on your entry point and config. This will be the folder you publish as your library, app, or website.
By default Webpack’s output will be compatible with browsers that support ES5 and above (so basically any vaguely modern browser), but it can be configured to target different environments, and when developing a library, it can be configured to produce multiple copies of the output targeted at different environments. We’ll be making use of this ability to generate two bundled versions of the new HSXKPasswd module — one targeted at all browsers, and one for use as an ES 6 module.
Webpack is first and foremost a JavaScript bundler, so out of the box it only deals with JavaScript code and JavaScript dependencies. That’s often all you need for a library, but web apps and web sites need more, they almost always need things like CSS and images, and may have more advanced requirements like a CSS pre-processor (we’ve not met any of those yet, but we will when we move on to rebuilding the XKPasswd site later in the series). To allow Webpack to handle all your bundling needs its core functionality can be extended using loaders for the relevant content types.
For now, we won’t be using any loaders, but we will need them when we revisit Webpack for bundling the new XKPasswd website.
While loaders add support for additional content types, Webpack’s functionality can be augmented even more with plugins. Again, we won’t be using any plugins at the moment, but it’s possible we’ll need some in the future.
Finally, Webpack can operate in two distinct modes — production and development (production being the default). In development mode Webpack performs fewer optimisations, so it builds large projects more quickly.
Webpack’s Configuration File
When Webpack executes, it expects to have its configuration in the form of a JavaScript dictionary (plain object), so you might expect that means it expects a JSON file, but no, it expects its configuration file as a JavaScript module that publishes a dictionary as it’s default export. In other words, the config file is actually a JavaScript file. The webpack CLI expects to find the JavaScript file to load in the root of your node project in a file named webpack.config.js.
Because NodeJS is only now transitioning to ES6 modules, and because most developers have not made the change yet, the fact that the config is a module is a little inconvenient when you’re one of the early-adopters opting to use NodeJS in ES6-mode (like us). Why? Because the example configs in both the official Webpack docs, and those you tend to find on the internet are all written in CommonJS format, and they need to be tweaked to use ES6 syntax instead.
This is going to become ever less of an issue over time, so while it will be an annoyance for a while, it seems the better choice than to start by learning an old technology that’s on the way out!
Worked Example — Packing our Joiner
This is all very abstract, so let’s learn how to bundle an ES6-based JavaScript library with Webpack by bundling an ES6 module with Webpack 🙂
Specifically, let’s bundle a tweaked version of the Joiner module we used as our example in instalment 136.
Before we Begin – a Small Tweak to add a Dependency
So far I’ve been going out of my way to avoid our joiner module having any dependencies, but to make this example more realistic, we need to add one.
I’ve been a huge fan of the is.js micro-typechecking library, but unfortunately it’s maintainer has gone quiet, and the library has been languishing. But, because it’s a good library, and because its open source, someone else has taken the baton and run with it by forking the code, upgrading it to an ES6 module, fixing a bunch of bugs, and re-naming it to is-it-check.
I decided to replace the clunky type checks like if (typeof conjunction !== 'string') with more readable alternative from is-it-check like if (is.not.string(conjunction)).
The first step was to install is-it-check into the project as a regular dependency with the command:
npm install --save is-it-check
Before doing anything else I also decided to take this opportunity to place the entire Node project into ES6 mode, and remove the need to use the .mjs file extension which confuses some editors into not using the right syntax highlighting. To do this I first re-named src/joiner.mjs to src/joiner.js then added the following two lines to my package.json file:
"type": "module",
"module": "src/joiner.js",
With that done I could import is-it-check into joiner.js with the name is by adding the following near the top of the file:
import is from 'is-it-check';
That allowed me to replace all the complex type checks with nice simple ones.
Note that having a Jest test suite in place helped me make this change with confidence — I could be confident my changes didn’t break the code because the test suite still passed when I was finished 🙂
Our Starting Point
If you’d like to play along, you’ll find the starting point in this instalment’s ZIP as the folder pbs137a-joiner-before. This folder is a NodeJS project with our existing tool-chain implemented — there is an ESLint config, JSDoc has been configured, there’s a Jest test suite in src/joiner.test.js, and there are NPM scripts defined named docs and test to build the docs and run the Jest test suite. The code for the module is in src/joiner.js.
Finally, to avoid the docs folder getting saved into the PBS Git repo, I also added a local .gitignore file into the folder to tell Git to ignore the docs sub folder.
Note: because we’re using NodeJS in ES6 mode, be sure you’re running at least the latest LTS version or this example may not work for you. (It’s been tested on v16.14.2, the most recent LTS as of 14 April 2022)
To gets started, change into the folder and initialise the Node project with the command:
npm ci
Verify everything is working as expected by running the Jest test suite with:
npm run test
Optionally, build the documentation for the module with the command:
npm run docs
Step 1 — Install Webpack as a Dev Dependency
Before we can use Webpack we need to install it into the project. Since we won’t be using any optional loaders or plugins we only need to install the two core packages:
npm install --save-dev webpack webpack-cli
Step 2 — Create a Webpack Config
With Webpack installed we now need to tell it what we’d like it to do by creating a file named webpack.config.js with the following contents:
// Needed hackery to get __filename and __dirname in ES6 mode
// see: https://stackoverflow.com/questions/46745014/alternative-for-dirname-in-node-js-when-using-es6-modules
import path from 'node:path';
import { fileURLToPath } from 'node:url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
export default [
// output an old-style universal module for use in browsers
{
entry: './src/joiner.js',
output: {
path: path.resolve(__dirname, 'dist'),
filename: 'joiner-universal.js',
library: {
name: 'joiner',
type: 'umd',
export: 'default',
},
},
},
// output an ES6 module
{
entry: './src/joiner.js',
experiments: {
outputModule: true,
},
output: {
path: path.resolve(__dirname, 'dist'),
filename: 'joiner-es6.js',
library: {
type: 'module',
},
},
},
];
The first 7 lines are just some boilerplate code to work around some limitations in caused by using NodeJS in ES6 mode. In traditional NodeJS there are two special variables available at all time, __filename is the full path to the JavaScript file that the line of code is contained within, and __dirname is the folder that that JavaScript file is contained within. Neither of these two variables are available when NodeJS is running in ES6 mode, so we need to create them, hence the boiler plate.
The real config starts on line 9, with a default export of an array of two dictionaries. This is not actually the most common scenario — it’s more usual to see the default export be of a single dictionary. The reason we are using an array of two dictionaries is that we are actually defining two separate Webpack configs in a single file — one to build a universal module, and one to build an ES6 module.
Note that both config dictionaries define the same entry point, our joiner module:
entry: './src/joiner.js',
This is a relative URL within our NodeJS project, so it’s in the format NodeJS expects of module imports.
Also notice that both config dictionaries define a dictionary to configure their output (named output). Both define a key named path to specify the same output folder, but this time the folder specification is different, it uses NodeJS’s path.resolve() function to build an absolute path relative to the config file’s location. The reason for this is that output folder needs to be specified as a full path in the appropriate formate for the OS the module is being built on, the path.resolve() function ensures the correct path separator is used for the current OS.
Note that I’m choosing to output both bundles to a folder named dist, this is short for distribution, and it’s one of two commonly used conventions, the other being a folder named build. Both are equally valid, and it really does just come down to a matter of taste, I just prefer dist because that’s the folder you ZIP up to distribute your module to others.
Both config dictionaries also specify the name for the bundle they’ll produce using the filename key. Finally, both output dictionaries define sub-dictionaries named library. The presence of this dictionary tells Webpack that we’d like it to bundle a library, not a website or web app, and the keys within specify the details. Both library dictionaries specify a type, but that’s where they diverge.
Let’s look at each in more detail, starting with the one for creating a universal module:
library: {
type: 'umd',
export: 'default',
name: 'joiner',
},
Setting the type to umd (for Universal Module Definition) tells Webpack that we want to build the bundle as a universal module. A universal module export one thing into the global name space with a specific name, so we need to tell it which of our module’s exports to map to what name, which we do with the export and name keys. In this case we are saying that we want to add the default export from our entry point into the global namespace as joiner.
Finally, let’s look at the much simpler library dictionary for exporting the ES6 module:
library: {
type: 'module',
},
Setting the type to module tells Webpack we want an ES6 module, and that’s all we need to tell it, so that’s the only key in the dictionary! One thing to note though is that as of April 2022 Webpack’s support for ES6 modules is still officially experimental, so, for a type of module to work, we must enable the appropriate experimental feature, hence the experiments sub-dictionary at the root of the containing config dictionary:
experiments: {
outputModule: true,
},
Running Webpack
At this stage we have all we need to run Webpack. We can execute it directly from the command line with:
npx webpack
This will give a warning that no mode is specified, and that it is defaulting to production. To avoid that warning specify a mode with the --mode flag, e.g.:
npx webpack --mode=production
To avoid having to remember the syntax, it’s good advice to add an NPM script to your package.json with the standard name build, we do that by adding an entry to the scripts dictionary like so:
"scripts": {
"build": "npx webpack --mode=production",
…
}
We can now build our module with the command:
npm run build
Using our Bundled Module in the Browser
OK, so we’ve now built our module, how do we use it?
Simple, we create a <script> tag with its src pointed at joiner-universal.js, and then we can use the joiner as joiner().
You’ll find both the built bundles and an example HTML file named example.html in the pbs137b-joiner-after/dist folder in the instalment ZIP.
We won’t go through the entire file, there are just two key points I want to highlight.
Firstly, we include our library with a regular <script> tag, but we don’t include the is-it-check library anywhere because it has been bundled into the single JavaScript file we include — our users don’t have to worry about the dependencies we’ve chosen to rely on!
<!-- Import the Universal version of the bundled Joiner library - imports as 'joiner' -->
<script src="joiner-universal.js"></script>
Secondly, our module’s default export, a function, has been added to the global scope as joiner(), so we can just use it in our scripts:
// build a joiner with the appropriate settings
const myJoiner = joiner(
$conjunctionSel.val(),
$quoteRadSet.filter(':checked').val(),
$sortCB.prop('checked')
);
// convert the list to a human-friendly string
const joinedList = myJoiner.join(list);
Finally, as an added little bonus, and because I need the practice, I’ve used Bootstrap 5 to style the page. The most significant change you’ll notice in this page is my use of the entirely new floating labels feature on the form inputs. If you’re very observant you’ll notice just two other really subtle changes from Bootstrap 4:
- I had to add a fluid container inside the nav bar that was not required in Bootstrap 4.
- In Bootstrap 4 each form element with all its matching labels and instructional text were wrapped in elements with the class
form-group, that’s gone from Bootstrap 5, the advice now is to simply use the margin utility classes, e.g.m-3, to space out your form’s elements.
Using our Bundled ES6 Module with NodeJS
In the pbs137b-joiner-after/dist in the instalment ZIP you’ll also find the file example.mjs, this is a NodeJS script that uses our bundled module. The script is very short, so here it is in its entirety:
// import the bundled ES 6 module
import joiner from './joiner-es6.js';
// define a list to join
const foodChoices = [
'pancakes',
'waffles',
'popcorn'
];
// join the list
const foods = joiner().or.sort.join(foodChoices);
// print it
console.log(`I wonder which Allison prefers — ${foods}?`);
If you change into the folder in your terminal (and you have the latest stable NodeJS installed) you can run it with:
node example.mjs
It will print:
I wonder which Allison prefers — pancakes, popcorn or waffles?
The key point to note is that we imported the module’s default export in the usual ES6 way:
// import the bundled ES 6 module
import joiner from './joiner-es6.js';
At that point we chose to name the default export joiner, so we can then use it as normal:
// join the list
const foods = joiner().or.sort.join(foodChoices);
Because NodeJS is so good at dealing with dependencies it’s not as obvious here, but again, the is-it-check module is not being loaded from ../../node_modules, but from inside the bundle.
If you don’t trust me and want to verify this for yourself, you can run node with the --prof flag to enable profiling. This will generate a log file reporting everything node did to execute the example script. If you search that log for lines starting code-creation,Script you’ll see that just two files were loaded, example.mjs, and joiner-es6.js.
Final Thoughts
It’s important to remember that we’ve look at just a tiny sub-set of what Webpack can do here — we’ve solved one very specific problem, and ignored everything else!
We’ll be re-visiting Webpack again later in the series, because we’ll need to learn more about how it works so it can solve a completely different problem for us — deploying a JavaScript web app with a whole bunch of dependencies as a stand-alone app that somehow embeds its dependencies in a manageable way, removing our dependence on CDNs to deliver commonly used libraries like jQuery. Don’t worry if that doesn’t make sense to you yet, it should do once we get as far as developing the new web front-end for the JavaScript version of Crypt::HSXKPasswd.
Speaking of which — we’ve just completed our preparation checklist!
- Introduce ES6 modules — done ✅
- Introduce Node & NPM — done ✅
- Choose a Linter and learn how to use it — ESLint (with thanks to guest teacher Helma van der Linden) ✅
- Chose a documentation generator and learn how to use it — JSDoc with the DocDash theme ✅
- Choose a Test Driven Development (TDD) platform and learn how to use it — Jest ✅
- Choose a bundler and learn how to use it — Webpack ✅
We’re now ready to start into the gargantuan task of porting Crypt::HSXKPasswd to JavaScript — yikes! 😬