PBS 127 of X — Introducing NPM (and Node)
We’re going to start the second phase of this series by expanding our developer’s toolbox. Our developer’s toolbox is basically code to help us write code!
We could install all our developer tools globally on our systems, but that comes with some notable drawbacks:
- You need admin access to a computer to install things system-wide
- You can only have one version of a tool when you install it system-wide, but different projects might need different versions
- If a tool gets a major update that requires some kind of tweak or change within your projects, you’ll have to do them all at once.
So, it’s much better to install developer tools locally, within a project. Each project gets its own copy of the tools at what ever version it wants, and no admin access is needed.
Manually managing local copies of tools in each project would be a chore, but thankfully, we can use a tool to help us manage the rest of our tools!
For the first two milestones for Phase 2 our developer tools will all be JavaScript-based, so the ideal tool to manage them is a JavaScript package manager, and the obvious choice there is the Node Package Manager, or NPM.
Matching Podcast Episode
Listen along to this instalment on episode 701 of the Chit Chat Across the Pond Podcast.
You can also Download the MP3
Instalment Resources
- The instalment ZIP file — pbs127.zip.
What are Node & NPM?
The NodeJS JavaScript Runtime
As you might infer from the name, the Node Package Manager is part of the Node JavaScript runtime, better known as NodeJS. So far in this series we have run all of our JavaScript in the browser. The browser provides both the core JavaScript language, and the DOM, and, it comes with a very severe restriction — the browser confines JavaScript code inside a very restrictive sandbox. NodeJS is an entirely different environment for executing JavaScript. Like the browser NodeJS can execute core JavaScript, but there is no DOM in NodeJS, instead, NodeJS provides its own set of APIs, and unlike the browser, NodeJS does not sandbox your JavaScript code. This means that code running in NodeJS is free to read and write files, to interact with processes, to receive incoming network connections, and even to create windows and run custom GUI code within them.
With NodeJS, JavaScript code can be used to write command line apps, to act as the back-end for a web server, or indeed to implement a server for any networked protocol, and even to power GUI apps. In short, NodeJS allows JavaScript to break free from the confines of the browser, and do pretty much anything!
NodeJS is free, open source, and cross-platform, so everyone can play along. While NodeJS is not a browser, it is very closely related to one very specific browser — Google’s open source Chromium. When Google set out to develop their own browser they needed to create a JavaScript runtime for the browser so it could support JavaScript. At the time one of the bottlenecks on then-modern websites was slow JavaScript execution. Google could have taken an existing open source JavaScript engine like SpiderMonkey, the JavaScript engine that powers Firefox, but they chose not to. Instead, Google wanted to start fresh so their JavaScript engine would be fast and efficient without the decade or so of technical debt SpiderMonkey had accumulated. They succeeded in developing a lightning-fast JavaScript engine, and they named it for one of the most iconic engines in the world — V8.
While Google developed V8 for their browser, because it’s open source, it’s found quite a few other homes since it was released in 2008, including NodeJS.
The Node Package Manager (NPM)
If all NodeJS did was run JavaScript outside the browser it would already be a darn useful tool, but it does a lot more than that — it elegantly solves the problem of sharing and reusing code without ending up in dependency hell. Before NodeJS, JavaScript as a language was missing a critical feature — the ability to easily bundle code into self-contained sharable and reusable chunks. Or, to use the correct jargon — JavaScript didn’t provide support for packages. The developers of NodeJS solved this problem very elegantly, having learned both what works and what doesn’t from earlier packaging systems like Perl’s Comprehensive Perl Archive Network, or CPAN. The end result is the Node Package Manager, or NPM.
When you install NodeJS you get two commands — node which gives you a JavaScript shell, and npm for managing JavaScript packages.
Why do we need NPM (or Node)?
The larger Node environment can do so much, it’s very easy to get overwhelmed, so we’re going to take a very different approach to the one we took with Git or Chezmoi recently. Rather than do a multi-instalment series-within-a-series, we’re going to learn about the pieces of the Node ecosystem as we need them.
NPM for Dependency Management
Our first encounter with Node is going to be as a tool for automatically fetching open source JavaScript resources and pulling them into our project.
NodeJS can import both local files and remotely sourced files using the same syntax, but how does it get the remote files? That’s where the NPM website comes in — www.npmjs.com hosts a massive repository of free and open source JavaScript modules. We can use the npm command can fetch our dependencies from this massive repository for us, and to handle all the interdependencies for us. For example, Bootstrap 4 depends on jQuery and Popper, so if we tell NPM we want Bootstrap 4, it will automatically fetch jQuery and Popper too.
NPM supports two kinds of dependencies — traditional dependencies and developer dependencies, or “things my code needs to run” and “things to help me create and ship my code”. If we look at the JavaScript modules we’ve already used, then jQuery, Bootstrap, Moment.js, and is.js are traditional dependencies. If I write a web app that uses jQuery, then my web app needs jQuery to function, or, to use the jargon, my web app depends on jQuery.
For example, to import the latest version of the MomentJS module into a NodeJS project we’d simply run the command npm install --save moment.
Almost all the 3rd-party code we’ve used so far in this series has been used as traditional dependencies, but there are two exceptions — the testing framework QUnit is a dev dependency, as is the document generator JSDoc. QUnit helps us create code, but the code we create runs just fine without it. Our code does not depend on QUnit, but our development process does.
Our first milestone is to ship a JavaScript version of the existing Crypt::HSXKPasswd Perl module. The first thing we’ll be using NPM for is to manage our dev dependencies, specifically, to provide us with:
- A code linter ESLint
- A testing framework (TBD)
- A documentation generator (probably documentation.js)
As we write the code we’re likely to encounter the need to make use of some third party code to save us reinventing the wheel. When that happens we’ll use NPM to manage those regular dependencies.
When it comes time to ship the code we’ll pick up another dev dependency, a so-called bundler or packager, and while that decision is a while off yet, I’m leaning towards WebPack.
NPM for Developer Task Automation
Having developer tools is all good and well, but you need to actually use them! Remembering all the arguments you want to pass to your documentation generator each time you generate docs would be a pain in the proverbials. The same goes for executing our hopefully ever growing test suite.
NPM to the rescue yet again! We can use NPM to define and execute named tasks, so instead of having to remember all the arguments, we can simply tell NPM to generate our docs, or run our test with commands like npm run docs and npm run test. Finally, when it’s time to publish our work, we’ll define a build script we can invoke with npm run build.
Node for Testing
To test a JavaScript module you need to run JavaScript. Previously we’ve used the browser to run our QUnit tests, but there’s no need for all that overhead. We can use the Node JavaScript runtime to do all our tests from the command line. What ever test suite we choose to use, we’ll be executing it from the command line using npm run, and it will be the NodeJS engine executing the code.
NPM for Module Bundling and Publishing
When we arrive at our first milestone and have some code ready for publishing as a module for others to import into their code, we’ll be using NPM to install and run a bundler, but we’ll also be using NPM to publish our module to the public NPM repository. That will allow anyone to use our module in their code by simply running npm install --save hsxkpasswd!
Node to Build a Command Line App
Finally, one of our long-term milestones is to replace the existing Perl-based hsxkpasswd terminal command with a new JavaScript-powered one. When we do that, NodeJS will be the runtime we use.
My First NodeJS Project — Sleeps to Christmas
Before we finish up this instalment, let’s get practical for a moment and create a very simple NodeJS project so we can get a hint of what it’s like to code JavaScript with NPM at our disposal.
Install NodeJS (and get NPM for free)
NPM ships as part of NodeJS, so to install NPM, simply install the latest LTS version of NodeJS from nodejs.org. There are packages for just about every OS, and at any one time there are always two supported versions, the most recent stable version, and a long term support or LTS version. I’ll be using the LTS version in this series, but feel free to install the latest stable if you prefer, that will give you more features, but you’ll need to keep updating more often.
NodeJS Projects
I like to think of NodeJS as a multiverse of independent projects. Each one has its own dependencies, and while the same copy of NodeJS manages them all, they can each be running different versions of the same dependencies. To NodeJS, each project is its own little universe, and all a project is is a folder with a file named package.json in it.
Over time our package.json files will grow more complex, but for now we’ll start very simple. First, create a folder named pbs127a-s2xmas and open a terminal in that folder.
We can either manually assemble a basic package.json file, or, we can have npm step us through the process. Since this is our first project, let’s let npm hold our hand and run the command:
npm init
This will ask us a series of questions, and make guesses along the way. It will ask us to name our project, and it will suggest the name of the folder. Usually that’s the right answer, but in this case, we don’t want the pbs127a- prefix, so change the name to just s2xmas.
Next it will ask for a version — NodeJS/NPM insist on Semantic Versioning (SemVer), so it’s suggested initial version of 1.0.0 is fine for today.
Next we’re asked for a description of our project. You can leave this blank, but it often helps your own sanity when you come back to a project later to have at least a one sentence description of what you thought you were doing! In this case simple enter Sleeps to Christmas calculator.
Next it will ask for the path to the JavaScript file that will act as the project’s entry point — this is the file people should run to start whatever it is you’re making. In our case, that will be s2xmas.mjs. (As we’ll learn soon, the .mjs file extension is important.)
Next it will ask for your test command. We don’t have one on a project this small, so we’ll just leave this empty.
Finally, NPM will ask for the metadata it would use when publishing the project to the NPM repository. We won’t be doing that with this little dummy project so we can leave the Git repo and keywords blank. We could also leave the author blank, but my vanity always seems to get me filling that in 🙂 And lastly, NPM wants to know what license the code is under. It suggests the permissive ISC open source license, which I’m happy to accept most of the time.
Having finally finished its game of 20 questions, NPM will show you the package.json file its assembled, and ask for confirmation before saving it.
Here are the answers I gave:
package name: (pbs127a-s2xmas) s2xmas
version: (1.0.0)
description: Sleeps to Christmas calculator.
entry point: (index.js) s2xmas.mjs
test command:
git repository:
keywords:
author: Bart Busschots
license: (ISC)
And here’s the resulting package.json file:
{
"name": "s2xmas",
"version": "1.0.0",
"description": "Sleeps to Christmas calculator.",
"main": "s2xmas.mjs",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "Bart Busschots",
"license": "ISC"
}
We’re now ready to start using NPM.
Installing a Regular Dependency
As you’ve probably guessed from the description in the package.json file, we’re writing a little calculator to tell us how many sleeps there are till the next Christmas. This means we’ll be doing date math, this means we need Moment.js as a regular dependency. Let’s ask NPM to install it for us:
npm install --save moment
This command does three things — firstly, it downloads the code and installs it into a folder named moment inside a folder named node_modules. All the dependencies NPM installs go inside this folder. Note that this folder should be added to your .gitignore file when using Git to version a NodeJS-powered project.
Secondly, it creates a file named package-lock.json. This is an important metadata file that’s needed when using various developer automation tools, and when collaborating with other developers via a source control system. For now, just know that you don’t ever manually edit that file, but that you absolutely should track it if you’re using Git.
Finally, the npm install --save command updated package.json to list moment as a dependency:
bart-imac2018:pbs127a-s2xmas bart% cat package.json
{
"name": "s2xmas",
"version": "1.0.0",
"description": "Sleeps to Christmas calculator.",
"main": "s2xmas.mjs",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "Bart Busschots",
"license": "ISC",
"dependencies": {
"moment": "^2.29.1"
}
}
bart-imac2018:pbs127a-s2xmas bart%
We can now write our code and save it as s2xmas.mjs:
import moment from "moment";
const now = moment();
if(now.date() === 25 && now.month() === 11){
// it's Christmas!
console.log("No more sleeps — it's Christmas 😀🎄🎁")
}else{
const xmas = moment(now.startOf('day')).date(25).month(11);
if(now.isAfter(xmas)) xmas.year(now.year() + 1);
const numDays = Math.abs(now.startOf('day').diff(xmas, 'days'));
console.log(`${numDays} sleeps 😴 till Christmas 🎄`);
}
We can now run the script with the node command:
bart-imac2018:pbs127a-s2xmas bart% node s2xmas.mjs
70 sleeps 😴 till Christmas 🎄
bart-imac2018:pbs127a-s2xmas bart%
But on POSIX OSes (Linux/Unix/Mac) we can go one better by first adding Node’s shebang line (#!/usr/bin/env node) to the top of the file:
#!/usr/bin/env node
import moment from "moment";
const now = moment();
if(now.date() === 25 && now.month() === 11){
// it's Christmas!
console.log("No more sleeps — it's Christmas 😀🎄🎁")
}else{
const xmas = moment(now.startOf('day')).date(25).month(11);
if(now.isAfter(xmas)) xmas.year(now.year() + 1);
const numDays = Math.abs(now.startOf('day').diff(xmas, 'days'));
console.log(`${numDays} sleeps 😴 till Christmas 🎄`);
}
And giving the file execute permission:
chmod +x s2xmas.mjs
We can now execute the file directly:
bart-imac2018:pbs127a-s2xmas bart% ./s2xmas.mjs
70 sleeps 😴 till Christmas 🎄
bart-imac2018:pbs127a-s2xmas bart%
Final Thoughts
Our first milestone is to publish the brains of XKPasswd as a JavaScript module, i.e. to make it usable by others in the same way that we used Moment.js in our example above. We’ll be doing that by publishing the code as an ES 6 module. We got a glimpse of how ES 6 modules work in our little example when we used the import command to load MomentJS into our little app, but there’s obviously a lot more to learn!
In the next instalment we’ll look at the import command in a lot more detail, but we’ll also learn how to publish our own code as a module that can be imported with the import command.
We’re fortunate to be learning about JavaScript modules at a time when there is a single official standard — ES 6 modules, but that’s a very recent development, and the road to that standard was a very very messy one. While it’s tempting to ignore the messy history of competing JavaScript module ‘standards’, we really can’t because the internet is still full of content referencing the now obsolete syntaxes. To avoid a world of frustration we need to be able to recognise the old ways of doing things so we can know which advice to follow, and which to ignore when we search online!