PBS 126 of X: Introducing Phase 2
This instalment marks a dramatic change in the structure of this series. Our aims remain un-changed — myself and Allison are striving to gently ease readers/listeners into ever more powerful programming activities without overwhelming people. Computers shouldn’t be intimidating, we shouldn’t be their slaves, we should all be able to assert control over them and make them work for us! What’s changing is how we’ll be organising and structuring our learning.
Because this is such a big change, I want to spend this instalment taking stock of what we’ve learned to date, what still remains as possible areas of exploration, and how we’ll be structuring this series for the next hundred or so episodes.
Matching Podcast Episode
Listen along to this instalment on episode 700 of the Chit Chat Across the Pond Podcast.
You can also Download the MP3
A Lot Done
At this stage in our programming journey we’ve come a long way — in terms of specifics, we’ve learned the three core languages that drive client-side web development; HTML for defining what’s on a page, CSS for defining how it should look, and JavaScript for defining how it should behave. We’ve also learned to use some specific third party libraries like jQuery, Bootstrap, is.js, Moment.js, and Numeral.js. We’ve also specifically learned about Git and most recently, Chezmoi. However, the specifics are not really what the series is about, those are simply the practical examples of the core concepts the series is really trying to teach. This isn’t Web Programming by Stealth, it’s Programming by Stealth!
The truly valuable things we’ve learned are core programming concepts like the difference between syntax and semantics — syntactically the if statement below is perfect JavaScript, but semantically it’s utter nonsense!
if(4/0 === 'boogers') console.log('why will this line not print???');
We’ve also learned about variables, types, operators, conditionals, loops, functions, objects, events, and exceptions. None of these concepts are JavaScript-specific, they all exist in other languages too, just with different syntaxes. These are concepts I explicitly taught as part of the series, but I also tried to instil some equally important soft skills too — things like the ability to read documentation and the appropriate vocabulary to describe your problems so you can search for solutions online.
Finally, we’ve made a start at building out our developer’s toolkit with the skills needed to manage real-world programming projects like documentation generation, testing, and version control.
A Lot More We Can Do
So far we’ve seen a single implementation of each concept — HTML for marking up information with semantic meaning, CSS for defining styles, JavaScript’s implementation of variables, loops, objects, events, etc., and Git’s implementation of version control.
It’s time to start consolidating our understanding by re-visiting many of these core concepts in different languages, and our developer’s toolbox is badly in need of having some additional tools added (and some of the old one sharpened). Finally, there are many many concepts we’ve not even met for the first time yet!
We have soooo much more we can learn — we’ve utterly ignored the server-side of the web so far, hence, we’ve not come across the Common Gateway Interface which allows requests from web clients to be answered by code written in literally any programming language invented by human kind. On the client side there’s just one markup language, one styling language, and one interaction language with any kind of widespread support — but things couldn’t be more different on the server — popular websites are implemented in Java, JavaScript, C#, Ruby, Python, PHP, and there’s still a little Perl holding up critical parts of our online lives too.
Unlike browsers, web servers are expected to store data, so they do that in all sorts of ways — files, structured data like JSON, traditional relational databases like MySQL, modern NoSQL databases like MongoDB, and high-speed short-term key-value caches like Redis. With the exception of JSON, we’ve not come across any of these concepts yet.
As well as new languages and new concepts, we also need to bulk out our toolbox — we need to learn about entirely new thing like code linters and package managers, and we need to update and expand our experience with documentation generators and testing frameworks.
All that’s just on the web, I also want to expand our horizons beyond the web — we should be able to write automation scripts and command-line tools too, not to mention traditional GUI apps!
Why a Big Change Now?
In an ideal world, one where I had infinite time, I’d keep this series going in the current format until we were done, i.e. indefinitely, because we’ll never be done! But of course, none of us have infinite time, so we all have to make tradeoffs.
This series has taken up close to all of my personal nerding-out time not already taken up by my two podcasts (Let’s Talk Apple & Let’s Talk Photo). Before this series started I used that time to work on personal coding projects and open source tools, especially my favourite digital child, HSXKPasswd. It’s not a coincidence that work on that project has utterly stalled in the last few years, the time and energy I used to put into that is now all going into this series!
XKPasswd has finally reached a crisis point where I can’t keep ignoring it, or it will die. The back-end library is written in Perl, a language I adore, but that the world has left behind, it needs to be re-platformed to something with a future, and soon. But there’s an even more urgent problem — the www.xkpasswd.net website is still powered by the original V1 Perl version of the library, the server it’s on is approaching end-of-live, and that old code can’t easily be run on a modern VM. The website design pre-dates the meteoric rise of the smartphone, so it’s a train-wreck on mobile devices.
To survive, at all XKPasswd needs two very urgent pieces of work — the core library needs to be migrated to a language with a future, and the website front and backends need to be re-built from scratch. The front end needs to become responsive, and the backend needs to be built on a widely supported server-side technology with a clear future. I’ve decided that means re-implementing the library as a modern JavaScript package, the web frontend in Bootstrap 5, and the server back-end in PHP.
To thrive, the perl-based command-line tool needs to be re-implemented in a modern language with a future, and since I’ve decided the library itself should go to JavaScript, that means the best technology for the re-implemented CLI is probably a NodeJS/NPM package, i.e. command-line JavaScript.
So, I could put this series on hiatus for at least six months and do all that work by myself, or, I could unite my two biggest personal projects into one, and use the re-platforming and re-designing of HSXKPasswd/xkpasswd.net as a practical vehicle to drive the learning in this series. The latter strikes me as better in every way, so that’s what I’ve chosen to do, hence Phase 2.
One big advantage I see in this approach is that we’ll be learning real-world project management. When it comes to getting bigger coding projects to produce code that actually works, the tools to manage the project are as important as the code for your actual site/app, so by building our learning around a real-world project, we’ll get to see those tools in action.
The Plan
I’ve been thinking a lot about how this crazy idea might work, and I’ve come up with this plan. But, I fully expect it to change as we get stuck in, because as Napoleon apparently said “no plan survives contact with the enemy”. So, given the caveat that all this will probably change, here’s my thinking as of October 2021.
Milestones
Firstly, these are the milestones I want to drive towards, in order (5 and 6 might get reversed), and the new things we’ll need to learn on our journey:
- Re-platform the Perl module
Crypt::HSXKPasswdto a JavaScript ES 6 module, probably namedHSXKPasswdJS.- ES 6 Modules
- The Node Package Manager (NPM)
- Code linting (probably with ESLint)
- Documentation generation (probably with documentation.js)
- Test Driven Development for modern JavaScript code (modules, classes, promises, etc.), with one of:
- Re-platform the
xkpasswd.netwebsite to Bootstrap 5 and the new JavaScript module. (No added functionality.)- Bootstrap 5
- A JavaScript bundler, probably Webpack
- Add support for user accounts to the
xkpasswd.netsite so users can create and save their own custom configs, and perhaps upload their own dictionaries. This will require a website back-end, which will be written in PHP 8. - Port the module to PHP
- PHP classes
- Develop a REST API to allow programmatic access to XKPasswd functionality for registered users (so they can develop scripts that fetch passwords from the site programmatically, or access the functionality via AJAX from within their own web apps).
- RESTful APIs
- Port the command-line app
hsxkpasswdto JavaScript using NodeJS.- Javascript programming outside the browser
- A Javascript CLI framework
A More Fluid Instalment Structure
The biggest change is going to be that the structure of instalments will become very variable. There will definitely still be regular instalments as we’re used to when ever we need to explore a new concept, or learn a new tool, but there will also be instalments dedicated to making technology choices, design decisions, and there will be episodes dedicated to reviewing specific pieces of work as worked examples.
I would also like there to be some new voices both in the written instalments, and on the podcast. I particularly hope this will be the case for the introduction of the core PHP language.
I feel very comfortable teaching JavaScript because it’s the main language I work in day-to-day, both in my professional and personal lives. At work I manage a codebase of a few hundred thousand lines of code spread over a few tens of classes, and all my personal projects are in JavaScript these days. I also maintain many legacy code bases consisting of many thousands of lines of Perl spread over a few tens of modules, and I’ve started to do a lot of smaller pieces of work in ZSH. What I spend very little time doing these days is writing PHP.
There was a time when PHP was my primary language, but that was a long time ago, back then the new shiny was PHP3, we’re now on PHP 8! PHP has gone through multiple major shifts since then, many of those shifts as significant as ES 6 was to JavaScript. My grasp of core programming principles and my fluency in many languages mean I’m comfortable going to php.net and reading the fine manual to learn specific syntax details, but, but what I don’t have is the real-world practical experience that’s foundational to all good advice!
Simply put — I need help!
A few members of the community have already volunteered to help out, but more help is definitely welcome. There’s room for all kinds of involvement — I could really do with a few people to bounce ideas off, and some help proof-reading instalments in-the-making would be very helpful. But, if you have the kind of real-world experience I’m missing, I’d love it if you could write the text for a few of the instalments, or even record segments with Allison. I don’t think it needs to be a single person writing an instalment, and it definitely wouldn’t need to be the same person writing the text and talking on the show. Some people are great writers but not comfortable on mic, and some love being on mic, but hate writing, I’ll take help in what ever for you’re comfortable giving it 🙂
More Listener Participation
The community building up around this series in the PBS channel on the NosillaCast Slack server is lovely to see, and I really hope this new phase will boost it even further.
Up until now there were small self-contained challenges for listeners/readers to work through alone, or maybe with a coding buddy or two from the community. There will still be challenges like that, particularly as we learn new technologies like JavaScript modules, and of course, PHP, but they’re going to become the exception more than the rule.
Instead, I foresee two distinct new avenues for participation — there’s the obvious one, participating in the HSXKPasswd project itself, but there is also a second one, parallel projects by one or more listeners to bring something you’ve been itching to exist into life. Every JavaScript project will benefit from code linting, documentation generation, a good test suite, the Node Package Manager, and ES 6 modules. Similarly, every web front-end will benefit from upgrading from Bootstrap 4 to 5, and learning about server-side code will open up huge array of possible new projects not yet imagined.
I really hope to see the community get stuck into the XKPasswd project in all sorts of ways, but I would be delighted if the community got stuck into more than just one project. It would be amazing if at the end of this new phase of the show we had three, five, or more real open source tools released to the world!
Regardless of how many or how few additional projects emerge from the community, there will many opportunities to engage with the HSXKPasswd community at all sorts of levels:
- There will be times when user feedback is needed — should be do something this way or some other way? Would that make your use of the tool easier or harder? What’s your biggest pain point? What’s the most important missing feature? And so on and so forth.
- This will be code written by human beings, it will have bugs! We’ll be using GitHub to manage the project, so reporting bugs as GitHub issues will be invaluable.
- Writing good code and writing good documentation are completely different skills. The documentation will be managed through GitHub, so anyone in the community will be able to submit improvements to the docs each time they run into something that confuses them, or that they believe is missing or wrong. Not to mention the inevitable typo fixes 🙂
- One of the advantages of using a linter is that it imposes a coding style, so submitting code that’s at home in the project will be easier than it would be without a linter. In the early days of the code base I’d like to do most of the coding to establish conventions and a style which I’ll be codifying in the project’s linter configuration and developer docs. But, once the core codebase is established, contributions by the community won’t just be welcome, they’ll be encourage!
If things turn out as a hope there’ll be similar opportunities in what ever other projects emerge from the community. Rather than everyone doing every challenge, I see people participating in the ways they feel comfortable, practicing the skills they want to nurture.
Sneak Peek
If you’d like to take a little sneak peek, why not check out the Perl version of the XKPasswd code, or, an unsuccessful first attempt of a JavaScript port on GitHub:
- The Crypt::HSXKPasswd Perl Module on GitHub
- The abortive first attempt at a JavaScript port on GitHub
Final Thoughts
What I’ve described here are my current thoughts on where this series is going. The milestones are unlikely to change, but the details of how we get there probably will. This second phase is likely to be much more community driven than the first phase, so now would be a great time to sign up to the NosillaCast Slack and join the Programming by Stealth channel 🙂
In the next instalment we’ll explore the path to the first milestone — HSXKPasswdJS existing as a portable JavaScript library that others can import into their JavaScript code, regardless of where it runs, be that in a browser, on a web server (yes, there is server-side JavaScript these days), in automation scripts, or on the command-line.