Site Down Time - December 19

For anyone tracking my sanity depletion efforts over the weekend, let me provide an update and explanation.

First, please note that after being down from 15:00-ish Saturday, Dec 19; the whateley site was active once again as of about 03:00 PST Sunday Dec 20 after being down for roughly 12 hrs. I then took a break for food/rest and handled the remaining problems in fully restoring things by setting up an SSL certificate and some additional minor details that weren't specifically tied into the restoration. I completed this at about 17:30 Sunday.

The problem occurred during an attempt to upgrade the version of our Debian OS and PHP stack, because Joomla and its extensions get feisty about needing up to date systems to work properly. While there are good security reasons to stay up to date and these improvements do tend to run 'better' as features are implemented, some of the problem is artificially created... after all, if the extension/software runs today, just because there is a new OS or PHP version released... the existing software still works on the versions; as long as you don't update to versions that rely on new versions of the foundation. But... artificial limitations are present that make these extensions stop working too far beyond the 'supported life span'. This is one of several reasons we had to build this new site... the parchment site (the one used for the past 5-6 yrs) had important extensions ceasing to work even though we had no ability to upgrade the OS/PHP versions. As we discovered yesterday, we STILL don't have a viable way to upgrade the OS/PHP. Between the way Google handles the VM upgrades and the way the stacks are installed; any upgrade in the stack results in the installed software (Joomla) ceasing to work.

I spent several hours trying to 'make' this upgrade work. But eventually, it became clear that I wasn't going to be able to do this with the resources available. Fortunately, I don't 'trust' Google far enough to throw them... and who can throw a multinational monopoly like Google? So I had make backups before starting. Two different types, actually. One using a Joomla extension, the second using Google's snapshot system.

Now, in theory, a VM snapshot should be more or less the ideal restore method. After all, the whole thing... server and all the components, active processes, everything... its all just in memory... virtual. A snapshot literally just records the current state of the whole thing and stores it. So a restore should just as accurately resume were the snapshot ended. And you should be able to do so with all the same details as you started with... same IP, same certificates, everything. But no. When it DOES work, you have to use the snapshot to create an entirely new virtual machine separate from the broken one you are restoring. Importantly, it doesn't use the same IP address. So certificates fail, domains need to be reset, and so on. We did that once, a number of years ago. This time? It didn't work. The restore seemed to work... and I could SSH into the account and see all the things. But on the web side? Any attempt to access the site just reached a non-configured Apache server as if the joomla / whateley account wasn't even present.

After time trying to make that work? I moved on to the extension based backup. This requires a system set up to serve a joomla site BUT... without Joomla present. In theory, I could have done this over the old server with existing certificate and IP addresses... but I decided that even broken, leaving that resource available for potential file copies was more valuable. So after deleting the failed snapshot based VM, I created yet ANOTHER Joomla VM. Then deleted the Joomla installation, tranferred in the backup file (and moving a Gig of archived data via the stock Google SSH program is SLOW), and then installed it.

This worked. I fixed the domain / DNS registries to point to the new IP address... and then called it a night.

This evening I finished the process... getting an SSL certificate via Let's Encrypt and getting it properly setup/configured on the site.

At this point, everything about the site should be restored to what it was prior to the attempt to upgrade PHP. If you experience any further issues, please let me know.

If you appreciate the effort, please remember that the site operates on the kindness of the community and Patreon donations are VERY welcome (there is a link in the right side bar below the Mail inbox and above the Discord link).