Routine maintenance and general configuration management of a new cloud server supporting EXPLOINSIGHTS installed as an LXC instance is going ok. Nextcloud’s security scan has given me a top rating for a new installation, which is encouraging but not enough to rest on laurels. This installation is currently a mirror (in terms of files) of a current install on a live server, but after testing, this will become the main cloud server for the organizations needs and the older server will be retired.
This version is of course running with server-side file encryption. As part of the testing process, client-side encryption (a feature of Nextcloud version 13) will be evaluated, as BoxCryptor (the current Exploinsights, Inc. end to end encryption service) causes a few operational issues.
Server Updates
Updating critical software is something not to be taken lightly. It’s nerve-wracking when your business operations rely upon such systems.
What has helped EXPLOINSIGHTS Inc. (EI) sleep better at night is the extensive use of unprivileged containers or so-called virtual machines. Most of the EI support software is installed on unprivileged LXC containers, which is a standard component of the Ubuntu 16.04 Linux distribution.
Today was a typical day for EI: an update to a major release of Nextcloud. This critical software houses EI’s data and is the hub for data sharing with customers and stakeholders. If this upgrade goes wrong, my customers can’t download their files. #Embarrassing – or maybe even worse; loss of critical data? To make it more enjoyable, I am not in the office today – so the update has to be performed remotely via secure SSH. That’s an excellent recipe for high stress…normally.
So how did EI mitigate this update risk? With the following simple command entered at the host machine terminal via secure SSH access (i.e. WITHOUT SuperUser privileges!):
LXC snapshot NC pre-13-upgrade
That’s it. Painless. Super-safe (no Superuser rights!). Blindingly fast. Very efficient. And this creates a full working snapshot of the EI current cloud configuration – files, links, settings, SSL-certs, SQL database, apache2 configs – absolutely everything needed to completely restore the setup should the upgrade process break something critical.
Breaking this command down:
- LXC – this is the command we issue to fire up the Ubuntu LXC/LXD virtual machine management hypervisor, followed by three parameters:
- snapshot – tells LXC to take a full working snapshot of the running instance;
- NC – the name of the EI container that runs the Nextcloud instance – the one we want to backup;
- pre-13-upgrade – a name assigned to the snapshot (easy to remember).
Yes, it’s that simple. After that, the Nextcloud upgrade process was initiated…and as it happens, everything went smoothly, so the snapshot was NOT actually needed to recover the pre-version-13 upgrade – but it will be kept for a while just to make sure there are no bugs waiting in the shadows. Here’s the new EI cloud instance:
If a major problem arises, then the following command entered at the same terminal, again as a non-SuperUser, restores the entire pre-version 13 instance:
LXC restore NC pre-13-upgrade
#NoWorries 🙂
This restore command overwrites the current instance with the pre-upgraded and fully functioning snapshot. The only risk is losing files/links created since completing the upgrade process – way better than a total rebuild.
LXC makes it so convenient to update major platforms. The entire process was fast, safe; easy. And because all the work is performed in non-SuperUser unprivileged mode, it comes with the confidence of knowing you can’t accidentally break an important part of the core system on the way. It’s so good, it’s almost boring – but only almost.
Checkout using LXC to run your small business support software, it’s better than prescription-grade sleeping tablets for helping you with the upgrade process! Official documents are here. And there’s a ton of useful tutorials to get you started – Google is your friend.
Managing Power Cuts
Well it’s not as if power cuts are a new thing. But I had to learn this before I was ready of course:
- Router resets itself after a power-cut. No surprises there;
- Ubuntu LXC containers assign a new MACVLAN address when you launch a new container from an old one (like after major changes, or a make-over, as I have done a few times now);
- If containers have a new MACVLAN, they won’t connect to your static assigned LAN IP’s after a power-cut, but they won’t drop a current router LAN connection beforehand even if the MACVLAN MAC ID changes. In other words, unless you remember to do this at the time of creating a new container, you have a surprise coming the next time the router reboots;
- ..After said power-cut, if your public-IP facing server is one that had a new MAC ID assigned then your your public-IP facing server suddenly has no port 80 and 443 traffic, as the router is sending it to the OLD fixed LAN IP;
- You go to access a site (say this blog…), and nothing works!
NET RESULT –> Everything goes down and it requires a little groping-in-the-dark to find out what happened.
- My fault. I should reset my connections when I launched new containers. Better yet, I might even one day soon learn to use the non-MACVLAN network connector so I don’t have to bother that anymore, but that’s a job for another day.
- Dumb-luck bonus: my SSH is tunneled directly to my host machine, not the LXC containers, and the MAC address of my hardware at least has not changed – so I could at least log back in via SSH to the LXC host after the router had power restored. That made it easy to see my new LAN IP addresses so I could make correct assignments to the router for forwarding ports from WAN to LAN; and to the haproxy server for routing to the LAN servers. So the tip here is – if you elect to use MACVLAN, always SSH into the LXC host, not the LXC container. 🙂
So today’s admin work: reset static LAN IP’s to the LXC containers that have new MAC ID’s via MACVLAN setting and forward ports to the new IP address of the haproxy public-ip facing proxy server. And all this was after power was restored, which in itself was more groping in the dark… 🙂
Tip: you can find/verify MAC ID from this command:
lxc info container_name
And, also today…after that self-inflicted fiasco, I ordered an uninterruptible power supply for my modem and router as power-cuts clearly have the potential to catch me flat-footed again. 🙂