Another stress-free Nextcloud update, courtesy of LXC

Image result for nextcloud 14 update

The above image is definitely one that would ruin our morning coffee.  Definitely.

We do literally ALL of our business document management including generating and sharing our client files using the totally excellent Nextcloud product operating on a server in the smokey mountains in the Great State of Tennessee.  We NEED this server.

So, when you see a message saying ‘hey, there’s an update’ (just as we saw yesterday), it can make us worry.  But then, we remember…we use LXC.  We run our Nextcloud instance in an LXC container, backed up by a zfs file system with full, reliable and totally brilliant ‘snapshot’ capabilities.

So, how did we do our update?  From the command line, running as a NON-ROOT user (no scary sudo commands here!):

lxc snapshot Nextcloud
lxc exec Nextcloud bash
apt update && apt upgrade && apt autoremove
exit

This updates the container OS software (as it turns out, ours was already up to date – not a surprise, we check it regularly).

Then we log into our two-factor credential protected Nextcloud server via an HTTPS web portal, and we click on the Nextcloud updater.  About two minutes later, we saw this:

The update worked as it was supposed to.  But we were not just lucky – we had our snapshot.  Had the upgrade failed, we could and would restore the pre-updated version, then contact the Nextcloud support folks to figure out why our upgrade broke, and what to do about it.  Our mission-critical file cloud server is working just great, as usual.  LXC is our insurance policy.

Our coffee this morning tastes just great!  🙂

Use LXC.  Make YOUR coffee taste good too!

Interactively Updating LXC containers

We love our LXC containers.  They make it so easy to provide and update services – snapshots take most of the fear out of the process, as we have discussed previously here.  But even so, we are instinctively lazy and are always looking for ways to make updates EASIER.  Now it’s possible to fully automate the updating of a running service in an LXC container BUT a responsible administrator wants to know what’s going on when key applications are being updated.  We created a compromise, a simple script that runs an interactive process to backup and update our containers.  It saves us repetitively typing the same commands, but it still keeps us fully in control as we answer yes/no upgrade related questions.  We thought our script is worth sharing.  So, without further ado, here’s our script, which you can just copy and paste to a file in your home directory (called say ‘update-containers.sh’).  Then just run the script when you want to update and upgrade your containers.  Don’t forget to change the name(s) of your linux containers in the ‘names=…’ line of the script:

#!/bin/bash
#
# Simple Container Update Script
# Interactively update lxc containers

# Change THESE ENTRIES with container names and remote path:
#
names='container-name-1 c-name2 c-name3 name4 nameN'

# Now we just run a loop until all containers are backed up & updated
#
for name in $names
do
echo ""
echo "Creating a Snapshot of container:" $name
lxc snapshot $name
echo "Updating container:" $name
lxc exec $name apt update
lxc exec $name apt upgrade
lxc exec $name apt autoremove
echo "Container updated. Re-starting..."
lxc restart $name
echo ""
done
echo "All containers updated"

Also, after you save it, don’t forget to chmod the file if you run it as a regular script:

chmod +x container-update.sh

Now run the script:

./container-update

Note – no need to run using ‘sudo’ i.e. as ROOT user- this is LXC, we like to be run with minimal privileges so as not to ever break anything important!

So this simple script, which runs in Ubuntu or equivalex distro, does the following INTERACTIVELY for every container you name:

lxc snapshot container #Make a full backup, in case the update fails
apt update             #Update the repositories
apt upgrade            #Upgrade everything possible
apt autoremove         #Free up space by deleting old files 
restart container      #Make changes take effect

This process is repeated for every container that is named.  The ‘lxc snapshot’ is very useful: sometimes an ‘apt upgrade’ breaks the system.  In our case, we can then EASILY restore the container to its prior updated state using the ‘lxc restore command.  All you have to do is firstly find out a containers snapshot name:

lxc info container-name

E.g. – here’s the output of ‘lxc info’ on one of our real live containers:

sysadmin@server1:~lxc info office

Name: office
Remote: unix://
Architecture: x86_64
Created: 2018/07/24 07:02 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 21139
Ips:
eth0: inet 192.168.1.28
eth0: inet6 fe80::216:3eff:feab:4453
lo: inet 127.0.0.1
lo: inet6 ::1
Resources:
Processes: 198
Disk usage:
root: 1.71GB
Memory usage:
Memory (current): 301.40MB
Memory (peak): 376.90MB
Network usage:
eth0:
Bytes received: 2.52MB
Bytes sent: 1.12MB
Packets received: 32258
Packets sent: 16224
lo:
Bytes received: 2.81MB
Bytes sent: 2.81MB
Packets received: 18614
Packets sent: 18614
Snapshots:
http-works (taken at 2018/07/24 07:07 UTC) (stateless)
https-works (taken at 2018/07/24 09:59 UTC) (stateless)
snap1 (taken at 2018/08/07 07:37 UTC) (stateless)

The snapshots are listed at the end of the info screen.  This container has three: the most recent being called ‘snap1’.  We can restore our container to that state by issuing:

lxc restore office snap1

…and then we have our container back just where it was before we tried (and hypothetically failed) to update it.   So we could do more investigating to find out what’s breaking and then take corrective action.

The rest of the script is boiler-plate linux updating on Ubuntu, but it’s interactive in that you still have to accept proposed upgrade(s) – we call that “responsible upgrading”.  Finally, each container is restarted so that the correct changes are propagated.  This gives a BRIEF downtime of each container (typically 1-several seconds).  Don’t do this if you cannot afford even a few seconds of downtime.

We run this script manually once a week or so, and it makes the whole container update process less-painful and thus EASIER.

Happy LXC container updating!

Server Updates

Updating critical software is something not to be taken lightly.  It’s nerve-wracking when your business operations rely upon such systems.
What has helped EXPLOINSIGHTS Inc. (EI) sleep better at night is the extensive use of unprivileged containers or so-called virtual machines.  Most of the EI support software is installed on unprivileged LXC containers, which is a standard component of the Ubuntu 16.04 Linux distribution.
Today was a typical day for EI: an update to a major release of Nextcloud.  This critical software houses EI’s data and is the hub for data sharing with customers and stakeholders.  If this upgrade goes wrong, my customers can’t download their files.  #Embarrassing – or maybe even worse; loss of critical data?  To make it more enjoyable, I am not in the office today – so the update has to be performed remotely via secure SSH.  That’s an excellent recipe for high stress…normally.
So how did EI mitigate this update risk? With the following simple command entered at the host machine terminal via secure SSH access (i.e. WITHOUT SuperUser privileges!):
LXC snapshot NC pre-13-upgrade
That’s it.  Painless.  Super-safe (no Superuser rights!).  Blindingly fast.  Very efficient.  And this creates a full working snapshot of the EI current cloud configuration – files, links, settings, SSL-certs, SQL database, apache2 configs – absolutely everything needed to completely restore the setup should the upgrade process break something critical.
Breaking this command down:

  • LXC – this is the command we issue to fire up the Ubuntu LXC/LXD virtual machine management hypervisor, followed by three parameters:
    • snapshot – tells LXC to take a full working snapshot of the running instance;
    • NC – the name of the EI container that runs the Nextcloud instance – the one we want to backup;
    • pre-13-upgrade – a name assigned to the snapshot (easy to remember).

Yes, it’s that simple.  After that, the Nextcloud upgrade process was initiated…and as it happens, everything went smoothly, so the snapshot was NOT actually needed to recover the pre-version-13 upgrade – but it will be kept for a while just to make sure there are no bugs waiting in the shadows.  Here’s the new EI cloud instance:

EI cloud software – UPDATED to latest version

If a major problem arises, then the following command entered at the same terminal, again as a non-SuperUser, restores the entire pre-version 13 instance:
LXC restore NC pre-13-upgrade
#NoWorries 🙂
This restore command overwrites the current instance with the pre-upgraded and fully functioning snapshot.  The only risk is losing files/links created since completing the upgrade process – way better than a total rebuild.
LXC makes it so convenient to update major platforms.  The entire process was fast, safe; easy.  And because all the work is performed in non-SuperUser unprivileged mode, it comes with the confidence of knowing you can’t accidentally break an important part of the core system on the way.  It’s so good, it’s almost boring – but only almost.
Checkout using LXC to run your small business support software, it’s better than prescription-grade sleeping tablets for helping you with the upgrade process!  Official documents are here.  And there’s a ton of useful tutorials to get you started – Google is your friend.