Tag: Nextcloud

Posted on

Smart Nextcloud Backup Strategy

You know I am a fan of Nextcloud. I got so used to having this for my consulting company that even though the company itself is no long with us, the Nextcloud servers found personal utility. that shouldn’t be a surprise for anyone who has looked into Nextcloud.

The big problem for personal users (and even some small businesses) is…the backup strategy for Nextcloud.

You can certainly backup your user data files easily enough – just take a snapshot copy of user files and copy it to e.g. a USB drive and pop it in your safe and you are done. Update it weekly or whatever, and you have a reasonable backup set for your user files. Not terriible, not perfect, but it’s fine. And certainly better than nothing.

But for Nextcloud, a users data files is just one small part of the server. There’s apps, settings, configurations; and shares and links with passwords and expiration dates, and all kinds of ‘nice’ features that, well, also ideally need to be backed up.

If you read online documets, they tell you to run a seperate mysql server via some complicated setup that’s really just not practical for home users. So it seems we’re doomed to just having our files backed up, and if the server goes pooft, we have to rebuild it from ground up – and then go back and add all the finishing touches: apps, user settings, Nextclould configuration options, links, theming etc. etc.

I have previously reported how if you run your Nextcloud server as an lxc container (really simple under Ubuntu linux) then you can copy the containers from one machine to another, and it helps with some of the configurations. But at the time of writing, you had to copy an entire server – that can quickly grow into TB+ storage drives for user data. That’s a slow copy even over a lan (and forget it for wan). And it becomes impracticable for routine (e.g. daily) backups of a server

BUT Now…we have LXD 4.0. And it has a new command option that fixes all of that:

~$: lxc copy container remote:container –refresh

the key word, or rather an option, is –refresh. If you setup a Nextcloud container called (e.g.) nextcloud on a local machine, and you have another, remote machine called (e.g.) BackupServer (can be on same lan or even halfway around the world) then this command copies your container to the other machine without having to stop the existing machine. It copies everything about your instance – it is an exact copy:

~$: lxc copy nextcloud BackupServer:nextcloud –refresh

All you need is two machies running LXD version 4.0 or later and enough HD space for your backup copy (copies). So far so good – the same as prior releases of lxd.

And if you re-run that command say every day you get your offsite copy refreshed to be an updated version of your server – and it only sync’s changes, not the whole server. So your first copy might be slow, but subsequent ones will be quite fast. And there’s no need to stop your local container to copy it (the remote copy however has to be stopped).

You can cron this, so that it makes refreshed copies say daily (or more), and then makes weekly unrefreshed backups after say 7 days, then monthly and so forth. You can have BACKUP copies of OLD servers if you want or need to go back, and you can always have your live version backup daily (or more). Note that multiple backup copies of a server image take up the same space per copy, so storage cost may become an option (but spinning-rust HD’s are cheap, so probably not amajor issue for the home enthusiast or even small business).

If your current server goes down, all you do is start the remote one, point a router to it, maybe update a DNS server (if it’s on a different public IP address) and you are back in business. Same name, links, shares, setttngs, users, data, files, confiuration, settings etc. It all just *works* – just as it should.

You can be up and running in less than five minutes from when you detect your primary server is down. That makes for pretty good uptime availability of your server.

We think this –refresh option is a game-changer for lxd.

Happy Backups!

Posted on

Securely Sharing Files from Nextcloud

So we’ve been asked ‘How do you share files from Nextcloud securely’?

It’s a fair question because if you END-TO-END encrypt a file, you can’t actually share the file unless you also share a means of decrypting it.  And we don’t share our end-to-end encryption keys, EVER.

So how do we manage sharing?  How do we protect our files as we hand them over to our customers?  The short answer is with LAYERS of security:

Our files are protected by one of several levels of independent security:

Robust SSL/TLS connectivity (HTTPS all digital coms)

End-to-end encryption for files

Server-side encryption for files

Full Disk Encryption (FDE) for computer drives (server or desktop)

We can ignore FDE here as it doesn’t really come into play (it’s only really applicable here to a stolen PC).

For normal at-rest files, we have end-to-end AND server-side encryption in play.  So our files are protected against threats of all kinds.  You just can’t get at our data without TWO separate sets of credentials, neither of which are connected, neither of which are stored in a file.

When we share a file, we have to remove one of these layers (the end-to-end encryption), and that of course makes the protection weaker during the process of sharing FOR THAT FILE, since it only has server-side encryption to keep it safe.  Note that this is not “bad” – the file is still encrypted, and it’s safe from even quite determined hackers.  But you know, we like to be a bit paranoid and very responsible, so we bring other measures in play.

Firstly, here’s a real file from our server we created with end-to-end encryption still applied:

It’s very exciting.  #notReally.  You can see, the name is complete gibberish.  And for the record, if you download this ‘blob’ and view it in Notepad, this is what you get:

“îe¤ÎÈCÀ’0pMi%?/£`ÕÕäÔ}CÇ+>ê3ؽ[Æ~©HïïþÑàß·”M*ÓÝì?L‚L*x)š!ýU\nW£7zšK‚ubÎ
91Ïií=¡l‰­¸üN#åS›¬Æàçý*¨”PÔãºÕë*,8ƒÇŠûìÒ†Áè‹™p\ \©Jšs|Ö_ž›”ÒÿEÊ ÈÈõÁÁ”

And as you will see later, the real file is actually a simple text message.

Once the file has end-to-end encryption removed, we can now more easily find it on the server AS THE SYS-ADMIN (or as a hacker that has somehow penetrated our system).  It takes a while, but you can find it:

Wow, there it is.  Plain as day.  This file is actually present in an lxc container.  If you can ‘see’ this, you are ither a SysAdmin or an unauthorized hacker.  But the good news, you are not at the system root (you just think you are), so your damage potential is still very limited.

The file we created especially for this demo is shown among other directories.  The server ADMIN or hacker (not necessarily the owner of the file) can “see” the file, so it seems.  But before everyone panics, let’s take a close look at what he or she can actually “see”.  There’s a file name and permissions and modification time and apparent size (meta-data if you will).  You can also see file/folder names too.  Gosh, does that mean the SYS-ADMIN or hacker can snoop on my files?  Let’s look at the contents using cat:

Well that doesn’t look terribly useful.  As you can see, the DATA are still encrypted.  Albeit this time via the SERVER keys.  A hacker and even the SysAdmin can’t look at file contents (but it’s true, he or she can see the name and other file meta data).  So, the file is still quite well protected against hacking and theft.

In fact, the ONLY way you can view these data is if you unlock the server-side encryption that’s now in effect for that file.  And for this, you either need an exploit that works against Nextcloud’s encryption (very unlikely) OR you need to somehow secure the credentials needed to login to the Nextcloud account which will then unlock these files.  For that, you need a Nextcloud password AND 2FA code (one that changes every 30 seconds), so that too we think is quite safe from unauthorized disclosure.  But when you do enter those data, the file becomes fully decrypted and is then more visible, albeit over an HTTPS connection only (our final level of protection):

So “finally”, this is a file we can read:  HelloWorld.txt, that says “Hello everyone.  Have a nice day”.  So the Nextcloud user can read the file (just as he/she should).

When it comes to SHARING this file, we actually share an HTTPS:// link for that file that includes the decryption credentials for that file.  In Nextcloud, this is what that looks like:

You can see on the bottom right that a sharing link has been created.  We also add a PASSWORD to that link and an EXPIRATION DATE for the link, so that the link will not be active for long AND you need the extra password to access the file.  We send the LINK to the file via regular email.  Regular email is not terribly safe.  It can be intercepted.  We know that.  But let’s continue:

This link looks like this:

https://cloudvault.exploinsights.com/index.php/s/aeX7irmbWfneipn

(The link won’t exist for long, so it’s probably not worth trying, sorry!).

If you click on the link and it’s still active, it will bring up THIS page:

Which is still not the file.  You need that password.  We do not send the password with the link.  We don’t even send the password via email.  We send that via another means, say WhatsApp, or Signal or even SMS.  So, to get to this file, you need the link (from an email sent to you) AND the password (sent in a different way) sent to you.  Finally, if you get the link and the password, you can click/enter the top secret password assigned to protect this file, you get rewarded in this case with this:

The file.  In decrypted form.  Finally!  You can download the file but ONLY via https:// (SSL/TLS) connection (our final layer), so even then it remains encrypted-in-transit until it hits your download folder.  Only then do we relinquish our management –> OVER TO YOU!  🙂

So that’s how we share our data.  it is ALWAYS protected with up to four layers of encryption: FDE, End-to-End, Server-side and lastly SSL/TLS.

These “layers” are gradually stripped off as we get closer and closer to sharing a file with you, but only when it lands on your machine is our last (but strong) layer of encryption removed.  We use 2FA credentials to share our data and even the Sys-Admin (or unauthorized hacker) has very little access to our data: they NEVER see raw decrypted data, but occasionally they can sometimes view a file name and some basic meta-data.

Is this perfect?  No.  We are always looking to improve it.  This system is our latest and greatest implementation, but it won’t be our last.  Is this pretty good?  We think so!  What do you think?

So this is how we ROLL our file-sharing; how we securely share data with our customer.  What do you do?

Comments by email only – [email protected] or you can catch us on Twitter @SysAdminEI

Posted on

Meeting Compliance with the aid of Nextcloud, Cryptomator and Mountain Duck

So like all SysAdmins, we have a lot to worry about in order to continually meet the burdens of compliance.  For us, data residency (location) and digital “state” (encryption status) is very important.

We have had a productive few days improving the PERFORMANCE of our systems by using better-integrated software.  So, why is PERFORMANCE being addressed under compliance?  Well, simply, because if you make a compliant system easier to use, users are more likely to use them, and thus be more compliant.

It’s no secret, we are great fans of Nextcloud – we self-host our cloud server, and because we use that to host our data, we use several different security layers to help thwart accidental and malicious exposure.  Our cloud server, based at our office HQ in Tennessee, is where we store all of our important data.  So we try to make it secure.

Firstly, our cloud server can only be accessed by HTTPS (SSL/TLS).  Logging in requires 2FA credentials:

This gets you to the next step:

So this is good.  And we have server-side encryption enabled in Nextcloud, which provides an additional layer of security – not even the SysAdmin can read the client data on the server.

This is cool.  Server-side encryption does help us with sharing of files to customers, but because the decryption key is stored on the server, we don’t like to rely solely on that for protecting our important data.

So our data are end-to-end encrypted with Cryptomator – an open source, AES encryption software package, with apps for Linux, Windows and Android (we use all three) – and even apps for IOS for those that like their Apples too (we don’t).

Of course, one of the things that makes this system “inconvenient” is that it’s hard to access end-to-end cloud-encrypted files quickly.  On the server, they look like THIS:

Not entirely useful.  Really hard to work on a file like this.  You have to  physically download your encrypted file(s) (say using the webdav sync software platform, like the great one provided by Nextcloud) and store them locally on your device, then decrypt them locally, on your device, so you can work on them.  This is fine when you are in your office, but what about when you are on the road (as we often are)?

Data residency and security issues can arise with this method of working when on travel, so you can’t download files en-mass to your device and decrypt them all when you are in the “wrong place”.  You have to wait until you can get back to a right location before you can work.  Customers don’t really like the delays this can cause them, and we don’t blame them.  And worse still, in that situation, even when you are done, you have to delete (or even ERASE) files on your PC if you are going back on travel etc. again to a location where data residency becomes an issue.  Then when you get to a secure location again, you have to repeat this entire process for working on your files again. This is hard to do.  Believe us, we know, as we have had to do this, but we now have a better way.

A more secure but LESS CONVENIENT way is to somehow only download the files you need as you need them, and decrypt and work on them etc.  ONLY AS YOU NEED THEM.   This is more secure, as you only have one decrypted file on your device (the one you are working on in your word processor etc), but how can that be done and be done CONVENIENTLY?

Obviously, this “convenience v security” issue is one we have spent a lot of time looking at.  We have used webdavs connected to our cloud servers and tried to selectively sync folder(s).  It’s not efficient, not pretty, not fast (actually really slow) and sometimes it just doesn’t work.

But thankfully we now have a much faster, reliable, efficient, effective yet totally compliant way of addressing the problem of keeping files end-to-end encrypted yet still be able to work on them ONE AT A TIME even when you are in a location that can otherwise bring data residency issues.

For us, this requires several systems that have to work together, but they are built (mostly) on Open Source software that has, in some cases, been tried and tested for many years so is probably as good as you can get today:

  • OpenSSH – for Secure Shell connectivity;
  • Nextcloud server;
  • Nextcloud sync clients;
  • Cryptomator;
  • 2FA Authentication credential management apps;
  • And last but not least…Mountain Duck.

SSH is an established, reliable secure service used globally to securely connect to servers.  If configured correctly, they are very reliable.  We believe ours are configured properly and hardened, not least because NONE of our SSH connections work with a username/password login.  Every connection requires a strong public/private key combination, and every key itself is further password protected; and, each SSH connection ALSO requires a second-factor (2FA) code.  We think that’s pretty good.  We have already explained Nextcloud and Cryptomator.  2FA apps are the apps that generate the six-digit code that changes on your phone every 30 seconds.  We have written about a new one we like (PIN protected), so we won’t go into that anymore.  That leaves ‘Mountain Duck’.  Yes ‘Mountain Duck‘.  We know, it’s not a name one naturally gives to a secure file-access system, but bear with us (and in any case, we didn’t name it!).  Put simply, Mountain duck allows us to use our 2FA protected SSH connections to our servers to access our files, but it does so in a really neat way:

In the image above, taken from their we site, note the text we circled in red.  Mountain Duck comes with CRYPTOMATOR integration.  So this effectively makes Mountain Duck a ‘Windows explorer’ that can be 2FA-connected via SSH to a server to access and decrypt in real-time Cryptomator end-to-end encrypted files stored on our HQ-based servers.  To that, we say:

Just WOW.

So how does this work in the real world; how do we use this?

Well we have a Nextcloud sync client that syncs each users files to a separate LXC container running on the corporate server.  Each container is used to sync the users files between the Nextcloud server and the Nextcloud user files.  Both the Nextcloud server AND the Nextcloud client files are ultimately stored in the HQ facilities, albeit in very different LXC containers.  Files are end-to-end encrypted in both the Nextcloud server and the Nextcloud client container.  All are further protected by full disk encryption and SSL/TLS connectivity.

Whether we are on the road OR in the office, we work on our files by logging into our Nextcloud client container using OpenSSH and Mountain Duck.

It’s all a lot simpler than it might sound:  First, we connect to our Nextcloud client containers via strong SSH credentials via Mountain Duck’s friendly interface, which asks first for our private key password:

And then asks for our 2FA code:

The 2FA code is generated on an our 2FA App on our encrypted, locked android smartphone, in a PIN protected app (We use ‘Protectimus‘ and also ‘andOTP‘).

With these credentials, Mountain Duck logs into our Nextcloud client container via the secure SSH connection, and then it accesses our end-to-end encrypted files but in doing so, it automatically detects our Cryptomator Vault (because it’s built into Mountain Duck) And it then allows us (if we want) to UNLOCK our Cryptomator Vault and access it:

So in one operation and three “codes”, we can securely connect to our Nextcloud client container via secure SSH and access our end-to-end encrypted files from anywhere in the world (where there’s internet!).

And Mountain Duck makes this easier because it allows you to bookmark the connection: open Mountain Duck and click a bookmark to the SSH server then enter your SSH password/2FA code.  Mountain Duck logs into your server and accesses the files located there.  It can even remember your passwords if you want (we don’t do that, as we don’t trust Windows much at all), but you could configure this to JUST require a 2FA code if you want.

The whole process takes, maybe, 30 seconds including the time to get the phone out and obtain a 2FA code.  Not bad!  And once you have done that, you can work on your END TO END encrypted files stored on your office based Nextcloud client container files from anywhere in the world.  No files are needed to be on your PC – in the office or on the road.  Everything remains solidly encrypted so if you do get hacked, there are no readable data that can be stolen, so at least that threat is low.  And everything is going through modern, secure OpenSSH transfer protocols, which in our case, makes us sleep a lot better than having some proprietary code with back-doors and all sorts of unpleasant surprises that always seem to arise.

The catch?  Well, you do need internet connectivity or you can do NOTHING on your data.  The risk in the office is low, but when your on the road it does happen, so it’s still not perfect. 🙂

Also, you do have to set this stuff up with some poor SysAdmin.  But if we can do it, probably anyone can?

Finally, this is what it looks like for real after we go through that hard-to-explain-but-easy-to-do login.  A window appears and you can see your files just like a normal file view on your Windows PC.

Above is a regular Windows Explorer window, accessing the Nextcloud client container files via fast and secure SSH.  The folder ‘Vault’ is actually a Cryptomator end-to-end encrypted folder, but because we have entered the credentials (above), it can be accessed like any other.  Files inside can be copied, edited, saved, shared, printed etc. just as files in the other folders.  It’s totally transparent, and it’s TOTALLY AWESOME.  The files in the Nextcloud client container (and the Nextcloud server) remain fully encrypted all the time.  A file ONLY gets decrypted when it’s selected and downloaded by the user.  Viewing a file in explorer (as in the view above) does NOT decrypt it – you have to double-click etc. the file to initiate downloading and decryption.  All your changes get immediately sync’d to the Nextcloud client container (which immediately sync’s everything back to the server).  Nothing gets stored on the PC you are using to work on your files unless you deliberately save it to the device.

So, THIS is how WE roll our own end-to-end encrypted data storage.  How do you do yours?

questions or comments by email only [email protected]

 

Posted on

LXC Container Migration – WORKING

So we found a spare hour at a remote location and thought we could tinker a little more with lxc live migration as part of our LXD experiments.

Related image

We executed the following in a terminal as NON-ROOT users yet again:

lxc copy Nextcloud EI:Nextcloud-BAK-13-Sep-18

lxc start EI:Nextcloud-BAK-13-Sep-18

lxc list EI: | grep Nextcloud-BAK-Sep-13-Sep-18

And we got this at the terminal (a little time later…)

| Nextcloud-BAK-13-Sep-18 | RUNNING | 192.168.1.38 (eth0) | | PERSISTENT | 0 |

Note that this is a 138GB file.  Not small by any standard.  It holds every single file that’s important to our business (server-side AND end-to-end encrypted of course).  That’s a big file-copy.  So even at LAN speed, this gave us enough time to make some really good coffee!

So we then modified our front-end haproxy server to redirect traffic intended for our primary cloud-file server to this lxc instance instead. (Two minor changes to a config, replacing the IP address of the current cloud to the new cloud).  Then we restarted our proxy server and….sharp intake of breath…

IT WORKED BEAUTIFULLY!

Almost unbelievably, our entire public-facing cloud server was now running on another machine (just a few feet away as it happens).   We hoped for this, but we really did not expect a 138GB file to copy and startup first time.  #WOW

We need to test and work this instance to death to make sure it’s every bit as SOUND as our primary server, which is now back online and this backup version is just sleeping.

Note that this is a complete working copy of our entire cloud infrastructure – the Nextcloud software, every single file, all the HTTPS: certs, databases, configurations, OS – everything.  A user changes NOTHING to access this site, in fact, it’s not even possible for them to know it’s any different.

We think this is amazing, and is a great reflection of the abilities of lxc, which is why we are such big fans,

With this set-up, we could create working copies of our servers in another geo-location every, say, month, or maybe even every week (once a day is too much for a geo-remote facility – 138GB for this one server over the intenet?  Yikes).

So yes, bandwidth needed IS significant, and thus you can’t flash the larger server images over the internet every day, but it does provide for a very resistant disaster-recovery situation: if our premises go up in a Tornado, we can be back online with just a few clicks from a web-browser (change DNS settings and maybe a router setting or two) and issue a few commands from an ssh terminal, connected to the backup facility.

We will develop a proper, sensible strategy for using this technique after we have tested it extensively, but for now, we are happy it works.  It gives us another level of redundancy for our updating and backup processes.

GOTTA LOVE LXD

Image result for love LXD

Posted on

Another stress-free Nextcloud update, courtesy of LXC

Image result for nextcloud 14 update

The above image is definitely one that would ruin our morning coffee.  Definitely.

We do literally ALL of our business document management including generating and sharing our client files using the totally excellent Nextcloud product operating on a server in the smokey mountains in the Great State of Tennessee.  We NEED this server.

So, when you see a message saying ‘hey, there’s an update’ (just as we saw yesterday), it can make us worry.  But then, we remember…we use LXC.  We run our Nextcloud instance in an LXC container, backed up by a zfs file system with full, reliable and totally brilliant ‘snapshot’ capabilities.

So, how did we do our update?  From the command line, running as a NON-ROOT user (no scary sudo commands here!):

lxc snapshot Nextcloud
lxc exec Nextcloud bash
apt update && apt upgrade && apt autoremove
exit

This updates the container OS software (as it turns out, ours was already up to date – not a surprise, we check it regularly).

Then we log into our two-factor credential protected Nextcloud server via an HTTPS web portal, and we click on the Nextcloud updater.  About two minutes later, we saw this:

The update worked as it was supposed to.  But we were not just lucky – we had our snapshot.  Had the upgrade failed, we could and would restore the pre-updated version, then contact the Nextcloud support folks to figure out why our upgrade broke, and what to do about it.  Our mission-critical file cloud server is working just great, as usual.  LXC is our insurance policy.

Our coffee this morning tastes just great!  🙂

Use LXC.  Make YOUR coffee taste good too!

Posted on

Why LXC

We deploy LXC containers in quite literally ALL of our production services.  And we have several development services, ALL of which are in LXC containers.

Why is that?  Why do we use LXC?

The truthful answer is “because we are not Linux experts”.  It really is true.  Almost embarrassingly so in fact.  Truth is, the SUDO command scares us: it’s SO POWERFUL that you can even brick a device with it (we know.  We did it).

We have tried to use single machines to host services.  It takes very little resources to run a Linux server, and even today’s laptops have more than enough hardware to support even a mid-size business (and we are not even mid-size).  The problem we faced was that whenever we tried “sudo” commands in Linux Ubuntu, something at sometime would go wrong – and we were always left wondering if we had somehow created a security weakness, or some other deficiency.  Damn you, SuperUser, for giving us the ability to break a machine in so many ways.

We kept re-installing the Linux OS on the machines and re-trying until we were exhausted.  We just could not feel COMFORTABLE messing around with an OS that was already busy dealing with the pervasive malware and hacker threats, without us unwittingly screwing up the system in new and novel ways.

And that’s when the light went on.  We thought: what if we could type in commands without worrying about consequences?  A world where anything goes at the command line is…heaven…for those that don’t know everything there is to know about Linux (which of course definitely includes us!).  On that day, we (re-) discovered “virtual machines”.  And LXC is, in our view, the BEST, if you are running a linux server.

LXC allows us to create virtual machines that use fewer resources than the host; machines that run as fast as bare-metal servers (actually, we have measured them to be even FASTER!).  But more than that, LXC with its incredibly powerful “snapshot” capability allows us to play GOD at the command line, and not worry about the consequences.

Because of LXC, we explore new capabilities all the time – looking at this new opensource project, or that new capability.  And we ALWAYS ALWAYS run it in an unprivileged LXC container (even if we have to work at it) because we can then sleep at night.

We found the following blog INCREDIBLY USEFUL – it inspired us to use LXC, and it gives “us mortals” here at Exploinsights, Inc. more than enough information to get started and become courageous with a command line!  And in our case, we have never looked back.  We ONLY EVER use LXC for our production services:

LXC 1.0: Blog post series [0/10]

We thank #UBUNTU and we thank #Stéphane Graber for the excellent LXC and the excellent development/tutorials respectively.

If you have EVER struggled to use Linux.  If the command line with “sudo” scares you (as it really should).  If you want god-like forgiveness for your efforts to create linux-based services (which are BRILLIANT when done right) then do yourself a favor: check out LXC at the above links on a clean Ubuntu server install.  (And no, we don’t get paid to say that).

We use LXC to run our own Nextcloud server (a life saver in our industry).  We operate TWO web sites (each in their own container), a superb self-hosted OnlyOffice document server and a front-end web proxy that sends the traffic to the right place.  Every service is self-CONTAINED in an LXC container.  No worries!

Other forms of virtualisation are also good, probably.  But if you know of anything as FAST and as GOOD as LXC…then, well, we are surprised and delighted for you.

Regards,

SysAdmin ([email protected])

 

 

 

Posted on

Server Security

Routine maintenance and general configuration management of a new  cloud server supporting EXPLOINSIGHTS installed as an LXC instance is going ok.  Nextcloud’s security scan has given me a top rating for a new installation, which is encouraging but not enough to rest on laurels.  This installation is currently a mirror (in terms of files) of a current install on a live server, but after testing, this will become the main cloud server for the organizations needs and the older server will be retired.

 
This version is of course running with server-side file encryption.  As part of the testing process, client-side encryption (a feature of Nextcloud version 13) will be evaluated, as BoxCryptor (the current Exploinsights, Inc. end to end encryption service) causes a few operational issues.

Posted on

Server Updates

Updating critical software is something not to be taken lightly.  It’s nerve-wracking when your business operations rely upon such systems.
What has helped EXPLOINSIGHTS Inc. (EI) sleep better at night is the extensive use of unprivileged containers or so-called virtual machines.  Most of the EI support software is installed on unprivileged LXC containers, which is a standard component of the Ubuntu 16.04 Linux distribution.
Today was a typical day for EI: an update to a major release of Nextcloud.  This critical software houses EI’s data and is the hub for data sharing with customers and stakeholders.  If this upgrade goes wrong, my customers can’t download their files.  #Embarrassing – or maybe even worse; loss of critical data?  To make it more enjoyable, I am not in the office today – so the update has to be performed remotely via secure SSH.  That’s an excellent recipe for high stress…normally.
So how did EI mitigate this update risk? With the following simple command entered at the host machine terminal via secure SSH access (i.e. WITHOUT SuperUser privileges!):
LXC snapshot NC pre-13-upgrade
That’s it.  Painless.  Super-safe (no Superuser rights!).  Blindingly fast.  Very efficient.  And this creates a full working snapshot of the EI current cloud configuration – files, links, settings, SSL-certs, SQL database, apache2 configs – absolutely everything needed to completely restore the setup should the upgrade process break something critical.
Breaking this command down:

  • LXC – this is the command we issue to fire up the Ubuntu LXC/LXD virtual machine management hypervisor, followed by three parameters:
    • snapshot – tells LXC to take a full working snapshot of the running instance;
    • NC – the name of the EI container that runs the Nextcloud instance – the one we want to backup;
    • pre-13-upgrade – a name assigned to the snapshot (easy to remember).

Yes, it’s that simple.  After that, the Nextcloud upgrade process was initiated…and as it happens, everything went smoothly, so the snapshot was NOT actually needed to recover the pre-version-13 upgrade – but it will be kept for a while just to make sure there are no bugs waiting in the shadows.  Here’s the new EI cloud instance:

EI cloud software – UPDATED to latest version

If a major problem arises, then the following command entered at the same terminal, again as a non-SuperUser, restores the entire pre-version 13 instance:
LXC restore NC pre-13-upgrade
#NoWorries 🙂
This restore command overwrites the current instance with the pre-upgraded and fully functioning snapshot.  The only risk is losing files/links created since completing the upgrade process – way better than a total rebuild.
LXC makes it so convenient to update major platforms.  The entire process was fast, safe; easy.  And because all the work is performed in non-SuperUser unprivileged mode, it comes with the confidence of knowing you can’t accidentally break an important part of the core system on the way.  It’s so good, it’s almost boring – but only almost.
Checkout using LXC to run your small business support software, it’s better than prescription-grade sleeping tablets for helping you with the upgrade process!  Official documents are here.  And there’s a ton of useful tutorials to get you started – Google is your friend.

Proudly powered by WordPress