So we found a spare hour at a remote location and thought we could tinker a little more with lxc live migration as part of our LXD experiments.
We executed the following in a terminal as NON-ROOT users yet again:
lxc copy Nextcloud EI:Nextcloud-BAK-13-Sep-18 lxc start EI:Nextcloud-BAK-13-Sep-18 lxc list EI: | grep Nextcloud-BAK-Sep-13-Sep-18
And we got this at the terminal (a little time later…)
| Nextcloud-BAK-13-Sep-18 | RUNNING | 192.168.1.38 (eth0) | | PERSISTENT | 0 |
Note that this is a 138GB file. Not small by any standard. It holds every single file that’s important to our business (server-side AND end-to-end encrypted of course). That’s a big file-copy. So even at LAN speed, this gave us enough time to make some really good coffee!
So we then modified our front-end haproxy server to redirect traffic intended for our primary cloud-file server to this lxc instance instead. (Two minor changes to a config, replacing the IP address of the current cloud to the new cloud). Then we restarted our proxy server and….sharp intake of breath…
IT WORKED BEAUTIFULLY!
Almost unbelievably, our entire public-facing cloud server was now running on another machine (just a few feet away as it happens). We hoped for this, but we really did not expect a 138GB file to copy and startup first time. #WOW
We need to test and work this instance to death to make sure it’s every bit as SOUND as our primary server, which is now back online and this backup version is just sleeping.
Note that this is a complete working copy of our entire cloud infrastructure – the Nextcloud software, every single file, all the HTTPS: certs, databases, configurations, OS – everything. A user changes NOTHING to access this site, in fact, it’s not even possible for them to know it’s any different.
We think this is amazing, and is a great reflection of the abilities of lxc, which is why we are such big fans,
With this set-up, we could create working copies of our servers in another geo-location every, say, month, or maybe even every week (once a day is too much for a geo-remote facility – 138GB for this one server over the intenet? Yikes).
So yes, bandwidth needed IS significant, and thus you can’t flash the larger server images over the internet every day, but it does provide for a very resistant disaster-recovery situation: if our premises go up in a Tornado, we can be back online with just a few clicks from a web-browser (change DNS settings and maybe a router setting or two) and issue a few commands from an ssh terminal, connected to the backup facility.
We will develop a proper, sensible strategy for using this technique after we have tested it extensively, but for now, we are happy it works. It gives us another level of redundancy for our updating and backup processes.
GOTTA LOVE LXD
