Server Updates

Updating critical software is something not to be taken lightly.  It’s nerve-wracking when your business operations rely upon such systems.
What has helped EXPLOINSIGHTS Inc. (EI) sleep better at night is the extensive use of unprivileged containers or so-called virtual machines.  Most of the EI support software is installed on unprivileged LXC containers, which is a standard component of the Ubuntu 16.04 Linux distribution.
Today was a typical day for EI: an update to a major release of Nextcloud.  This critical software houses EI’s data and is the hub for data sharing with customers and stakeholders.  If this upgrade goes wrong, my customers can’t download their files.  #Embarrassing – or maybe even worse; loss of critical data?  To make it more enjoyable, I am not in the office today – so the update has to be performed remotely via secure SSH.  That’s an excellent recipe for high stress…normally.
So how did EI mitigate this update risk? With the following simple command entered at the host machine terminal via secure SSH access (i.e. WITHOUT SuperUser privileges!):
LXC snapshot NC pre-13-upgrade
That’s it.  Painless.  Super-safe (no Superuser rights!).  Blindingly fast.  Very efficient.  And this creates a full working snapshot of the EI current cloud configuration – files, links, settings, SSL-certs, SQL database, apache2 configs – absolutely everything needed to completely restore the setup should the upgrade process break something critical.
Breaking this command down:

  • LXC – this is the command we issue to fire up the Ubuntu LXC/LXD virtual machine management hypervisor, followed by three parameters:
    • snapshot – tells LXC to take a full working snapshot of the running instance;
    • NC – the name of the EI container that runs the Nextcloud instance – the one we want to backup;
    • pre-13-upgrade – a name assigned to the snapshot (easy to remember).

Yes, it’s that simple.  After that, the Nextcloud upgrade process was initiated…and as it happens, everything went smoothly, so the snapshot was NOT actually needed to recover the pre-version-13 upgrade – but it will be kept for a while just to make sure there are no bugs waiting in the shadows.  Here’s the new EI cloud instance:

EI cloud software – UPDATED to latest version

If a major problem arises, then the following command entered at the same terminal, again as a non-SuperUser, restores the entire pre-version 13 instance:
LXC restore NC pre-13-upgrade
#NoWorries 🙂
This restore command overwrites the current instance with the pre-upgraded and fully functioning snapshot.  The only risk is losing files/links created since completing the upgrade process – way better than a total rebuild.
LXC makes it so convenient to update major platforms.  The entire process was fast, safe; easy.  And because all the work is performed in non-SuperUser unprivileged mode, it comes with the confidence of knowing you can’t accidentally break an important part of the core system on the way.  It’s so good, it’s almost boring – but only almost.
Checkout using LXC to run your small business support software, it’s better than prescription-grade sleeping tablets for helping you with the upgrade process!  Official documents are here.  And there’s a ton of useful tutorials to get you started – Google is your friend.

OpenVPN Server

EXPLOINSIGHTS now has its own VPN service.  As expected, the transfer speeds are much slower then when in the main office or on a trusted network:
But it is at least operational as a beta-service today, and the speeds are more than adequate for simple file transfers/emails.  This provides me with greater end to end data protection as and when I connect via hotels/airports/coffee-houses etc.
I know the service is working because this blog entry is made via the VPN network, and it shows my IP location as being:

And I am not physically near Kingsport now, just as the VPN service should so indicate.
And it works great on my Android devices too:

For the techies: I tried installing this in an LXC container but failed, with all kinds of permission errors that I could not fix, so the service is installed on a completely separate device for now.  I would be grateful for some pointers from anyone on how to overcome the permission-issues of openvpn installation on an unprivileged LXC container.  🙂

SSH Public-Private Keys

So in reviewing the var/log/auth.log etc. files today, I note a drastic (as in 100%!) reduction in unauthorized SSH login attempts at the EXPLOINSIGHTS servers.  Bloody brilliant!
This either means:

  • All the hackers have turned into good guys; OR
  • Russia and China are on holiday (prior month SSH hack attempts, of which there were many had IP addresses “from these nations”, which is hardly a smoking gun as IP’s can be spoofed, but it’s all I had to go on); OR
  • Using public-private SSH login keys AND two-factor login has done it’s job* – yay!

Two-factor login is enabled for the EI servers and the services thereon.  If you are not using two-factor then I say THANK YOU – it makes you a more attractive target, so hopefully the hackers will continue to leave me alone for a while.  🙂
Tip:  And on the subject of SSH two-factor login, it took me quite a while to figure out why I couldn’t log back into my servers AFTER the first reboot after I enabled two-factor.  All the tests via SSH worked before the reboot.  It drove me NUTS.  The reason was that my Ubuntu encrypted /home/user directory is NOT unlocked via SSH login, so the SSH tunnel could not read my keys at ~.ssh/authorized_keys thus I failed the first and biggest login hurdle.  Either disable /home/user folder encryption or move your encryption keys to another location (say /var/SSH).


*I actually also did something else beyond two-factor and pub-private keys, but I am not sharing that publicly.  Message me, and if I feel like trusting you (a real email address goes a long way…), I will let you know my third factor.  🙂

Managing Power Cuts

Well it’s not as if power cuts are a new thing.  But I had to learn this before I was ready of course:

  • Router resets itself after a power-cut.  No surprises there;
  • Ubuntu LXC containers assign a new MACVLAN address when you launch a new container from an old one (like after major changes, or a make-over, as I have done a few times now);
  • If containers have a new MACVLAN, they won’t connect to your static assigned LAN IP’s after a power-cut, but they won’t drop a current router LAN connection beforehand even if the MACVLAN MAC ID changes.  In other words, unless you remember to do this at the time of creating a new container, you have a surprise coming the next time the router reboots;
  • ..After said power-cut, if your public-IP facing server is one that had a new MAC ID assigned then your your public-IP facing server suddenly has no port 80 and 443 traffic, as the router is sending it to the OLD fixed LAN IP;
  • You go to access a site (say this blog…), and nothing works!

NET RESULT  –>  Everything goes down and it requires a little groping-in-the-dark to find out what happened.

  • My fault.  I should reset my connections when I launched new containers.  Better yet, I might even one day soon learn to use the non-MACVLAN network connector so I don’t have to bother that anymore, but that’s a job for another day.
  • Dumb-luck bonus: my SSH is tunneled directly to my host machine, not the LXC containers, and the MAC address of my hardware at least has not changed – so I could at least log back in via SSH to the LXC host after the router had power restored.  That made it easy to see my new LAN IP addresses so I could make correct assignments to the router for forwarding ports from WAN to LAN; and to the haproxy server for routing to the LAN servers.  So the tip here is – if you elect to use MACVLAN, always SSH into the LXC host, not the LXC container.  🙂

So today’s admin work: reset static LAN IP’s to the LXC containers that have new MAC ID’s via MACVLAN setting and forward ports to the new IP address of the haproxy public-ip facing proxy server.   And all this was after power was restored, which in itself was more groping in the dark… 🙂
Tip:  you can find/verify MAC ID from this command:
lxc info container_name
And, also today…after that self-inflicted fiasco, I ordered an uninterruptible power supply for my modem and router as power-cuts clearly have the potential to catch me flat-footed again.  🙂

New Office Capability

Part of my NIST-800-171 Implementation and Compliance Plan:

I now have a portal to an “OnlyOffice” document office server to create and edit Office documents ONLINE.  The “OnlyOffice” document server is linked to the EXPLOINSIGHTS Nextcloud file storage server so I can create and store documents that are stored at my work office, all via HTTPS connection of course:

And of course I can share directly from the server, using two-factor authentication with expiring links:

But I have work to do on the SSL side, not that this is “terrible”, but I know it can be better.  This is the system at launch (1-Feb-2018).  Watch this space…

UPDATE 6th Feb 2018 – FIXED SSL cert (via web-server configuration):

I could probably beef up the other parameters, but I reduce my audience a little: older browsers might not be able to handle my already tight configuration.  So this is it for a while, until I learn of a reason to spend more time on this.  🙂

2FA – WordPress

WordPress fans FYI.  This plugin:-

  • Two-Factor Authentication for WordPress using the Android/iPhone/Blackberry app as One Time Password generator.
  • Version 0.48 | By Henrik Schack | View details  🙂

WORKS with the release of WordPress 4.9.2, even though the plugin records say it’s “untested” with this version of WP.  🙂

Self-Hosting – The Journey Begins

So this blog site is part of EXPLOINSIGHTS (EI) journey into a self-hosted IT infrastructure, caused by trying to navigate the seemingly conflicting information system requirements for complying with NIST-800-171 and ITAR.  Maintaining an effective office document information system that works with your business model is not as easy as a small business would like.  EI quickly concluded that achieving compliance is easier if you DON’T use the convenient services that we have all taken for granted.  For EI’s journey, this meant moving away from the admittedly excellent Microsoft Office 365 product.  But baby steps are needed, as Office 365 (and the Google Apps equivalent) is robust and capable.  It’s not easy trying to match that.  However, as of the date of this article, EI is now the owner-operator of a server that hosts:-

  • A cloud file system using Nextcloud;
    • Files shared with Customers are via a link to this secure site.  Two factor authentication is enabled for all users and file shares.  That’s already at least as good as Microsoft’s OneDrive service, but it’s self-hosted so data residency compliance at least is a non-issue.
  • A WordPress blog site;
    • If you’re reading this article, then the site works!  Let me know how your experience is please.  EI’s blog site is also self-hosted.
  • A web site.

The web site was in fact established first (, but there’s actually nothing but a placeholder on it today, as it’s really just being used as a portal to the other two self-hosted services.  That’s a work-in-progress.
The journey to get here started a long time ago, and it’s not been easy even though I used to consider myself to be reasonably tech-savvy (not anymore – Linux OS and command-line driven programs like Apache web-server have made me rethink how good my IT skills are).
This is still a beginning for EI.  This self-hosting journey has more to go.  Microsoft Office 365 is a low-cost small-office service that EI once used with confidence, but their privacy-mining practices are discomforting at best and more importantly, their data servers and thus your company and customer information are located in places we don’t really know or manage as well as we need to for maximum compliance .  The solutions to these privacy and data-residency etc. problems using proprietary software from cloud-based organizations like Microsoft and Google are difficult on the best of days.  So EI believes it has more to do in terms of self-hosting on company owned hardware.  On the list of potential projects includes:

  • Self-hosted email;
    • Several options exist.  “Open Source” is the only decision made so far.  Watch this space!
  • Self-hosted OnlyOffice file server –
    • An Open Source but otherwise very Microsoft-format-compatible office suite (spreadsheets, documents and presentations);
    • Desktop versions exist BUT sometimes you need to work on a secure web portal.  A self-hosted option exists!
  • Self-hosted Virtual Private Network (VPN).
    • Open Source of course – do you see the trend?

And this has to be done all the while trying to keep up with the latest bad-guy malware and security threats; and keeping effective backups that can mitigate anything the bad guys can throw at us.
A few tips from EI for anyone who ventures down the self-hosting path:
1. Reconsider your Operating System

  • Microsoft Windows is part of the problem for small businesses, not the solution.  Proprietary software makes things worse from a compliance perspective.
    • EI runs on Ubuntu 16.04.  Linux rules when it comes to self-hosting options.  Ubuntu does use some proprietary code, but most of it is based on Debian, which is…Open Source.

2. Use (Linux) Containers

  • Using non-privileged containers is the way to go with Linux.  Virtual Machine snapshots make backups a breeze AND they are so forgiving when you screw things up (just restore your prior snapshot).  More importantly, they add an enterprise-level of defense against hackers.  If one web-service is compromised, containers make it harder for them to spread, so damage can be minimized.  Oh, and take snapshots often!  EI runs on LXD containers, but you have other Virtual Machine options if you prefer something different.

3. End to end file-encryption is your inconvenient friend

  • Cloud file servers are so useful, but if you use one, use client-side encryption.  “Security is not convenient” (that phrase brazenly plagiarized from an EI customer) – but it is essential.  Client-side encryption with strong keys and passwords WILL save your data from being exposed.  It won’t stop you losing it, but it WILL stop it being used by someone else.
EI’s cloud files – encrypted BEFORE they hit the server
  • EI employs Linux Encfs (Open Source) and Boxcryptor (Proprietary) for client-side file encryption.  The next official release of Nextcloud has end-to-end encryption built in, so it might provide an even better option.

4. Check things THOROUGHLY before you go live

  • Use online services to check and validate your security before
  • you go live.  These two helped me a lot, and there are multiple free web-server security checking sites:

Hello world!

Welcome to EXPLOINSINGHTS Inc. Blog Site!
This is my first ever posting here, and the concept of a blog is itself an experiment.  The intention is to use this site to share things that EXPLOINSIGHTS Inc. cares about – both from a customer and stakeholder perspective:

  • Information and insights on the services provided by EI to the ammunition and energetic materials community; and
  • The challenges that EI as an organization has to overcome to survive and hopefully thrive in the industry we serve.