Category: Uncategorized

Posted on

An LXC Experiment for Live Backups

Don’t we all love our backups?  We all have them.  Some of us have backups done poorly, and some of us worry that ours is still not as good as we would like.  Few have it nailed.  We don’t have it nailed…

Here at EXPLOINSIGHTS, Inc. we think we are in the second camp (“not as good as we would like it to be”).  We have a ton of backups of our data, much of it offline (totally safe from e.g. malware), and some of those are in different locations (protected against theft, fire, flood, mayhem AND hackers), but they would all require a lot of work to get going if we ever needed them.  So, if we suffer a disaster (office burned to ground or swallowed up by a Tornado or house stolen by Hillary Clinton’s Russian Hackers), then rebuilding our system would still take time.  What we ALL want, but can seldom get, is a live backup that runs in parallel to our existing system.  Like a hidden ghost system that mirrors every change we make, without being exposed to the hazards of users and such.

So hold that thought…

Onto our favourite Linux Ubuntu capability: LXC.  LXC has a capability of basically exporting (copying or moving) containers from one machine to another – LIVE.  This means you don’t have to stop a container to take a backup copy of it and place it on ANOTHER MACHINE.  Theoretically, this is similar to taking a copy of a stopped container but without the drag of stopping it.

We know it has to be a pretty complicated under the hood for this to work,  and it’s evident it’s not really intended for a production environment, but we are going to play with this a little to see if we can use live migration to give us full working copies of our system servers on another machine.  And if we can, to place that machine not on the LAN, but on the WAN.

Our largest container is our Nextcloud instance.  We have multiple users, with all kinds of end-to-end encrypted AND simultaneously server-side encrypted data in multiple accounts.  All stored in one CONVENIENT container.  We are confident it’s SECURE from theft – we have tried to hack it.  All the useful data are encrypted.  But the container is growing.  Today it stands at about 138 GB.  Stopping that container and copying it even over the LAN is a slow process.  And that container is DOWN at that time.  If a user (or worse, a customer), tries to access the container to get a file, all they see is “server is down” in their browser.  #NotUseful

So for this reason, we don’t like to “copy” our containers – we hate the downtime risk.

So….we are going to play with live copying.  We have installed criu on two of our servers, and we are doing LAN-based experiments.  It’ll take time, as we have to copy-and-test.  Copy-and-test.  We have to make sure all accounts can be accessed AND that all data (double encrypted at rest, triple if you count the full-disk-encryption; quadruple if you count the https:// transport mechanism for copying) can be accessed without one byte of corruption.

Let the FUN begin.   We have written this short article as the first trial is underway.  We have our 138GB container on our “OB1” container (a Star Wars throwback):

See the last entry?  We are copying the RUNNING container to our second server (boringly called ‘exploinsights’).  It’s a big file, even for our super fast LAN router, and it’s not there even now:

The image has not yet appeared, but we have confirmed it’s still up ‘n running on the OB1 host.

Lots of testing to do, and clearly files this large can’t be backed up easily over the internet, so this is definitely an “as-well-as” non-routine option for a machine-to-machine back up, but we like the concept of this, so we are spending calories exploring it.

#StayTuned for an update.  Also, please let us know what you think of this – drop us an email at:

[email protected] or

[email protected]

UPDATE:

I need a good Plan B: Thecopy failed after a delay of about two hours.  It would not take that long to copy, so something is broken, and we are not going to try to dig our way out of that hole.  LXC live migration died on the day we tried it.  #RIP

Posted on

Another stress-free Nextcloud update, courtesy of LXC

Image result for nextcloud 14 update

The above image is definitely one that would ruin our morning coffee.  Definitely.

We do literally ALL of our business document management including generating and sharing our client files using the totally excellent Nextcloud product operating on a server in the smokey mountains in the Great State of Tennessee.  We NEED this server.

So, when you see a message saying ‘hey, there’s an update’ (just as we saw yesterday), it can make us worry.  But then, we remember…we use LXC.  We run our Nextcloud instance in an LXC container, backed up by a zfs file system with full, reliable and totally brilliant ‘snapshot’ capabilities.

So, how did we do our update?  From the command line, running as a NON-ROOT user (no scary sudo commands here!):

lxc snapshot Nextcloud
lxc exec Nextcloud bash
apt update && apt upgrade && apt autoremove
exit

This updates the container OS software (as it turns out, ours was already up to date – not a surprise, we check it regularly).

Then we log into our two-factor credential protected Nextcloud server via an HTTPS web portal, and we click on the Nextcloud updater.  About two minutes later, we saw this:

The update worked as it was supposed to.  But we were not just lucky – we had our snapshot.  Had the upgrade failed, we could and would restore the pre-updated version, then contact the Nextcloud support folks to figure out why our upgrade broke, and what to do about it.  Our mission-critical file cloud server is working just great, as usual.  LXC is our insurance policy.

Our coffee this morning tastes just great!  🙂

Use LXC.  Make YOUR coffee taste good too!

Posted on

SSH attacks – MOVE YOUR PORT #

Image result for ssh hackSo when we first set-up our remote linux logins, we used the standard SSH port # (22).  We use public keys for our SSH logins, so we weren’t especially worried about port scanners and bot attacks,

But it’s a funny thing going over your logs and seeing remote systems (with IP addresses that trace back to China and Russia, if that means anything) that are trying to login, even though we know for certain there are NO AUTHORIZED USERS at IP locations listed.  The logins fail, but even so…

So, some time ago, we moved to a different SSH port.  It’s not hard – just go to your ssh_config file and pick another unused port.  We have seen people pick ‘222’ and ‘2222’ and even ‘22222’ because, well, they associate ‘2’ with SSH.  We picked different numbers (and that’s all we are going to say on the subject).  Since that time (many months ago), we have not seen a SINGLE ssh login hack.  Not one.  Such a remarkable outcome was not expected.  Even today, going through two log files for our /var/log/auth.log, the only odd IP address we noted was actually from one of our approved mobile devices that was roaming in Europe (so that was not an issue).

It seems hackers are sometimes lazy.  They run a  port-scan of web sites starting presumably at port 22 and working from there.

We like that.  We hope that never changes.

What it tells us is…CHANGE YOUR SSH PORT # FROM 22 to <some-other-number> on any new Linux install.  Do not use port 22 unless you enjoy seeing a log entry of someone try to hack your server.

Enjoy your weekend, and if you adopt our suggestion, you might better enjoy reviewing your log files!  🙂

Posted on

Simple Folder copy using rsync over ssh

So sometimes you have a folder on one machine that you just need on another one.

We are GREAT FANS of ssh, and we think it’s a perfect secure tunnel for performing such operations.  So the question asked was how do you synchronise /path/to/folder  on the local machine to a remote machine using:

  1. Linux Ubuntu
  2. SSH
  3. A private SSH keyfile
  4. Over a non-standard SSH port

Well, since this linux, it’s actually rather boringly easy:

rsync -r -a -v -e 'ssh -p12345 -i .ssh/rsa-keyfile' /path/to/folder [email protected]:/home/username/path/to

This copies (if necessarily CREATING) ‘folder’ on the local machine to our remote one.  There are however some very important details.

Do NOT add a training ‘/’ to the paths if you want rsync to automatically create the directories.  So use this:

/path/to/folder   [email protected]:/home/uname/path/to

Not this:

/path/to/folder/   [email protected]:/home/uname/path/to/

The latter copies the files (and sub-directories) in /path/to/folder, whereas the former copies the folder, its contents and all its sub-directories, and thus creates the directory ‘folder’ at the remote machine if it does not exist.

Happy rsync’ing

 

Posted on

OpenSSH Vulnerability

We really don’t like headlines like this, but that doesn’t stop them coming:

The good news is that this is an FYI post, but the bad news is that ssh with public key authentication is affected.  And we use public keys.  And we are not changing as it’s still way safer than a friggin username/password login.

A private key password (that is never ‘remembered’) and 2FA helps.  But an OpenSSH update can’t come soon enough for us.

 

 

Posted on

Dodging the latest Intel CPU issues

****  UPDATE 27-AUG-2018   ****

We came across a nice article from our favourite cloud hosting software company (Nextcloud) on this vulnerability issue:

Security flaw in CPU’s breaks isolation between cloud containers

**** END OF UPDATE  ****

We noted a report on new Intel CPU security flaws:

“Intel’s chips are riddled with more speculative execution vulnerabilities”

Details are here.

On further reading, we note that two of the three issues are already fixed by Intel. Very cool. The unpatched issue relates to potential malicious containers operated by a third party.  Well the good news for us is we self host all of our containers, and we don’t allow third parties to host anything on our servers. Dodged a bullet there then.

If you use a Virtual Private Server (VPS) – i.e. if you basically host your IT services in someone else’s machine, you might want to rethink that strategy, as your service provider might be unwittingly hosting hackers who are out to Snoop on your containers.

Posted on

Why LXC

We deploy LXC containers in quite literally ALL of our production services.  And we have several development services, ALL of which are in LXC containers.

Why is that?  Why do we use LXC?

The truthful answer is “because we are not Linux experts”.  It really is true.  Almost embarrassingly so in fact.  Truth is, the SUDO command scares us: it’s SO POWERFUL that you can even brick a device with it (we know.  We did it).

We have tried to use single machines to host services.  It takes very little resources to run a Linux server, and even today’s laptops have more than enough hardware to support even a mid-size business (and we are not even mid-size).  The problem we faced was that whenever we tried “sudo” commands in Linux Ubuntu, something at sometime would go wrong – and we were always left wondering if we had somehow created a security weakness, or some other deficiency.  Damn you, SuperUser, for giving us the ability to break a machine in so many ways.

We kept re-installing the Linux OS on the machines and re-trying until we were exhausted.  We just could not feel COMFORTABLE messing around with an OS that was already busy dealing with the pervasive malware and hacker threats, without us unwittingly screwing up the system in new and novel ways.

And that’s when the light went on.  We thought: what if we could type in commands without worrying about consequences?  A world where anything goes at the command line is…heaven…for those that don’t know everything there is to know about Linux (which of course definitely includes us!).  On that day, we (re-) discovered “virtual machines”.  And LXC is, in our view, the BEST, if you are running a linux server.

LXC allows us to create virtual machines that use fewer resources than the host; machines that run as fast as bare-metal servers (actually, we have measured them to be even FASTER!).  But more than that, LXC with its incredibly powerful “snapshot” capability allows us to play GOD at the command line, and not worry about the consequences.

Because of LXC, we explore new capabilities all the time – looking at this new opensource project, or that new capability.  And we ALWAYS ALWAYS run it in an unprivileged LXC container (even if we have to work at it) because we can then sleep at night.

We found the following blog INCREDIBLY USEFUL – it inspired us to use LXC, and it gives “us mortals” here at Exploinsights, Inc. more than enough information to get started and become courageous with a command line!  And in our case, we have never looked back.  We ONLY EVER use LXC for our production services:

LXC 1.0: Blog post series [0/10]

We thank #UBUNTU and we thank #Stéphane Graber for the excellent LXC and the excellent development/tutorials respectively.

If you have EVER struggled to use Linux.  If the command line with “sudo” scares you (as it really should).  If you want god-like forgiveness for your efforts to create linux-based services (which are BRILLIANT when done right) then do yourself a favor: check out LXC at the above links on a clean Ubuntu server install.  (And no, we don’t get paid to say that).

We use LXC to run our own Nextcloud server (a life saver in our industry).  We operate TWO web sites (each in their own container), a superb self-hosted OnlyOffice document server and a front-end web proxy that sends the traffic to the right place.  Every service is self-CONTAINED in an LXC container.  No worries!

Other forms of virtualisation are also good, probably.  But if you know of anything as FAST and as GOOD as LXC…then, well, we are surprised and delighted for you.

Regards,

SysAdmin ([email protected])

 

 

 

Posted on

A Place for Everything…

…And Everything in its Place!

EXPLOINSIGHTS, INC. (EI) has created this separate site for documenting the home-office IT infrastructure journey.  This keeps the main (‘www.exploinsights.com‘) business web-site free of non-core business material.  Hopefully this helps customers who are looking for EI core services and also those who are trying to find solutions to the IT problems that arise when trying to operate small-business office IT services, but are not interested in any paid-for services by EI.

It’s hard to run a small business, and it’s even harder if you have or want to self-host some of your IT services to better comply with complicated regulations such as data-residency requirements.  This site is intended to provide a record of some of the practices, methodologies, issues and solutions that the IT System Administrator has to address and employ to running server(s) that support the office administration.

All the IT-related posts that were published on the main website have now been moved to the WordPress instance running on the ‘www.sysadmin.exploinsights.com’ sub-domain (which, of course runs in an LXD container).  The EI Sys-Admin will also of course create more articles in the future and post them here, including software reviews for the services provided as part of the EI office infrastructure.

Posted on

Why you should use Two-Factor credentials

This:
Backdoored Python Library Caught Stealing SSH Credentials
https://search.app.goo.gl/Sg2J7
If you have an SSH private key that is NOT password protected (i.e. a second factor) then your ssh logins can be completely stolen via this malware.
But even if you do protect your private key, maybe you should also look at two-factor ssh login using Google authenticator, which makes a private key just a small part of the authentication process.
 
 

Posted on

Linux – Full Disk Encryption with Remote Access

So if anyone is enjoying the ride bought on by NIST-800-171, you’ll know that data encryption is one of the many rages.  And a lovely thing it is too – AES encryption is safe from ANYONE this side of Quantum computers.  What’s not to love here?
Windows has Bitlocker.  Easy to use, albeit proprietary.  EXPLOINSIGHTS, Inc. uses it for mobile Windows devices as one of three major data encryption regimes employed as part of the NIST compliance for the company (translated into English: We don’t trust BitLocker alone :-)).
So, how hard can it be to enable full disk encryption on a Linux server?  Answer: dead easy – it’s part of the installer process for Ubuntu, for example.


The more INTERESTING question, and the subject of this article: how EASY is it to remotely and securely decrypt a Linux server that employs full disk encryption?  Answer: it’s a nightmare.  Trust us, we know. 🙁
BUT it can be done.  And this record here is to point people in the right direction for doing just that.  Firstly, google is your friend, so whatever your distro, ask Google.  I use Ubuntu Linux, so I found several useful online articles.  The rest of this article is Ubuntu based:

Perform your server (or Desktop) Linux install using full disk encryption.  This HAS to be done locally, as does the next steps:

This post:  https://www.theo-andreou.org/?p=1579  is genius.  Except where it isn’t, if you see what we mean.

  1. The article will work for you until you get to the line: 
    echo IP=10.0.0.67::10.0.0.1:255.255.255.0:encrypted-system:eth0:off >> /etc/initramfs-tools/initramfs.conf

    He doesn’t explain the above line well.  Here’s our attempt:

    echo IP=HOSTIPADDRESS::HOSTDHCPSERVER:255.255.255.0:HOSTNAME:<SEE-BELOW>:off
  1. Hopefully our version clarifies things.  When you create your server, it has a name.  It could be, e.g. “MyServer”.  Whatever it is, it goes in place of “HOSTNAME” above.  This is not terribly important; but this next bit is:
  2. The <SEE-BELOW>:off  really means you have to insert the name of your network adapter.  In days of old, it was called ‘eth0’, but rarely now.  So if you use the ‘eth0’ expression, as stated in the article, the darn thing WON’T WORK.  We know.  We tried.  We failed. It took us ages to figure that out.
  3. To find your network adapter name, use this command: 
    ls /sys/class/net

    …and it will show you the names of your network adaptors.  Ours is enp3s0, so We have to enter:

    HOSTIP::HOSTDHCP:255.255.255.0:enp3s0:off

    …at the end of the /etc/initramfs-tools/initramfs.conf file.  Your adaptor name will likely be different.

  4. Example, for a regular small-business router, that has 192.168.1.1 as the router address and also the place where dhcp requests go): 
    echo IP=192.168.1.115::192.168.1.1:255.255.255.0:mypcservername:enp3s0:off
  5. If you are trying a remote LUKS unlock then…GOOD LUCK!  It’s hard.  It’s also totally satisfying to watch a linux-OS hard disk encryption screen disappear after a remote login, and is perhaps even worth the effort (PERHAPS!!).  It’s not easy…  #JustSaying
  6. Here’s a screen-grab of me successfully unlocking one of our servers remotely.  We use public-key SSH login, and then busybox gives us a small menu of commands.  The one we want is ‘unlock’.  After entry, you are prompted for the decryption password.  If the key is accepted, the drive unlocks and you see ‘cryptsetup:  <drive-ref> set up successfully:

Proudly powered by WordPress