#security

Recently, one of our shared hosting webservers at Onlime GmbH got hit by a DoS attack. The attacker started a larger vulnerability scan against common Wordpress security issues. We already had common brute-force attack patterns on Wordpress covered by a custom Fail2Ban jail, which mainly trapped POST requests to xmlrpc.php or wp-login.php (the usual dumb WP brute-force attacks...). But this DoS attack had hundreds of customer sites as target and did not get trapped by our existing rules.

After having blocked the attacker's IP (glad this was no large-scale DDoS!), I wrote an extra Fail2Ban jail which traps such simple DoS attacks. It's a very basic Fail2Ban jail that should cover common attacks and should not cause any false positives as it is only getting triggered by a large amount of failed GET requests.

SSH authorized_keys allows you to define a command which is executed upon authentication with a specific key by prefixing it with the command="cmd" option.

authorized_keys man page explains this wonderful feature as follows:

Let's improve our Simple and Secure External Backup solution I have published back in 2018. Back then, I was using rsync over SSH to pull backup data, and LUKS encryption as full disk encryption for the external drives. As we all know, transferring data with rsync can get horribly slow and blow up your I/O if you're transferring millions of small files. Also, LUKS encryption may be a bit low level and inflexible. What we want to accomplish: A performant and secure backup solution based on ZFS, using zfs send|recv for efficient data transfer, and ZFS native encryption to secure our external drives. So let's go ahead and built that thing from scratch on a fresh 2021 stack!

macOS offers us a super simple way to fully encrypt an APFS formatted external drive: Right-click on the volume in the Finder and choose the option "Encrypt". Carbon Copy Cloner (CCC) also recommends this in its KB article Enabling encryption on a volume that will not contain an installation of macOS.

But macOS is a bit too transparent on that task. It simply starts encrypting the volume without any feedback or progress bar (WTF?). Even Disk Utility just shows the volume as APFS (Encrypted) - as it would have already completed full volume encryption.

If you are using Fail2ban, there is no standard recommended way to persistently ban IPs. Some people recommend to do this outside of Fail2ban, using e.g. iptables-persistent, which is actually super easy to install and configure. But let's say, we don't want to install any extras and want to accomplish the same with Fail2ban, as we already have fail2ban on every single host (which is a must!).

Back in 2013, I wrote about Linux process hiding using hidepid capabilities of procfs. On shared webhosting servers at Onlime GmbH, I have used the hidepid=2 mount option for procfs (/proc filesystem) for improved security. Like this, a regular system user (which could potentially be an evil customer that has gained SSH access and tries to spy on other's processes) does only see his own processes, all other processes are hidden.

This is great and super simple to enable, as it is part of the official Linux kernel for quite a while now. But things start to get a little trickier when we try to set up hidepid procfs mount option inside an LXC container. Enabling the mount option on the host system will not do! Inside an LXC container, a regular system user is still able to see all processes. Before LXC 2.1 (released in Sept 2017), this was also quite doable, as we just had to create a new AppArmor profile on the host system to allow the LXC container to set the /proc mount options. But since LXC 2.1 it got super tricky. I will present both solutions below, in case you have struggled with this hard one in newer LXC versions.

What we are going to set up here is a simple and secure offsite and offline backup server. Let's assume you already have an existing backup server that is connected to the internet 24/7 and does daily/weekly/monthly backups. We would now like to set up a second offsite backup server that just cares about storing data to encrypted external drive and after each backup run, you are going to physically detach that drive.

So, we are talking about offline backups in addition to the fact having this server offsite - at a different location than your main backup server.

Preferably, your main backup server would also be offsite. But as it needs to pull data frequently, its storage is always available and not getting detached.

Let's call your main backup server backup and the one we are going to set up here extbackup.

Years ago I wrote about the extensive sendmail wrapper with sender throttling, a pretty simple Perl script. It reliably provided throttling of the email volume per day by the sender’s original UID (user id). It also logged the pathes of scripts that sent emails directly via sendmail (e.g. via PHP’s mail() function). The main flaw in the original sendmail wrapper was security, though. As in Linux, every executable script must be readable by the user that calls it, the throttle table in MySQL was basically open and every customer could manipulate it. Every customer could raise his own throttling limit and circumvent it.

Today, I’m publishing my new sendmail-wrapper that is going to fix all the flaws of the previous version and add some nice extras. The new sendmail-wrapper is written entirely in PHP and does not require any external libraries. It is a complete rewrite and has pretty much nothing in common with the old Perl version.

Webhosting customers are messies, at least some of them - or (sadly, that's the truth) the bigger part of them. Some people still think they can run the same blog software or CMS for years without ever caring about upgrading. I tell my customers over and over how important it is, to keep their website up-to-date and don't let any outdated code lying around. Still, as long as their website doesn't get hacked or defaced, they don't really seem to care.

If you're in the same situation as me and you are providing webhosting services to your friends or customers, read on.

Five years ago I wrote about kernel based process hiding in Linux (see my old blog posts: Simple process hiding kernel patch, Process hiding Kernel patch for 2.6.24.x, RSBAC – Kernel based process hiding). It got time to continue the story and finally present you a real solution without the hassle of a self-compiled kernel.

How can I prevent users from seeing processes that do not belong to them?

In January 2012, Vasiliy Kulikov came up with a kernel patch that solved the problem nicely by adding a hidepid mount option for procfs. The patch landed in Linux kernel 3.3.

In the meantime, this patch luckily also landed in the 3.2 kernel of Debian Wheezy (see backport request in Debian bug report #669028). This feature has been also pushed back into the kernel of Red Hat Enterprise Linux 6.3 (see RHEL 6.3 Release Notes), and from there to CentOS 6.3 and Scientific Linux 6.3. Recently, this feature was even backported to the 2.6.18 kernel in RHEL 5.9.

With the release of Proxmox VE 3.0 back in May 2013, the Proxmox VE web interface does no longer require Apache. Instead, they're using now a new event driven API server called pveproxy. That was actually a great step ahead, as we all know Apache get's bulkier every day and the new pveproxy is a much more lightweight solution. But the question arose: How do I protect my Proxmox VE WebUI with basic user authentication?

Basically, we do not trust any web application out there so we better double protect the whole WebUI with plain old basic auth - previously done in Apache by .htaccess.